Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Contrastive learning peaks at three clinical modalities, and a modality-gated LSTM then beats the five-modality plateau.

desk verdict Useful systematic contrastive sweep in MIMIC with a promising gated fusion, but Appendix 8.3's split description is ambiguous enough to put the headline numbers on hold until clarified. read the letter →

arxiv 2507.03165 v1 pith:YKWVFJBM submitted 2025-07-03 cs.LG

Michal Golovanevsky , Pranav Mahableshwarkar , Carsten Eickhoff , Ritambhara Singh This is my paper
classification cs.LG
keywords multimodallearningcontrastiveMIMIC-IVModality-GatedLSTMOne-Versus-Otherslossin-hospitalmortalitypredictionphenotypingclinical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes PiCME, a pipeline that pre-trains contrastive models on all 26 combinations of two to five clinical modalities from MIMIC (discharge summaries, radiology reports, chest X-rays, demographics, and time-series) and evaluates them on in-hospital mortality and phenotyping. It finds that contrastive representations are competitive with, and sometimes better than, fully supervised training when only two or three modalities are used, with the best three-modality combination reaching 75.94% AUROC on mortality. Performance degrades as more modalities are added, for both contrastive and supervised fusion. To recover from that plateau, the paper introduces a Modality-Gated LSTM that weights each modality by a contrastively learned importance parameter, achieving the best five-modality results: 76.93% AUROC and 62.26% AUPRC for mortality and 74.36% AUROC for phenotyping. It also reports that these learned weights align with Integrated Gradients attributions and that the gated model generalizes more consistently across demographic subgroups.

What carries the argument

The load-bearing mechanism is the weighted One-Versus-Others (OvO) contrastive loss with learnable per-modality weights $\lambda_i$, combined with the Modality-Gated LSTM (mLSTM). In the OvO loss, each modality is pulled toward the averaged embedding of all other modalities, and each loss term is scaled by a softmax-normalized $\lambda_i$ that is trained jointly with the encoders; these $\lambda$ values become the modality-importance signal. The mLSTM then injects those frozen $\lambda$ values into the LSTM cell update as $C_t = F_t \odot C_{t-1} + (I_t \odot \tilde{C}_t) \odot \vec{\lambda}_t$, so that modalities judged important during pre-training contribute more to the hidden state during supervised fine-tuning.

What would settle it

Open the data-split implementation and check whether any patient in the fine-tuning test split contributed a modality view to contrastive pre-training; if any do, rerun contrastive pre-training on only the 80% subset and compare the five-modality mLSTM results (reported 76.93% AUROC and 62.26% AUPRC against a 73.19% AUROC and 51.27% AUPRC supervised baseline). If the margin collapses, leakage is confirmed; if it persists, the claim survives.

Watch

Extended reading notes

Core claim

The central claim is that systematically scaling contrastive pre-training in a clinical multimodal setting reveals a three-modality peak and a subsequent plateau, and that this plateau can be overcome by transferring contrastively learned modality importances into a supervised fusion model. Concretely, the paper asserts that across all 26 modality subsets of MIMIC, the contrastive model peaks at discharge text plus chest X-ray plus demographics (75.94% AUROC for in-hospital mortality), then declines to 61.44% AUROC with all five modalities, while a fully supervised concatenation baseline saturates around 73% AUROC. The Modality-Gated LSTM, which scales the candidate memory by modality-specific $\lambda$ values learned during contrastive pre-training, is claimed to be the fusion method that exceeds the plateau, reaching 76.93% AUROC and 62.26% AUPRC for mortality and 74.36% AUROC for phenotyping with all five modalities. The paper further claims that contrastively learned $\lambda$ values are validated by Integrated Gradients attributions and that the gated model is the most consistent across gender, ethnicity, and age subgroups.

Load-bearing premise

The paper's central comparison assumes that no patient used to fine-tune and test the models ever appears in contrastive pre-training, yet the appendix states that contrastive learning uses the full original training set while fine-tuning uses an 80/10/10 split of the remaining data, with sample sizes left as placeholders.

Editorial extensions

If this is right

  • For two- or three-modality clinical tasks, contrastive pre-training with frozen encoders can match or beat end-to-end supervised training, so a single reusable embedding model can support multiple downstream tasks without task-specific labeled data.
  • Adding a fourth or fifth modality by simple concatenation hurts both contrastive and supervised models, so raw modality count is not a reliable proxy for predictive value.
  • Contrastively learned modality weights, normally a byproduct of pre-training, can act as a regularizer in supervised fusion and recover the performance lost to integration noise.
  • Modality importance estimated without labels (the $\lambda$ values from pre-training) tracks task-specific attribution (Integrated Gradients), suggesting that the same pre-training can inform interpretability and model selection.
  • The gated fusion's consistency across age, gender, and ethnicity subgroups suggests that dynamic modality weighting can reduce demographic performance gaps relative to fixed fusion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its three-modality peak may be partly shaped by encoder choice: both text modalities share ClinicalBERT, which likely inflates text-text alignment and could make text-heavy triples look stronger than a different encoder suite would.
  • A natural extension not tested here is to use the contrastive $\lambda$ values as priors in attention-based fusion or as a guide for dropping low-weight modalities at inference, which could cut compute without repeating the full 26-combination sweep.
  • Because the evaluation keeps only complete-case patients, the reported AUROC numbers describe a selected population; the paper's own future-work plan of handling missing modalities suggests the method should be re-tested when entire modalities are absent at inference.
  • If the contrastive gains survive a leakage-free split, a practical implication the authors do not spell out is that PiCME's per-modality $\lambda$ could serve as a cheap modality-selection screen before training expensive supervised models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. PiCME proposes a systematic pipeline for contrastive representation learning across all 26 modality combinations of five clinical modalities in MIMIC-IV/MIMIC-CXR. After contrastive pre-training with InfoNCE for two modalities and a weighted One-Versus-Others loss for three or more, the authors fine-tune on in-hospital mortality and phenotyping, and they introduce a Modality-Gated LSTM that injects contrastively learned λ weights into the LSTM cell update for five-modality supervised fusion. The reported results show that three-modality contrastive models are competitive with or better than fully supervised baselines, that performance degrades with more modalities, and that mLSTM achieves the best five-modality results (IHM AUROC 76.93%, AUPRC 62.26%; phenotyping AUROC 74.36%). The paper also evaluates alignment via t-SNE and top-5 cosine similarity, demographic subgroup performance, and Integrated Gradients attribution, and it reports 10-seed averages with code.

Significance. If the central results hold, PiCME would provide a useful reference point for modality selection in MIMIC and a demonstration that contrastively learned weights can be transferred into a supervised fusion mechanism. The systematic all-combination evaluation is a genuine contribution, and the multi-seed evaluation with released code exceeds common practice in this area. The Modality-Gated LSTM idea is interesting but needs a complete specification. The significance of the paper hinges on split hygiene and on statistical support for the headline improvements, both of which require attention before the claims can be accepted.

major comments (5)
  1. [Appendix 8.3] The data-split description is ambiguous and the sample counts are placeholders ('X training samples'). The paper states that 'Contrastive learning uses the full original training set, while downstream fine-tuning is performed on a 80/10/10 split of the remaining data,' but it does not define what 'remaining data' excludes. Because the headline comparison in Table 1 is between contrastive pre-training plus fine-tuning and fully supervised training, the result is valid only if the fine-tuning test patients were never seen during contrastive pre-training. Please provide the exact MedFuse split definitions, the number of patients in each split, and a patient-ID intersection check between the contrastive pre-training set and the fine-tuning test split.
  2. [Section 5.2] The paper states that supervised baselines were trained only on the 'top three most effective' combinations, yet Table 1 draws general conclusions such as 'contrastive models remain competitive with supervised baselines' across the modality-combination sweep. The selection rule for the supervised subset is not defined, and the untested combinations cannot support those conclusions. Please either report supervised baselines for all 26 combinations or explicitly restrict the comparison claims to the combinations actually tested.
  3. [Section 3.3.4] The Modality-Gated LSTM is not fully specified. The equations for the input gate I_t, forget gate F_t, output gate O_t, and candidate memory C̃_t are omitted, and the mapping from modalities to LSTM timesteps is not described. As written, λ_t appears in the cell update, but it is unclear how a scalar λ_i per modality is broadcast over the hidden dimension and over time. Please provide the complete forward pass and state explicitly how each modality is assigned to a timestep.
  4. [Section 4.3.2, Table 2] It is not stated which model is used to compute Integrated Gradients. If the attributions are computed on the concatenated classifier rather than on the mLSTM that consumes λ, then Table 2 does not validate the gating mechanism. In addition, because both the λ weights and the attributions are estimated from the same data and model family, their agreement needs a control condition (e.g., uniform λ or randomly permuted λ) before it can be interpreted as validation.
  5. [Table 1] The main claims of improvement are based on mean differences over 10 seeds without significance tests, and several headline differences have overlapping standard deviations (e.g., IHM AUPRC 51.27±6.65 vs 62.26±2.79; phenotyping AUROC 73.43±0.52 vs 74.36±0.71). Please report paired tests or confidence intervals over the 10 seeds for the key comparisons: contrastive versus supervised, and mLSTM versus supervised.
minor comments (5)
  1. [Abstract and Section 4.2] The statement that performance declines beyond three modalities is not supported for the supervised IHM baseline in Table 1, where the five-modality AUROC (73.19) exceeds the three-modality TD+I+D AUROC (65.50). Please adjust the wording to distinguish the IHM and phenotyping tasks.
  2. [Appendix 8.3] Replace the placeholders 'X training samples' with actual counts for both tasks and specify the sizes of the training, validation, and test splits.
  3. [Appendix 8.6] 'NVIDIA GeForce RYX 3090' should be 'RTX 3090', and Table 5 still contains 'hrs' placeholders for the four- and five-modality runtime rows.
  4. [Section 3.1] The citation 'BERT [ devlin2018bert]' is malformed and should be completed; the paper also uses the phrase 'train all modality encoders from scratch' for the supervised baselines while initializing ClinicalBERT from pretrained weights, so the wording should be clarified.
  5. [Figure 2] The y-axis of the top-5 cosine similarity accuracy plot should be labeled, and the metric definition should be repeated in the caption for readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the λ-weighted mLSTM is an empirical transfer of learned weights, the Integrated Gradients check is a correlation rather than an identity, and the only self-citation is non-load-bearing.

full rationale

The paper's central chain is: (i) contrastively pre-train encoders with InfoNCE/OvO losses, (ii) freeze or fine-tune them for IHM and phenotyping, and (iii) introduce an mLSTM whose cell update is modulated by contrastively learned λ values. None of these steps makes the reported downstream AUROC/AUPRC equal to a training loss or to λ by construction. The λ values in Eq. (5) are optimized to align modalities during pre-training; the mLSTM then uses them as multiplicative gates during supervised fine-tuning. The comparison against a fully supervised baseline is an empirical evaluation of that transfer, not a tautology. The Integrated Gradients 'validation' in Table 2 compares two different quantities—contrastive alignment weights and classifier input attributions—and reports correlation, not equality; no equation in the paper shows that IG must equal λ. The only self-citation, [12], supports the statement that the OvO method 'has been successfully applied in attention mechanisms'; the method itself is independently credited to [17–19], so the self-citation is not load-bearing. Appendix 8.3's split wording ('Contrastive learning uses the full original training set, while downstream fine-tuning is performed on a 80/10/10 split of the remaining data') is ambiguous and could imply leakage, but leakage is a data-hygiene and validity concern, not a circularity of the derivation. No fitted parameter is renamed as an independent prediction, and no cited uniqueness theorem is used to forbid alternatives. The empirical claims are therefore self-contained relative to the paper's stated pipeline, with only a minor non-load-bearing self-citation preventing a score of zero.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces the Modality-Gated LSTM as a new architecture, but no new physical or conceptual entity (particle, force, dimension) is postulated. The free parameters are the modality lambda weights, the contrastive temperature, and the usual hyperparameters. The key unexamined axiom is the clean split between pre-training and fine-tuning, which the appendix describes ambiguously.

free parameters (4)
  • lambda_i (modality weights) = TD 0.297, TR 0.245, D 0.187, I 0.172, TS 0.100
    Learned during contrastive pre-training with weighted-OvO loss and softmax-normalized to sum to 1. Used in the mLSTM to gate modality contributions. These are fitted values, not derived from theory.
  • Temperature tau in InfoNCE/OvO = trainable
    Included in Equations 1, 4, and 5 as a trainable temperature parameter. Its final value is not reported.
  • Hyperparameters (learning rate, batch size, epochs) = not fully reported
    Appendix 8.2 lists ranges (learning rate 10^-2 to 10^-6, batch size 16 to 128, max 75 epochs, early stopping 15) but not the selected values for each model.
  • Class weights for weighted binary cross-entropy = not reported
    Used for IHM to handle class imbalance; the exact weights are not specified.
assumptions (4)
  • domain assumption Contrastive pre-training and supervised fine-tuning use disjoint patient splits.
    Appendix 8.3 says contrastive learning uses the full original training set while fine-tuning uses an 80/10/10 split of the remaining data, but the text is ambiguous. If this premise fails, test results are leakage-inflated. This is load-bearing for all performance comparisons.
  • domain assumption The subset of patients with complete data across all five modalities is representative of the target clinical population.
    The paper filters to patients with all five modalities and acknowledges this in Section 5.2. This complete-case analysis may not reflect real-world missing-modality scenarios.
  • ad hoc to paper The lambda weights learned during contrastive pre-training can be transferred as importance weights for supervised fusion.
    The mLSTM design assumes the contrastively learned lambdas reflect task-relevant modality importance. No theoretical justification is given, and the improvement is not compared against an ungated LSTM baseline, so this assumption is unvalidated.
  • standard math Standard LSTM equations and softmax normalization behave as expected.
    The mLSTM uses the standard LSTM cell update with an added lambda scaling term. This is a straightforward application of known recurrent network mathematics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset." pith.science (2026). https://pith.science/paper/YKWVFJBM

@misc{pith2026250703165,
  author       = {Pith},
  title        = {Pith review of: PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKWVFJBM}},
  note         = {Machine review of arXiv:2507.03165}
}
read the original abstract

Multimodal deep learning holds promise for improving clinical prediction by integrating diverse patient data, including text, imaging, time-series, and structured demographics. Contrastive learning facilitates this integration by producing a unified representation that can be reused across tasks, reducing the need for separate models or encoders. Although contrastive learning has seen success in vision-language domains, its use in clinical settings remains largely limited to image and text pairs. We propose the Pipeline for Contrastive Modality Evaluation and Encoding (PiCME), which systematically assesses five clinical data types from MIMIC: discharge summaries, radiology reports, chest X-rays, demographics, and time-series. We pre-train contrastive models on all 26 combinations of two to five modalities and evaluate their utility on in-hospital mortality and phenotype prediction. To address performance plateaus with more modalities, we introduce a Modality-Gated LSTM that weights each modality according to its contrastively learned importance. Our results show that contrastive models remain competitive with supervised baselines, particularly in three-modality settings. Performance declines beyond three modalities, which supervised models fail to recover. The Modality-Gated LSTM mitigates this drop, improving AUROC from 73.19% to 76.93% and AUPRC from 51.27% to 62.26% in the five-modality setting. We also compare contrastively learned modality importance scores with attribution scores and evaluate generalization across demographic subgroups, highlighting strengths in interpretability and fairness. PiCME is the first to scale contrastive learning across all modality combinations in MIMIC, offering guidance for modality selection, training strategies, and equitable clinical prediction.

Figures

Figures reproduced from arXiv: 2507.03165 by the authors.

Figure 1
Figure 1. Overview of the MIMIC Pipeline for Contrastive Modality Evaluation (PiCME). Panel (A): Con [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Top-5 cosine similarity accuracy for each [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. T-SNE visualization of contrastive learning through epochs. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: AUROC scores of the five-modality models across gender, ethnicity, and age group for In-Hospital [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Well-conditioned encoder Jacobians, achieved via residual paths and LeakyReLU activations, improve trimodal contrastive learning retrieval and linear-probe performance across objectives and datasets.

Reference graph

Works this paper leans on

31 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Mimic-iii, a freely accessible critical care database,

    A. E. Johnson et al., “Mimic-iii, a freely accessible critical care database,” Scientific data, vol. 3, no. 1, pp. 1–9, 2016. 12

    work page 2016

  2. [2]

    Mimic-iv, a freely accessible electronic health record dataset,

    A. E. Johnson et al., “Mimic-iv, a freely accessible electronic health record dataset,” Scientific data, vol. 10, no. 1, p. 1, 2023

    work page 2023

  3. [3]

    Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports,

    A. E. Johnson et al., “Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports,” Scientific data, vol. 6, no. 1, p. 317, 2019

    work page 2019

  4. [4]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11 975–11 986

    2023

  5. [5]

    Align before fuse: Vision and language representation learning with momentum distillation,

    J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” Advances in neural information processing systems, vol. 34, pp. 9694–9705, 2021

    work page 2021

  6. [6]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International conference on machine learning , PMLR, 2022, pp. 12 888–12 900

    work page 2022

  7. [7]

    Medclip: Contrastive learning from unpaired medical images and text,

    Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “Medclip: Contrastive learning from unpaired medical images and text,” arXiv preprint, 2022. [Online]. Available: https://arxiv.org/abs/2210. 10163

    work page 2022

  8. [8]

    A visual–language foundation model for pathology image analysis using medical twitter,

    Z. Huang, F. Bianchi, M. Yuksekgonul, T. J. Montine, and J. Zou, “A visual–language foundation model for pathology image analysis using medical twitter,”Nature medicine, vol. 29, no. 9, pp. 2307– 2316, 2023

    work page 2023

Show all 31 references
  1. [9]

    Gloria: A multimodal global-local repre- sentation learning framework for label-efficient medical image recognition,

    S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung, “Gloria: A multimodal global-local repre- sentation learning framework for label-efficient medical image recognition,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 3942–3951

    2021

  2. [10]

    Contrastive learning of medical visual representations from paired images and text,

    Y . Zhang, H. Jiang, Y . Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” in Machine learning for healthcare conference, PMLR, 2022, pp. 2–25

    2022

  3. [11]

    Multi-granularity cross-modal alignment for generalized medical visual representation learning,

    F. Wang, Y . Zhou, S. Wang, V . Vardhanabhuti, and L. Yu, “Multi-granularity cross-modal alignment for generalized medical visual representation learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 33 536–33 549, 2022

    2022

  4. [12]

    One-versus-others atten- tion: Scalable multimodal integration for biomedical data,

    M. Golovanevsky, E. Schiller, A. Nair, E. Han, R. Singh, and C. Eickhoff, “One-versus-others atten- tion: Scalable multimodal integration for biomedical data,” in Biocomputing 2025: Proceedings of the Pacific Symposium, World Scientific, 2024, pp. 580–593

    2025

  5. [13]

    Llava-next: Improved reasoning, ocr, and world knowledge,

    H. Liu et al., “Llava-next: Improved reasoning, ocr, and world knowledge,” arXiv preprint, 2024. [Online]. Available: https : / / llava - vl . github . io / blog / 2024 - 01 - 30 - llava - next/

    2024

  6. [14]

    Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images,

    N. Hayat, K. J. Geras, and F. E. Shamout, “Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images,” inMachine Learning for Healthcare Conference, PMLR, 2022, pp. 479–503

    2022

  7. [15]

    Multimodal attention-based deep learning for alzheimer’s disease diagnosis,

    M. Golovanevsky, C. Eickhoff, and R. Singh, “Multimodal attention-based deep learning for alzheimer’s disease diagnosis,”Journal of the American Medical Informatics Association, vol. 29, no. 12, pp. 2014– 2022, 2022

    2014

  8. [16]

    Learning transferable visual models from natural language supervision,

    A. Radford et al., “Learning transferable visual models from natural language supervision,” arXiv preprint, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020

    2021 arXiv

  9. [17]

    Fastsiam: Resource-efficient self-supervised learn- ing on a single gpu,

    D. Pototzky, A. Sultan, and L. Schmidt-Thieme, “Fastsiam: Resource-efficient self-supervised learn- ing on a single gpu,” in DAGM German Conference on Pattern Recognition, Springer, 2022, pp. 53– 67. 13

    2022

  10. [18]

    Contrasting multiple representations with the multi- marginal matching gap,

    Z. Piran, M. Klein, J. Thornton, and M. Cuturi, “Contrasting multiple representations with the multi- marginal matching gap,” arXiv preprint arXiv:2405.19532, 2024

    2024 arXiv

  11. [19]

    Sleepfm: Multi-modal representation learning for sleep across brain activity, ecg and respiratory signals,

    R. Thapa et al., “Sleepfm: Multi-modal representation learning for sleep across brain activity, ecg and respiratory signals,” arXiv preprint arXiv:2405.17766, 2024

    2024 arXiv

  12. [20]

    Weakly-supervised multimodal learning on mimic-cxr,

    A. Agostini et al., “Weakly-supervised multimodal learning on mimic-cxr,”arXiv preprint arXiv:2411.10356, 2024

    2024 arXiv

  13. [21]

    Integrated multimodal artificial intelligence framework for healthcare appli- cations,

    L. R. Soenksen et al., “Integrated multimodal artificial intelligence framework for healthcare appli- cations,” NPJ digital medicine, vol. 5, no. 1, p. 149, 2022

    2022

  14. [22]

    Publicly available clinical bert embeddings,

    E. Alsentzer et al., “Publicly available clinical bert embeddings,” arXiv preprint arXiv:1904.03323, 2019

    1904 arXiv

  15. [23]

    Attention-based multimodal fusion with contrast for robust clinical prediction in the face of missing modalities,

    J. Liu, D. Capurro, A. Nguyen, and K. Verspoor, “Attention-based multimodal fusion with contrast for robust clinical prediction in the face of missing modalities,” Journal of Biomedical Informatics , vol. 145, p. 104 466, 2023

    2023

  16. [24]

    Lora: Low-rank adaptation of large language models.,

    E. J. Hu et al., “Lora: Low-rank adaptation of large language models.,”ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  17. [25]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  18. [26]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  19. [27]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  20. [28]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning , PmLR, 2020, pp. 1597– 1607

    2020

  21. [29]

    Momentum contrast for unsupervised visual rep- resentation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual rep- resentation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738

    2020

  22. [30]

    Learning transferable visual models from natural language supervision,

    A. Radford et al., “Learning transferable visual models from natural language supervision,” in Inter- national conference on machine learning, PmLR, 2021, pp. 8748–8763

    2021

  23. [31]

    Axiomatic attribution for deep networks,

    M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in International conference on machine learning, PMLR, 2017, pp. 3319–3328. 14 8 Appendix 8.1 Additional Details on Predictive Tasks We evaluate our approach on two clinical tasks: in-hospital mor...

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.