Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that representing every patient data modality as an image plus text lets a single vision-text transformer outperform dedicated multi-modal baselines on MIMIC-IV mortality and phenotyping.

desk verdict The two-modality results are worth a look; the all-modality mortality headline is undercut by feeding ICD diagnoses as text, so the paper needs an ablation before the strong claim holds. read the letter →

arxiv 2501.18237 v1 pith:N4BVHBVE submitted 2025-01-30 cs.CV cs.AI

Malte Tölle , Mohamad Scharaf , Samantha Fischer , Christoph Reich , Silav Zeid , Christoph Dieterich , Benjamin Meder , Norbert Frey
show 2 more authors
Philipp Wild Sandy Engelhardt
This is my paper · ORCID
classification cs.CVcs.AI
keywords MIMIC-IVmulti-modalfusionvisiontransformerirregularlysampledtimeseriesvisualpromptengineeringin-hospitalmortalitypredictionphenotypingmedicalimaging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the usual complexity of multi-modal medical machine learning—separate encoders, interpolation schemes, and fusion modules—is largely unnecessary. It shows that if each patient data stream is rendered as an image (lab values and medications as line graphs, ECG scans as waveforms, chest X-rays as radiographs) and patient metadata is appended as text, a standard vision-text transformer can learn to predict in-hospital mortality and phenotyping from the visual layouts. On 6,175 MIMIC-IV patients, this one-pipeline model, ViTiMM, outperforms the two established two-modality baselines and reaches an AUROC of 0.922 for mortality when all four modalities plus text metadata are used. The care-relevant claim is that modeling effort shifts from architecture design to visual prompt engineering—choosing how to draw the data—which lowers the barrier to applying multi-modal learning in new clinical settings.

What carries the argument

The load-bearing object is the image-as-modality representation, which the paper calls visual prompt engineering: every non-image modality is converted into a standardized plot before any network sees it. This converts irregular sampling into spatially encoded information—gaps become blank regions, interpolation becomes line segments, observed points are marked with asterisks—so that a single pretrained vision transformer can process all modalities with identical patch embeddings. Late fusion at the feature level lets modalities be added or removed by changing only the final linear projection, which is what makes four-modality training possible without new encoders.

What would settle it

Retrain the all-modality model with the diagnosis-code text removed from the RoBERTa input, on the same test split, and compare AUROC to 0.922; if it falls to near the two-modality value of 0.875, the four-modality gain is largely outcome leakage rather than image-based fusion. A complementary check is to give MeTra and MedFuse the same text metadata and see whether the architectural gap persists.

Watch

Extended reading notes

Core claim

ViTiMM is a late-fusion vision-text transformer in which every modality enters through the same visual interface: irregularly sampled clinical measurements become standardized line graphs with observed points marked by asterisks and normal ranges drawn as red lines; medications become cumulative-dose curves grouped into ten pharmacological categories; the 12-lead ECG is plotted with each lead in its own row; and chest X-rays are used as-is. A Swin transformer pretrained on ImageNet encodes each image, a RoBERTa model encodes demographics, CXR report findings and impressions, ECG machine readings, and ICD diagnosis codes, and the resulting features are concatenated and linearly projected to a classification head. The central discovery is that this uniformity does not cost performance: for paired clinical measurements plus X-rays, ViTiMM reaches AUROC 0.875 for mortality versus 0.859 for MeTra and 0.805 for MedFuse, and adding medications and ECG raises mortality AUROC to 0.922 and phenotyping AUROC to 0.784. The paper interprets this as evidence that the information needed for these clinical predictions is preserved in the visual layout, and that explicit temporal and cross-modal modeling can be replaced by learned attention over plotted data.

Load-bearing premise

The load-bearing assumption is that including the patient's diagnosis codes in the text input does not leak the outcome; if those codes already reflect severity or the fatal event, the 0.922 mortality AUROC is inflated and the comparison to baselines without such text is unfair.

Editorial extensions

If this is right

  • Adding a new modality to ViTiMM requires only plotting it and concatenating features, so the same code base can absorb ECG, medications, or future modalities such as echocardiography without architectural changes.
  • Because blank plots are valid inputs, the method handles missing ECG data, present for only 56% of the cohort, without imputation or special masking.
  • For paired clinical measurements and X-rays, ViTiMM's AUROC of 0.875 for mortality and 0.778 for phenotyping beats both MeTra and MedFuse under comparable training, supporting the claim that visual representation preserves temporal information better than averaging or discretization.
  • On the full four-modality set, mortality AUROC reaches 0.922 and phenotyping AUROC reaches 0.784, improving over all two-modality configurations.
  • Attention maps localize predictions to specific physiological variables such as respiratory rate and heart rate, and to medication classes such as vasopressors and inotropes, offering a route to clinician-facing interpretability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the diagnosis-code text is removed from the mortality input, I would expect the all-modality AUROC to drop; the clean test of the visual-fusion claim should be run without outcome-adjacent text.
  • The same plot-everything recipe is a candidate generic strategy for sensor fusion beyond medicine, but the paper provides evidence only on MIMIC-IV, so transfer to other domains remains an open question.
  • A testable extension is to vary the plot design, such as colors, axis normalization, and marker style, and measure AUROC; if performance swings with visualization choices, visual prompt engineering is a real tuning dimension, and if not, the specific rendering matters less than the unified transformer.
  • External validation on a second ICU dataset or on a held-out hospital would tell whether the visual-layout advantage comes from the method itself or from properties of the MIMIC-IV cohort.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ViTiMM, a method that converts irregularly sampled clinical measurements, medications, and ECG waveforms into line-graph images, then combines these with chest X-ray images and unstructured text metadata for classification. All image modalities are processed by ImageNet-pretrained Swin transformers and fused late with a RoBERTa text embedding. The method is evaluated on MIMIC-IV for in-hospital mortality and phenotyping (25 diagnoses), comparing with MeTra and MedFuse on clinical measurements and chest X-rays, and reporting an extension to four modality inputs plus text. The paper claims that representing all modalities as images simplifies modeling and outperforms the two baselines, with a headline all-modality mortality AUROC of 0.922.

Significance. If the two-modality comparison (C|X) is taken at face value, the paper offers a genuinely simpler preprocessing pipeline that is competitive with or better than specialized fusion models on the same paired data: ViTiMM achieves AUROC 0.875 versus 0.859 (MeTra) and 0.805 (MedFuse) for mortality, with a similar pattern for phenotyping. The paper also releases code and model weights, which would lower the barrier for others to apply the same 'data as image' strategy. However, the central quantitative claim of the paper—the all-modality mortality AUROC of 0.922 that 'outperforms the two modality training across all methods'—is undermined by including ICD diagnosis codes as text input, which is outcome leakage for mortality. The four-modality result therefore does not currently support the paper's headline inference that visualizing more modalities improves prediction.

major comments (3)
  1. [Results, Extension to All Modalities; Methods, Patient Metadata] The all-modality mortality model includes 'the diagnoses encoded in the international classification of diseases (ICD) codes' as text input (Methods, Patient Metadata). MIMIC-IV ICD codes are assigned during the hospitalization and include conditions such as septicemia, respiratory failure, and shock that are on the causal pathway to in-hospital death. The paper itself excludes diagnoses from the phenotyping task because 'we cannot feed the patient's diagnoses as they shall be predicted from the input data' (Results, Extension to All Modalities). The same reasoning applies to mortality: ICD codes are not a legitimate input modality for this task. Consequently, the comparison of 0.922 (all modalities with ICD text) against 0.875 (C|X) does not measure the benefit of visual fusion; it measures the combined effect of adding diagnoses, demographics, CXR reports, ECG machine measurements, and medication strings. The claim that 'training with all modalities represents the benchmark' is therefore not supported. Please rerun the all-modality experiment without ICD diagnosis codes, and ideally also without the other text metadata, so that the ablation isolates the contribution of the image-based fusion.
  2. [Results, Extension to All Modalities] Even setting aside the ICD codes, the all-modality input includes text metadata (demographics, CXR findings/impressions, ECG machine measurements, medication strings) that is not available to the MeTra and MedFuse baselines. The reported 0.922 vs 0.875 gap is thus not attributable to the proposed method's image-based fusion of C, M, X, and E alone. A proper comparison requires (a) an ablation of the all-modality model without any text metadata, and (b) a baseline that receives the same text metadata through a comparable text encoder. Without such ablations, the paper's strongest claim—that adding more modalities improves performance—is confounded.
  3. [Table 1; Supplementary Tables 5 and 6] The statistical reporting is insufficiently specified. Table 1 reports a single AUROC/AUPRC/Bal.Acc. per method with no confidence intervals or number of repeated runs, while the supplementary material reports pairwise paired t-tests without stating the unit of analysis (per-patient scores? per-fold? per-seed?). A paired t-test on a single scalar AUROC per method is not meaningful, and the correct protocol would use bootstrapping or repeated training with different seeds. Please clarify the statistical procedure and report variance estimates for at least the headline comparisons.
minor comments (5)
  1. [Abstract; Table 2] The abstract states the evaluation is on '6,175 patients', but Table 2 reports 'Patients 6125'. Please correct the inconsistency.
  2. [Table 1] The rows for MeTra and MedFuse in Table 1 are labeled 'L|X', but the text consistently refers to the two-modality setup as 'C|X' (clinical measurements plus X-ray). If 'L' stands for laboratory values, it should be reconciled with 'C'; otherwise this is a typographical error.
  3. [Methods, Architecture with Vision Transformers] The paper claims to harmonize feature extraction across modalities, but the implementation uses a separate Swin transformer for each image modality, each pretrained on ImageNet. This is a legitimate design, but the phrasing 'no per-modality encoders' or 'unifies the data extraction pipeline' is too strong; the unification is at the level of architecture and preprocessing, not at the level of a shared encoder.
  4. [Discussion, Benchmark Tasks] The statement that MIMIC is the 'only publicly available' multi-modal medical dataset is too broad; datasets such as eICU with text and vitals, or other public imaging-plus-EHR collections, exist. The claim should be qualified.
  5. [Results, Interpretability] The attention visualization discussion would be more convincing if the authors reported whether the attention maps are averaged over heads/layers and how stable they are across patients; as it stands, the interpretation is anecdotal.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ViTiMM is trained on a train split and evaluated on a held-out test set, with baselines compared on the same paired data; the central claim is empirical, and the only adjacent concern (ICD-code text input for mortality) is label leakage, not circularity.

full rationale

The paper's claimed derivation chain is empirical rather than definitional. Each modality is converted into images or text, a Swin/RoBERTa model is trained on a stratified train split, and all reported AUROC/AUPRC/balanced-accuracy numbers come from a held-out test set. No quantity in the paper is defined in terms of a fitted parameter that also constitutes the result, and no fitted value is renamed as a prediction. The visual prompt engineering choices (plot formats, normalization, clipping, standardized variables) were tuned on validation data, which is standard model selection rather than circular reasoning. The line-graph representation is explicitly adopted from an external prior work (Li et al. [29]) and the contribution is presented as an extension to multimodal data, so there is no ansatz smuggled in through a self-citation chain. The only self-citation in the references (Ref. 46, a federated learning cardiac CT paper) is not load-bearing for the central method or results. The ICD-code text input for the mortality task raises a legitimate benchmark-integrity concern, because discharge diagnosis codes can encode conditions on the pathway to in-hospital death; however, this is outcome leakage or an unfair comparison, not circularity. The model output is not equal to its input by construction, and the paper's own phenotyping caveat ('we cannot feed the patient's diagnoses as they shall be predicted from the input data') shows the authors recognized the analogous issue for phenotyping without making the mortality result definitionally dependent on its own label. No circular step meets the evidence bar of exhibiting a specific reduction of a claimed result to its inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. The free parameters are all design choices in the visualization pipeline, and the axioms are domain assumptions about the validity of image-based representation and the fairness of the evaluation. The most consequential is the temporal availability of ICD codes, which directly affects the headline mortality result.

free parameters (4)
  • Cumulative medication dosage global max = Unstated, computed from full cohort
    Methods, Medications: 'Afterwards all cumulative amounts are normalized between 0 and 1 with the maximum amount across all patients.' This data-derived scale is a cohort-level constant that the input plots depend on.
  • Medication plot clipping quantile = 95th percentile
    Methods, Medications: 'to remove outliers we clip the cumulative values at the 95th quartile'. This hand-chosen threshold shapes the visual input.
  • Clinical variable set = 36 variables
    Methods, Clinical Measurements: 'We identified 36 different clinical variables in consultation with expert physicians.' The choice of which variables to plot is manual and affects the input representation.
  • Medication category grouping = 10 categories
    Methods, Medications: 'we first grouped the medications into ten categories such as e.g. beta-blockers or antiarrhythmics.' The grouping is a design choice.
assumptions (4)
  • domain assumption A vision transformer pretrained on ImageNet can extract clinically meaningful features from synthetic line plots
    The entire method relies on transfer from natural images to medical plots. No evidence is provided that this transfer works beyond the reported benchmark results, which are the conclusion, not an independent justification.
  • domain assumption Linear interpolation between irregularly sampled observations preserves the information needed for mortality and phenotyping prediction
    The paper plots observations connected by straight lines and marks observations with '*'. This is an assumption that the visual representation retains clinically relevant trends; the paper does not compare against other interpolation schemes.
  • domain assumption In-hospital mortality labels and ICD diagnosis codes are temporally available before the prediction point
    For the mortality task, ICD codes are fed as text. The paper does not establish whether these diagnoses were assigned during the first 48 hours or later, which is critical for the leak concern.
  • domain assumption Evaluating all methods on paired C and X samples is a fair representation of clinical performance
    The paper restricts comparison to paired samples for 'maximal comparability', but this omits the partial-data setting where MedFuse was originally designed to excel, potentially biasing the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers." pith.science (2026). https://pith.science/paper/N4BVHBVE

@misc{pith2026250118237,
  author       = {Pith},
  title        = {Pith review of: Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N4BVHBVE}},
  note         = {Machine review of arXiv:2501.18237}
}
read the original abstract

A patient undergoes multiple examinations in each hospital stay, where each provides different facets of the health status. These assessments include temporal data with varying sampling rates, discrete single-point measurements, therapeutic interventions such as medication administration, and images. While physicians are able to process and integrate diverse modalities intuitively, neural networks need specific modeling for each modality complicating the training procedure. We demonstrate that this complexity can be significantly reduced by visualizing all information as images along with unstructured text and subsequently training a conventional vision-text transformer. Our approach, Vision Transformer for irregular sampled Multi-modal Measurements (ViTiMM), not only simplifies data preprocessing and modeling but also outperforms current state-of-the-art methods in predicting in-hospital mortality and phenotyping, as evaluated on 6,175 patients from the MIMIC-IV dataset. The modalities include patient's clinical measurements, medications, X-ray images, and electrocardiography scans. We hope our work inspires advancements in multi-modal medical AI by reducing the training complexity to (visual) prompt engineering, thus lowering entry barriers and enabling no-code solutions for training. The source code will be made publicly available.

Figures

Figures reproduced from arXiv: 2501.18237 by the authors.

Figure 1
Figure 1. Example patient journey in the first 48 hours in the intensive care unit for [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The four modalities from the MIMIC-IV dataset used in our model. We tested various formats [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Interpretability with Visualizing Attention Maps for in-hospital mortality prediction for [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Multi-modal medical vision transformer with arbitrary modalities as images. Each modality is [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Scientific Human-Agent Reproduction Pipeline

    hep-ph 2026-04 unverdicted novelty 6.0 of 10

    SHARP is a human-AI collaboration pipeline for reproducing scientific analyses, demonstrated by recreating a jet classification task from a particle physics paper.

  2. CXR-TFT: Multi-Modal Temporal Fusion Transformer for Predicting Chest X-ray Trajectories

    cs.LG 2025-07 reject novelty 5.0 of 10

    CXR-TFT predicts future chest X-ray embeddings hour-by-hour from clinical time series and prior X-rays, claiming 95% accuracy for abnormal findings 12 hours before the next scan.

Reference graph

Works this paper leans on

48 extracted references · 23 canonical work pages · cited by 2 Pith papers

  1. [1]

    Nature Medicine 28(9), 1773–1784 (2022)

    Acosta, J.N., Falcone, G.J., Rajpurkar, P., Topol, E.J.: Multimodal biomedical ai. Nature Medicine 28(9), 1773–1784 (2022). https://doi.org/10.1038/s41591-022-01981-2

    doi:10.1038/s41591-022-01981-2 2022

  2. [2]

    Nature Biomedical Engineering (2024)

    Bluethgen, C., Chambon, P., Delbrouck, J.B., van der Sluijs, R., Po lacin, M., Zambrano Chaves, J.M., Abra- ham, T.M., Purohit, S., Langlotz, C.P., Chaudhari, A.S.: A vision–language foundation model for the gen- eration of realistic chest x-ray images. Nature Biomedical Engineering (2024). https://doi.org/10.1038/ s41551-024-01246-y

    work page 2024

  3. [3]

    Scientific Reports 8(1), 6085 (2018)

    Che, Z., Purushotham, S., Cho, K., Sontag, D., Liu, Y.: Recurrent neural networks for multivariate time series with missing values. Scientific Reports 8(1), 6085 (2018). https://doi.org/10.1038/s41598-018-24271-9

    doi:10.1038/s41598-018-24271-9 2018

  4. [4]

    Unconditional Latent Diffusion Models Memorize Patient Imaging Data: Implications for Openly Sharing Synthetic Data

    Dar, S.U.H., Seyfarth, M., Ayx, I., Papavassiliu, T., Schoenberg, S.O., Siepmann, R.M., Laqua, F.C., Kah- mann, J., Frey, N., Baeßler, B., Foersch, S., Truhn, D., Kather, J.N., Engelhardt, S.: Unconditional la- tent diffusion models memorize patient imaging data: Implications for openly sharing synthetic data (2024). https://doi.org/10.48550/arXiv.2402.01054

    work page Pith review arXiv doi:10.48550/arxiv.2402.01054 2024

  5. [5]

    Biometrics 44(3), 837–845 (1988)

    DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L.: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3), 837–845 (1988)

    work page 1988

  6. [6]

    In: Conference on Computer Vision and Pattern Recognition (CVPR)

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. In: Conference on Computer Vision and Pattern Recognition (CVPR). pp. 248–255 (2009). https: //doi.org/10.1109/CVPR.2009.5206848

    arXiv 2009

  7. [7]

    In: International Conference on Learning Representations (2021), https://openreview.net/forum?id=YicbFdNTTy

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Trans- formers for image recognition at scale. In: International Conference on Learning Representations (2021), https://openreview.net/forum?id=YicbFdNTTy

    work page 2021

  8. [8]

    PLoS computational biology 17(8), e1009224 (2021)

    Duan, R., Gao, L., Gao, Y., Hu, Y., Xu, H., Huang, M., Song, K., Wang, H., Dong, Y., Jiang, C., et al.: Eval- uation and comparison of multi-omics data integration methods for cancer subtyping. PLoS computational biology 17(8), e1009224 (2021)

    work page 2021

Show all 48 references
  1. [9]

    Circulation 101(23), e215–e220 (2000)

    Goldberger, A.L., Amaral, L.A.N., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.K., Stanley, H.E.: PhysioBank, PhysioToolkit, and PhysioNet: Components of a New Research Resource for Complex Physiologic Signals. Circulation 101(23), e...

    2000

  2. [10]

    https://doi.org/10.13026/ef48-v217, physioNet Fusion of Patient Data Across Modalities and Irregular Intervals 15

    Gow, B., Pollard, T., Greenbaum, N., Moody, B., Johnson, A., Herbst, E., Waks, J.W., Eslami, P., Chaudhari, A., Carbonati, T., Berkowitz, S., Mark, R., Horng, S.: MIMIC-IV-ECHO: Echocardiogram Matched Subset (version 0.1) (2023). https://doi.org/10.13026/ef48-v217, physioNet F...

    2023 doi

  3. [11]

    https://doi.org/ 10.13026/4nqg-sb35

    Gow, B., Pollard, T., Nathanson, L.A., Johnson, A., Moody, B., Fernandes, C., Greenbaum, N., Waks, J.W., Eslami, P., Carbonati, T., Chaudhari, A., Herbst, E., Moukheiber, D., Berkowitz, S., Mark, R., Horng, S.: MIMIC-IV-ECG: Diagnostic Electrocardiogram Matched Subset (version...

    2023 doi

  4. [12]

    In: Machine Learning for Health symposium

    Gupta, M., Gallamoza, B., Cutrona, N., Dhakal, P., Poulain, R., Beheshti, R.: An Extensive Data Processing Pipeline for MIMIC-IV. In: Machine Learning for Health symposium. vol. 193, pp. 311–325 (2022), https: //proceedings.mlr.press/v193/gupta22a.html

    2022

  5. [13]

    Critical Care Medicine 38(1), 65–71 (Jan 2010)

    Halpern, N.A., Pastores, S.M.: Critical care medicine in the United States 2000-2005: an analysis of bed numbers, occupancy rates, payer mix, and costs. Critical Care Medicine 38(1), 65–71 (Jan 2010). https: //doi.org/10.1097/CCM.0b013e3181b090d0

    2000 doi

  6. [14]

    In: Proceedings of the 7th Machine Learning for Healthcare Conference

    Hayat, N., Geras, K.J., Shamout, F.E.: MedFuse: Multi-modal Fusion with Clinical Time-Series Data and Chest X-Ray Images. In: Proceedings of the 7th Machine Learning for Healthcare Conference. vol. 182, pp. 479–503 (2022), https://proceedings.mlr.press/v182/hayat22a.html

    2022

  7. [15]

    In: Conference on Computer Vision and Pattern Recognition (CVPR)

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

    2016 doi

  8. [16]

    Neural Computation 9(8), 1735–1780 (1997)

    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    1997 doi

  9. [18]

    npj Digital Medicine 3(1), 136 (2020)

    Huang, S.C., Pareek, A., Seyyedi, S., Banerjee, I., Lungren, M.P.: Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. npj Digital Medicine 3(1), 136 (2020). https://doi.org/10.1038/s41746-020-00341-z

    2020 doi

  10. [19]

    Scientific reports 10(1), 22147 (2020)

    Huang, S.C., Pareek, A., Zamanian, R., Banerjee, I., Lungren, M.P.: Multimodal fusion with deep neural networks for leveraging ct imaging and electronic health record: a case-study in pulmonary embolism detection. Scientific reports 10(1), 22147 (2020). https://doi.org/10.1038...

    2020 doi

  11. [20]

    Clinical nuclear medicine 44(12), 956–960 (2019)

    Hyun, S.H., Ahn, M.S., Koh, Y.W., Lee, S.J.: A machine-learning approach using pet-based radiomics to predict the histological subtypes of lung cancer. Clinical nuclear medicine 44(12), 956–960 (2019). https: //doi.org/10.1097/RLU.0000000000002810

    2019 doi

  12. [21]

    Journal of the American Medical Informatics Association 29(6), 1060–1068 (2022)

    Jabbour, S., Fouhey, D., Kazerooni, E., Wiens, J., Sjoding, M.W.: Combining chest x-rays and electronic health record (ehr) data using machine learning to diagnose acute respiratory failure. Journal of the American Medical Informatics Association 29(6), 1060–1068 (2022)

    2022

  13. [22]

    In: International Conference on Learning Representations (ICLR) (2021)

    Jaegle, A., Borgeaud, S., Alayrac, J.B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., H´ enaff, O., Botvinick, M.M., Zisserman, A., Vinyals, O., Carreira, J.: Perceiver io: A general architecture for structured inputs and outputs. In: I...

    2021 doi arXiv

  14. [23]

    In: International Conference on Machine Learning (ICML) (2021).https://doi.org/ 10.48550/2103.03206

    Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., Carreira, J.: Perceiver: General perception with iterative attention. In: International Conference on Machine Learning (ICML) (2021).https://doi.org/ 10.48550/2103.03206

    2021 doi arXiv

  15. [24]

    https://doi.org/10.13026/4jqj-jw95

    Johnson, A., Pollard, T., Mark, R., Berkowitz, S., Horng, S.: MIMIC-CXR Database (version 2.1.0) (2024). https://doi.org/10.13026/4jqj-jw95

    2024 doi

  16. [25]

    Scientific Data 10(1), 1 (2023)

    Johnson, A.E.W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T.J., Hao, S., Moody, B., Gow, B., Lehman, L.w.H., Celi, L.A., Mark, R.G.: Mimic-iv, a freely accessible electronic health record dataset. Scientific Data 10(1), 1 (2023). https://doi.org/...

    2023 doi

  17. [26]

    Nature Scientific Data 6, 317 (2019)

    Johnson, A.E.W., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., ying Deng, C., Mark, R.G., Horng, S.: MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Nature Scientific Data 6, 317 (2019). https://doi.org/10.1...

    2019 doi

  18. [27]

    Nature Scientific Reports 13(1), 10666 (2023)

    Khader, F., Kather, J.N., M¨ uller-Franzes, G., Wang, T., Han, T., Arasteh, S.T., Hamesch, K., Bressem, K., Haarburger, C., Stegmaier, J., Kuhl, C., Nebelung, S., Truhn, D.: Medical transformer for multimodal survival prediction in intensive care: integration of imaging and no...

    2023 doi

  19. [28]

    npj Digital Medicine 5(1), 171 (2022)

    Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., Luo, Y.: Multimodal machine learning in precision health: A scoping review. npj Digital Medicine 5(1), 171 (2022). https: //doi.org/10.1038/s41746-022-00712-8 16 M. T¨ olle et al

    2022 doi

  20. [29]

    In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS) (2023), https: //openreview.net/forum?id=ZmeAoWQqe0

    Li, Z., Li, S., Yan, X.: Time Series as Images: Vision Transformer for Irregularly Sampled Time Series. In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS) (2023), https: //openreview.net/forum?id=ZmeAoWQqe0

    2023

  21. [30]

    In: Proceedings of the 1st Machine Learning for Healthcare Conference

    Lipton, Z.C., Kale, D., Wetzel, R.: Directly Modeling Missing Data in Sequences with RNNs: Improved Classification of Clinical Time Series. In: Proceedings of the 1st Machine Learning for Healthcare Conference. vol. 56, pp. 253–270 (18–19 Aug 2016), https://proceedings.mlr.pre...

    2016

  22. [32]

    MICCAI (2023)

    Liu, J., Zhang, Y., Chen, J.N., Xiao, J., Lu, Y., Landman, B.A., Yuan, Y., Yuille, A., Tang, Y., Zhou, Z.: CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection. MICCAI (2023)

    2023

  23. [33]

    https://doi.org/10.48550/ arXiv.1907.11692

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019). https://doi.org/10.48550/ arXiv.1907.11692

    2019 doi arXiv

  24. [35]

    In: International Health Informatics Symposium

    Marlin, B.M., Kale, D.C., Khemani, R.G., Wetzel, R.C.: Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In: International Health Informatics Symposium. p. 389–398 (2012). https://doi.org/10.1145/2110363.2110408

    2012

  25. [36]

    In: Advances in Neural Information Processing Systems (NeuRIPS) (2023)

    McDermott, M.B.A., Nestor, B., Argaw, P., Kohane, I.: Event stream gpt: A data pre-processing and mod- eling library for generative, pre-trained transformers over continuous-time sequences of complex events. In: Advances in Neural Information Processing Systems (NeuRIPS) (2023...

    2023

  26. [37]

    https://doi.org/10.48550/arXiv.2408.05178

    McKeen, K., Oliva, L., Masood, S., Toma, A., Rubin, B., Wang, B.: ECG-FM: An Open Electrocardiogram Foundation Model (2024). https://doi.org/10.48550/arXiv.2408.05178

    2024 doi arXiv

  27. [38]

    Nature 616(7956), 259–265 (2023)

    Moor, M., Banerjee, O., Hossein Abad, Z.S., Krumholz, H.M., Leskovec, J., Topol, E.J., Rajpurkar, P.: Foundation models for generalist medical artificial intelligence. Nature 616(7956), 259–265 (2023). https: //doi.org/10.1038/s41586-023-05881-4

    2023 doi

  28. [39]

    Information Fusion 25, 72–84 (2015)

    Nejati, M., Samavi, S., Shirani, S.: Multi-focus image fusion using dictionary-based sparse representation. Information Fusion 25, 72–84 (2015). https://doi.org/10.1016/j.inffus.2014.10.004

    2015 doi

  29. [40]

    Scientific reports 9(1), 1103 (2019)

    Nie, D., Lu, J., Zhang, H., Adeli, E., Wang, J., Yu, Z., Liu, L., Wang, Q., Wu, J., Shen, D.: Multi-channel 3d deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages. Scientific reports 9(1), 1103 (2019). https://doi.org/10.103...

    2019 doi

  30. [41]

    In: International Conference on Machine Learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763 (2021)

    2021

  31. [42]

    IEEE ACCESS 12, 174107–174121 (2024)

    Rasekh, A., Heidari, R., Hosein Haji Mohammad Rezaie, A., Sharifi Sedeh, P., Ahmadi, Z., Mitra, P., Nejdl, W.: Robust fusion of time series and image data for improved multimodal clinical prediction. IEEE ACCESS 12, 174107–174121 (2024). https://doi.org/10.1109/ACCESS.2024.3497668

    2024

  32. [43]

    Nature Communications 11(1), 1760 (2020)

    Ribeiro, A.H., Ribeiro, M.H., Paix˜ ao, G.M.M., Oliveira, D.M., Gomes, P.R., Canazart, J.A., Ferreira, M.P.S., Andersson, C.R., Macfarlane, P.W., Jr., W.M., Sch¨ on, T.B., Ribeiro, A.L.P.: Automatic diagnosis of the 12-lead ecg using a deep neural network. Nature Communication...

    2020 doi

  33. [44]

    Ross, A., Willson, V.L.: Paired Samples T-Test, pp. 17–19. SensePublishers, Rotterdam (2017). https:// doi.org/10.1007/978-94-6351-086-8_4

    2017 doi

  34. [45]

    Journal of Digital Imaging 35(5), 1308–1325 (2022)

    Tawfik, N., Elnemr, H.A., Fakhr, M., Dessouky, M.I., El-Samie, F.E.A.: Multimodal medical image fusion using stacked auto-encoder in nsct domain. Journal of Digital Imaging 35(5), 1308–1325 (2022). https: //doi.org/10.1007/s10278-021-00554-y

    2022 doi

  35. [46]

    npj Digital Medicine (2025)

    T¨ olle, M., Garthe, P., Scherer, C., Seliger, J.M., Leha, A., Kr¨ uger, N., Simm, S., Martin, S., Eble, S., Kelm, H., Bednorz, M., Andr´ e, F., Bannas, P., Diller, G., Frey, N., Groß, S., Hennemuth, A., Kaderali, L., Meyer, A., Nagel, E., Orwat, S., Seiffert, M., Friede, T., ...

    2025

  36. [47]

    https://doi.org/10.48550/arXiv.2302.13971

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozi` ere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: LLaMA: Open and Efficient Foundation Language Models (2023). https://doi.org/10.48550/arXiv.2302.13971

    2023 doi arXiv

  37. [48]

    In: Advances in Neural Information Processing Systems (NeuRIPS)

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems (NeuRIPS). vol. 30 (2017). https://doi.org/10.48550/arXiv.1706.03762

    2017 doi arXiv

  38. [49]

    Heliyon 10(5), e26772 (2024)

    Wang, Y., Yin, C., Zhang, P.: Multimodal risk prediction with physiological signals, medical images and clinical notes. Heliyon 10(5), e26772 (2024). https://doi.org/10.1016/j.heliyon.2024.e26772

    2024 doi

  39. [50]

    In: International Conference on Learning Representations (ICLR) (2022)

    Zhang, X., Zeman, M., Tsiligkaridis, T., Zitnik, M.: Graph-guided network for irregularly sampled mul- tivariate time series. In: International Conference on Learning Representations (ICLR) (2022). https: //doi.org/10.48550/arXiv.2110.05357

    2022 doi arXiv

  40. [51]

    In: International Joint Conferences on Artificial Intelligence Organization (IJCAI)

    Zhang, Y., Yang, X., Ivy, J., Chi, M.: Attain: Attention-based time-aware lstm networks for disease progression modeling. In: International Joint Conferences on Artificial Intelligence Organization (IJCAI). pp. 4369–4375 (2019). https://doi.org/10.24963/ijcai.2019/607 18 M. T¨...

    2019 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.