Multimodal Federated Learning With Missing Modalities through Feature Imputation Network

Aavash Chhetri; Binod Bhattarai; Georgios Leontidis; Pranav Poudel; Prashnna Gyawali

REVIEW 3 major objections 4 minor 26 references

Multimodal Federated Learning With Missing Modalities through Feature Imputation Network

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Feature imputation beats generative models for missing modalities

desk verdict A useful, practical feature-level imputation method for missing modalities in multimodal FL, but the key real-world claim rests on an unvalidated cross-dataset transfer and the paper needs error bars, a mean-imputation baseline, and a corrected gradient-flow statement. read the letter →

arxiv 2505.20232 v1 pith:2CLXPCZU submitted 2025-05-26 cs.LG cs.CV

Pranav Poudel , Aavash Chhetri , Prashnna Gyawali , Georgios Leontidis , Binod Bhattarai This is my paper · ORCID

classification cs.LGcs.CV

keywords multimodalfederatedlearningmissingmodalitiesfeatureimputationbottleneckfeatureschestX-raymedicalimagingaggregationfeature-level

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal federated learning works best when every client has all modalities, but in healthcare many sites have only X-rays while others have only reports. The paper's central claim is that the missing modality can be reconstructed at the bottleneck feature level: a lightweight imputation network, trained in the same federated rounds on the paired examples held by multimodal clients, learns to translate the available modality's feature vector into the missing one's feature vector. Uni-modal clients then use these imputed features instead of raw data imputation. Evaluated on three chest X-ray datasets in both homogeneous and heterogeneous client splits, this approach consistently outperforms zero-filling, uniform filling, and a federated generative baseline, and it approaches a method that borrows public data. If the claim holds, federated systems with only a few complete-modality clients can still train strong multimodal models without generating raw text or images.

What carries the argument

The central object is the Feature Imputation Network (FIN), a lightweight six-layer Transformer decoder with four attention heads, which acts as a conditional translator between modality-specific bottleneck features. It is the device that converts the problem of missing modalities from a high-dimensional input-space generation task into a low-dimensional feature-space regression task, and it is trained only at multimodal clients, then federated to uni-modal clients for inference. The imputation network makes the approach lightweight and keeps the missing-modality encoder's features untouched by gradient updates during imputed training, which the paper credits for stability compared with token-space generation.

What would settle it

Run the heterogeneous 8:0:2 configuration but source the image-only clients from a dataset with a markedly different imaging protocol or patient population (for example, a non-chest X-ray set), and compare macro AUC against zero-filling; if the latent features are not aligned across institutions, FIN's imputed features should fall to or below the zero-filling baseline.

Watch

Extended reading notes

Core claim

The paper establishes that feature-level imputation is a viable and efficient substitute for input-level generation in multimodal federated learning. Concretely, with image and text as the two modalities, the Feature Imputation Network (FIN) learns maps Φ_T: z_I → z_T and Φ_I: z_T → z_I, where z are 256-dimensional L2-normalized bottleneck features from pretrained ResNet-50 and BERT encoders. FIN is a 6-layer Transformer decoder (6.3M parameters) trained on paired feature pools at multimodal clients by minimizing MSE with the ground-truth missing feature, then aggregated with FedAvg. At image-only or text-only clients, FIN runs in inference mode to synthesize the missing feature before fusion and classification. Across configurations, FIN improves macro AUC over zero-filling and uniform-filling in every reported homogeneous and heterogeneous split, and it beats a federated R2Gen report generator, with the largest gap (about 10 AUC points) in the heterogeneous 8:0:2 setting.

Load-bearing premise

The load-bearing premise is that a feature imputation network trained on paired image-text features from the multimodal clients (NIH Open-I) transfers to image-only clients from a different institution (CheXpert) whose data distribution differs, which requires the shared encoders to produce aligned latent feature spaces across datasets.

Editorial extensions

If this is right

With only two multimodal clients in an eight-client image-only federation, feature imputation reaches AUC comparable to a six-client all-multimodal setting, so complete-modality data can be leveraged far more efficiently.
Because FIN has about 6.3M parameters and 6.3M FLOPs versus R2Gen's roughly 59.7M parameters and 94G FLOPs, communication per round drops by about an order of magnitude and per-inference compute by about three orders of magnitude.
The method requires no external public dataset for imputation, removing the dependency on curated real data that prior public-data-based methods need.
Training signal from imputed features updates only the classification head, not the missing modality's encoder, which the paper credits for the stability of these methods compared with generative token-space imputation.
The approach generalizes across client configurations with different ratios of image-only, text-only, and multimodal clients, including settings with only two multimodal clients.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same bottleneck-imputation principle should transfer to other modality pairs (for example, time-series vital signs paired with imaging), provided at least some clients hold paired samples; the only requirement is a shared encoder whose latent space is aligned across sites.
A natural stress test is to replace the shared pretrained encoders with locally fine-tuned encoders in the heterogeneous setup; if latent spaces drift, FIN's imputed features should degrade, which would reveal how much of the gain depends on encoder alignment.
Performance in the heterogeneous setup likely depends on how well the NIH Open-I image features (used for training FIN) cover the CheXpert image distribution; a targeted experiment varying the ratio of multimodal to uni-modal clients while holding total data fixed would quantify this sensitivity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hi [Name],

Straight to it: this is a well-motivated, practical paper on handling missing modalities in multimodal federated learning. The idea is to train a small feature imputation network (FIN) at the clients that have both modalities, then use it to reconstruct bottleneck features for clients missing one modality. It is lightweight (6.3M params vs R2Gen's 59.7M) and the authors provide a first direct comparison of feature-level versus input-level imputation in an FL setting. That comparison is the paper's genuine contribution, and the code is out.

The homogeneous results are consistent and clearly beat zero-filling, uniform-filling, and the trained-from-scratch federated R2Gen. The authors also fairly note that CAR-MFL is not directly comparable because it leans on public data. The computational complexity table is a nice addition.

Now the soft spots, in order of severity.

First, the heterogeneous setup is under-evidenced. In the 8:0:2 real-world setup, FIN is trained only on paired NIH image/text features and then applied, without adaptation, to CheXpert image features. The paper gives no diagnostic that CheXpert features lie on the NIH manifold FIN saw. The t-SNE is computed on MIMIC validation features, not on the CheXpert features FIN actually consumes. And there is no mean-imputation baseline. So the 5.18 AUC advantage over zero-filling in that setting could in principle come from FIN learning a stable, text-like vector plus the classifier adapting, rather than genuine per-sample reconstruction. The authors need to show feature-space alignment, or at least compare against a mean-text imputation.

Second, the claim in Section 3.4 that feature imputation methods 'influence only the classification head' is contradicted by Equations 5 and 6: gradients flow through the available encoder, since the imputed feature is a function of it. This is a real error, though not fatal; it just needs to be corrected.

Third, no standard deviations or significance tests are reported, despite only three seeds. For a paper whose headline is 'consistently improve', that is a gap.

The central idea holds up in the homogeneous case; the heterogeneous claim is suggestive but not proven. This paper deserves a serious referee. A revision with mean-imputation baseline, feature alignment analysis, significance testing, and a fix to the gradient claim would make it solid.

Worth bringing to reading group, and I'd probably cite it when discussing feature-level methods in FL.

My recommendation: send to peer review.

Referee Report

3 major / 4 minor

Summary. This paper addresses missing modalities in multimodal federated learning by proposing the Feature Imputation Network (FIN), a lightweight transformer-based module that reconstructs the bottleneck feature vector of a missing modality from the available modality's feature vector. FIN is trained only at multimodal clients on paired image/text features and used for inference at unimodal clients, with FedAvg aggregation. The authors evaluate on MIMIC-CXR, NIH Open-I, and CheXpert in homogeneous and heterogeneous client configurations, comparing against zero-filling, uniform-filling, a federated R2Gen generative baseline, and the retrieval-based CAR-MFL. Results in Tables 1 and 2 show consistent AUC improvements over the imputation baselines and competitive performance with CAR-MFL, along with a large reduction in communication and compute cost relative to R2Gen (Table 3).

Significance. The paper's main contribution is a practical, low-complexity alternative to generative imputation for missing modalities in multimodal federated learning, and it is, to the authors' knowledge, the first direct feature-level versus input-level imputation comparison in this setting. If the empirical claims hold, the approach is attractive because it avoids public-data dependence and reduces per-round communication by roughly 10x and inference cost by roughly 1000x compared with the generative baseline. The authors provide code and implementation details, which supports reproducibility. However, the evidence for the key real-world heterogeneous scenario is incomplete: the cross-dataset transfer of FIN is not validated, the reported means lack error bars, and the explanatory claim in Section 3.4 is internally inconsistent with the gradient flow in Equations 5 and 6. These issues are fixable with additional experiments and analyses.

major comments (3)

[Section 3.4, Equations 5 and 6] The explanation in Section 3.4 that feature imputation methods 'influence only the classification head' is contradicted by Equations 5 and 6. In Equation 5, the loss is a function of the image encoder fI_e both directly and through the imputation path Phi_T(fI_e(XI)); since Phi_T is used for inference but its input is the output of fI_e, gradients with respect to fI_e flow through the imputation network during backpropagation. The image encoder is therefore affected by the imputed feature, not only the classification head. The authors should either remove or qualify this claim, or explicitly use a stop-gradient operation before Phi_T if that was the intended design.
[Section 3.1 and Section 2.2] In the heterogeneous 8:0:2 setup, FIN is trained exclusively on paired NIH Open-I features and then applied to CheXpert image-only clients, but no evidence is provided that CheXpert image features lie on the NIH feature manifold. The shared image encoder is updated by CheXpert clients with the task loss of Equation 5, so its output distribution can drift away from the distribution on which FIN was trained. The t-SNE in Figure 3 is computed on MIMIC validation features, not on the CheXpert features that FIN actually consumes in the heterogeneous setup. The authors should add source-target feature alignment diagnostics (e.g., distance between CheXpert and NIH image feature distributions, or per-client FIN adaptation) and include a constant/mean-text baseline to rule out that the 5.18 AUC gain over zero-filling in Table 1 comes from a stable bias rather than genuine per-sample reconstruction.
[Tables 1 and 2, Section 3.2] Tables 1 and 2 report only the mean macro AUC over three random seeds, with no standard deviations, confidence intervals, or significance tests. Given the small number of seeds, the claim in Section 3.4 that Feature Imputation 'significantly outperforms' the baselines is not supported. Please report per-seed results or standard deviations and, if feasible, paired significance tests across seeds.

minor comments (4)

[Table 1] The header 'Heteregeneous' contains a typo and should read 'Heterogeneous'.
[Table 3] The table reports total FLOPs but labels the column 'FLOPS' (Floating Point Operations Per Second); please use 'FLOPs' and clarify that the values are per-forward-pass operations, not a rate.
[Section 2.2] The architecture description '6-layer Transformer decoder with n=4 heads and 1024 feed-forward dimensions' leaves 'n' undefined; please state explicitly that there are 6 layers and 4 attention heads.
[Section 3.4] The statement 'the performance gap of nearly 10%' is ambiguous; in the homogeneous 8:0:2 setting the gap between R2Gen and the proposed method is 8.84 AUC points, while in the heterogeneous 8:0:2 setting it is 10.62 points. Please specify which comparison is meant.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the feature imputation network is trained on paired bottleneck features and evaluated on an independent test set; self-citations are present but not load-bearing.

full rationale

The paper's central derivation is an empirical comparison rather than a chain that reduces to its own inputs. FIN is trained in Section 2.2 by minimizing MSE in Eq. 4 between predicted and ground-truth bottleneck features z_hat_T = Phi_T(z_I) and z_T, using paired image-text features pooled from multimodal clients. This target is an auxiliary representation, not the final classification label, and the downstream classifier is trained on labels from a held-out MIMIC-CXR split. No equation in the manuscript defines a parameter in terms of the reported AUC values, and no result is imported from the authors' prior CAR-MFL work beyond the experimental setup and baseline comparison. The self-citations (refs. 15, 19, 21) are normal and non-load-bearing: ref. 15 is used for dataset partitioning and as a baseline, not to justify the proposed method's mechanism. The cross-dataset transfer concern raised by the skeptical reading is a real external-validity risk (FIN trained on NIH Open-I features is applied to CheXpert features without an alignment diagnostic), but that is an assumption about distribution shift, not circularity. Likewise, the absence of a constant-filling baseline is an experimental omission, not a constructed equivalence. The claim that feature-level imputation outperforms input-level generation is supported by independent test-set evaluation with three seeds, so the paper is self-contained with respect to its main comparison.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim does not depend on any fitted constants or invented scientific entities. The only learned quantities are the weights of the imputation network and the classification head, which are trained on multimodal clients and evaluated on an independent test set. The key unstated costs are the domain-transfer and averaged-network assumptions listed above.

assumptions (3)

domain assumption The feature imputation network trained on paired image-text features from multimodal clients transfers to unimodal clients from other datasets.
In the heterogeneous setup (Section 3.1), FIN is trained on NIH Open-I pairs but used to impute text features for CheXpert image-only clients and evaluated on MIMIC-CXR. This requires a shared latent feature space across datasets.
domain assumption Pretrained ResNet-50 and BERT-base encoders produce feature spaces adequate for MSE-based feature translation.
The method fixes the encoders and trains a Transformer to map between their outputs (Section 3.2). If the pretrained features are not informative enough for the medical domain, the imputation will fail.
domain assumption Averaging imputation networks from multimodal clients with FedAvg yields a usable global imputation function.
The server aggregates imputation networks from multimodal clients (Section 2.2), which assumes that model averaging preserves the image-to-text and text-to-image mappings. This is standard in FL but not theoretically guaranteed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multimodal Federated Learning With Missing Modalities through Feature Imputation Network." pith.science (2026). https://pith.science/paper/2CLXPCZU

@misc{pith2026250520232,
  author       = {Pith},
  title        = {Pith review of: Multimodal Federated Learning With Missing Modalities through Feature Imputation Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2CLXPCZU}},
  note         = {Machine review of arXiv:2505.20232}
}

read the original abstract

Multimodal federated learning holds immense potential for collaboratively training models from multiple sources without sharing raw data, addressing both data scarcity and privacy concerns, two key challenges in healthcare. A major challenge in training multimodal federated models in healthcare is the presence of missing modalities due to multiple reasons, including variations in clinical practice, cost and accessibility constraints, retrospective data collection, privacy concerns, and occasional technical or human errors. Previous methods typically rely on publicly available real datasets or synthetic data to compensate for missing modalities. However, obtaining real datasets for every disease is impractical, and training generative models to synthesize missing modalities is computationally expensive and prone to errors due to the high dimensionality of medical data. In this paper, we propose a novel, lightweight, low-dimensional feature translator to reconstruct bottleneck features of the missing modalities. Our experiments on three different datasets (MIMIC-CXR, NIH Open-I, and CheXpert), in both homogeneous and heterogeneous settings consistently improve the performance of competitive baselines. The code and implementation details are available at: https://github.com/bhattarailab/FedFeatGen

Figures

Figures reproduced from arXiv: 2505.20232 by the authors.

**Figure 2.** Illustration of Feature Imputation Network-based Multimodal Federated Learning. (a) Multimodal Federated Learning system with different types of clients. (b) Training of the Feature Imputation Network in multimodal client. (c) Architecture of Feature Imputation Network (d) Unimodal image client training with the help of the Feature Imputation Network. L(ΦT ) = 1 |Dc| | X Dc| i=1 ∥ΦT (z (i) I ) − z (i) T ∥ 2 2 (4) A… view at source ↗

**Figure 3.** t-SNE plot of feature vectors from the model trained in (a) the homogeneous setup and (b) the heterogeneous setup. In the Figure, Upperbound refers to the model trained in a federated manner with complete modalities. Feature vectors are generated using the validation data [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 9 canonical work pages

[1]

Nature Medicine28(9), 1773–1784 (2022)

Acosta, J.N., Falcone, G.J., Rajpurkar, P., Topol, E.J.: Multimodal biomedical ai. Nature Medicine28(9), 1773–1784 (2022)

2022
[2]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., Ballas, N.: Self-supervised learning from images with a joint-embedding pre- dictive architecture. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15619–15629 (2023)

work page 2023
[3]

In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining

Chen, J., Zhang, A.: Fedmsplit: Correlation-adaptive federated multi-task learn- ing across multimodal split networks. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. pp. 87–96 (2022)

work page 2022
[4]

Computerized Medical Imaging and Graphics p

Chen, J., Pan, R.: Medical report generation based on multimodal federated learn- ing. Computerized Medical Imaging and Graphics p. 102342 (2024)

work page 2024
[5]

In: Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (Nov 2020)

Chen, Z., Song, Y., Chang, T.H., Wan, X.: Generating radiology reports via memory-driven transformer. In: Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (Nov 2020)

work page 2020
[6]

Journal of the American Medical Informatics Association23(2), 304–310 (2016)

Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiol- ogy examinations for distribution and retrieval. Journal of the American Medical Informatics Association23(2), 304–310 (2016)

2016
[7]

arXiv preprint arXiv:1810.04805 (2018)

Devlin,J.,Chang,M.W.,Lee,K.,Toutanova,K.:Bert:Pre-trainingofdeepbidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

arXiv 2018
[8]

Feng, T., Bose, D., Zhang, T., Hebbar, R., Ramakrishna, A., Gupta, R., Zhang, M., Avestimehr, S., Narayanan, S.: Fedmultimodal: A benchmark for multimodal feder- atedlearning.In:Proceedingsofthe29thACMSIGKDDConferenceonKnowledge Discovery and Data Mining. pp. 4035–4045 (2023)

work page 2023

Show all 26 references

[9]

He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

2016
[10]

In: Proceedings of the AAAI conference on artificial intelligence

Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial in...

2019
[11]

Scientific data6(1), 317 (2019)

Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data6(1), 317 (2019)

2019
[12]

arXiv preprint arXiv:1412.6980 (2014)

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

2014 arXiv
[13]

arXiv preprint arXiv:2401.13898 (2024)

Le, H.Q., Thwal, C.M., Qiao, Y., Tun, Y.L., Nguyen, M.N., Hong, C.S.: Cross- modal prototype based multimodal federated learning under severely missing modality. arXiv preprint arXiv:2401.13898 (2024)

2024 arXiv
[14]

In: Artificial intelligence and statistics

McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. pp. 1273–1282. PMLR (2017)

2017
[15]

Poudel, P., Shrestha, P., Amgain, S., Shrestha, Y.R., Gyawali, P., Bhattarai, B.: Car-mfl: Cross-modal augmentation by retrieval for multimodal federated learning withmissingmodalities.In:InternationalConferenceonMedicalImageComputing and Computer-Assisted Intervention. pp. 10...

2024
[16]

Qayyum, A., Ahmad, K., Ahsan, M.A., Al-Fuqaha, A., Qadir, J.: Collaborative federatedlearningforhealthcare:Multi-modalcovid-19diagnosisattheedge.IEEE Open Journal of the Computer Society3, 172–184 (2022)

2022
[17]

SN Computer Science4(5), 674 (2023)

Sachin, D., Annappa, B., Ambasange, S., Tony, A.E.: A multimodal contrastive federated learning for digital healthcare. SN Computer Science4(5), 674 (2023)

2023
[18]

arXiv preprint arXiv:2402.05294 (2024)

Saha, P., Mishra, D., Wagner, F., Kamnitsas, K., Noble, J.A.: Examining modality incongruityinmultimodalfederatedlearningformedicalvisionandlanguage-based disease detection. arXiv preprint arXiv:2402.05294 (2024)

2024 arXiv
[19]

arXiv preprint arXiv:2312.06224 (2023)

Shrestha, P., Amgain, S., Khanal, B., Linte, C.A., Bhattarai, B.: Medical vision language pretraining: A survey. arXiv preprint arXiv:2312.06224 (2023)

2023 arXiv
[20]

In: European Conference on Computer Vision

Sun, G., Mendieta, M., Dutta, A., Li, X., Chen, C.: Towards multi-modal trans- formers in federated learning. In: European Conference on Computer Vision. pp. 229–246. Springer (2024)

2024
[21]

arXiv preprint arXiv:2310.09650 (2023)

Thrasher, J., Devkota, A., Siwakotai, P., Chivukula, R., Poudel, P., Hu, C., Bhat- tarai, B., Gyawali, P.: Multimodal federated learning in healthcare: a review. arXiv preprint arXiv:2310.09650 (2023)

2023 arXiv
[22]

Advances in neural information pro- cessing systems30(2017)

Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)

2017
[23]

Scientific reports 11(1), 3254 (2021)

Venugopalan, J., Tong, L., Hassanzadeh, H.R., Wang, M.D.: Multimodal deep learning models for early detection of alzheimer’s disease stage. Scientific reports 11(1), 3254 (2021)

2021
[24]

arXiv preprint arXiv:2409.07825 (2024)

Wu, R., Wang, H., Chen, H.T., Carneiro, G.: Deep multimodal learning with miss- ing modality: A survey. arXiv preprint arXiv:2409.07825 (2024)

2024 arXiv
[25]

arXiv preprint arXiv:2302.08888 (2023)

Yu, Q., Liu, Y., Wang, Y., Xu, K., Liu, J.: Multimodal federated learning via contrastive representation ensemble. arXiv preprint arXiv:2302.08888 (2023)

2023 arXiv
[26]

arXiv preprint arXiv:2302.08646 (2023)

Zheng, T., Li, A., Chen, Z., Wang, H., Luo, J.: Autofed: Heterogeneity-aware federated multimodal learning for robust autonomous driving. arXiv preprint arXiv:2302.08646 (2023)

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Nature Medicine28(9), 1773–1784 (2022)

Acosta, J.N., Falcone, G.J., Rajpurkar, P., Topol, E.J.: Multimodal biomedical ai. Nature Medicine28(9), 1773–1784 (2022)

2022

[2] [2]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., Ballas, N.: Self-supervised learning from images with a joint-embedding pre- dictive architecture. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15619–15629 (2023)

work page 2023

[3] [3]

In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining

Chen, J., Zhang, A.: Fedmsplit: Correlation-adaptive federated multi-task learn- ing across multimodal split networks. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. pp. 87–96 (2022)

work page 2022

[4] [4]

Computerized Medical Imaging and Graphics p

Chen, J., Pan, R.: Medical report generation based on multimodal federated learn- ing. Computerized Medical Imaging and Graphics p. 102342 (2024)

work page 2024

[5] [5]

In: Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (Nov 2020)

Chen, Z., Song, Y., Chang, T.H., Wan, X.: Generating radiology reports via memory-driven transformer. In: Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (Nov 2020)

work page 2020

[6] [6]

Journal of the American Medical Informatics Association23(2), 304–310 (2016)

Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiol- ogy examinations for distribution and retrieval. Journal of the American Medical Informatics Association23(2), 304–310 (2016)

2016

[7] [7]

arXiv preprint arXiv:1810.04805 (2018)

Devlin,J.,Chang,M.W.,Lee,K.,Toutanova,K.:Bert:Pre-trainingofdeepbidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

arXiv 2018

[8] [8]

Feng, T., Bose, D., Zhang, T., Hebbar, R., Ramakrishna, A., Gupta, R., Zhang, M., Avestimehr, S., Narayanan, S.: Fedmultimodal: A benchmark for multimodal feder- atedlearning.In:Proceedingsofthe29thACMSIGKDDConferenceonKnowledge Discovery and Data Mining. pp. 4035–4045 (2023)

work page 2023