Pith. sign in

REVIEW 2 major objections 5 minor 35 references

ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities

T0 review · 2 major / 5 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A prototype-conditioned mixture of experts synthesizes missing modalities in federated medical learning without public data.

desk verdict Solid empirical FL paper: client-aware prototype bank + modality-routed MoE for missing CXR modalities beats FeatImp/PmcmFL without public data, especially under hetero partitions. read the letter →

arxiv 2607.06633 v1 pith:AWHYVVJC submitted 2026-07-07 cs.CV cs.AI

Aavash Chhetri , Bibek Niroula , Eduard Vazquez , Yash Raj Shrestha , Prashnna Gyawali , Loris Bazzani , Binod Bhattarai This is my paper · ORCID
classification cs.CVcs.AI
keywords multimodalfederatedlearningmissingmodalitiesprototypemixtureofexpertschestX-rayfeaturesynthesisnon-IIDclients
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hospitals hold complementary imaging and report data but cannot pool it, and many sites only have one modality. Prior federated methods either demand a public multimodal dataset or invent the missing features from the available modality alone, which fails when institutions differ. ProMoE-FL instead keeps a global bank of client-aware class prototypes that summarize how each modality looks across sites, then routes those prototypes through a shared mixture of experts so a unimodal client can reconstruct the missing representation. On four chest X-ray collections under both matched and mismatched client partitions, the method raises diagnostic AUC over feature-imputation and pure-prototype baselines, and it preserves separation of rare pathologies that simpler synthesis collapses.

What carries the argument

Prototype-Conditioned Mixture of Experts (ProMoE): a Transformer decoder bank whose experts attend over the global prototype set of the target modality, gated by a modality-aware router that also receives the source feature and the (from, to) modality indices.

What would settle it

Train the same architecture on the heterogeneous 8:0:2 or 4:4:2 partitions but replace the accumulated client-aware prototype bank with either averaged global prototypes or pure source-only imputation; if macro AUC and rare-pathology centroid separation then match or exceed ProMoE-FL, the claimed value of the bank collapses.

Watch

Extended reading notes

Core claim

Conditioning a shared mixture-of-experts decoder on an accumulated, client-aware global prototype bank of the missing modality produces more accurate cross-modal features than either public-data augmentation or synthesis that uses only the observed modality, and the gain is largest under realistic non-IID client mixtures.

Load-bearing premise

The paper assumes that cross-modal maps learned by reconstruction on multimodal clients, when steered by the accumulated prototype bank, still work for unimodal clients whose data distributions differ from those multimodal sites.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. ProMoE-FL addresses multimodal federated learning with missing modalities by synthesizing missing latent features via a prototype-conditioned Mixture-of-Experts. Clients maintain class-conditioned modality prototypes that are accumulated (not averaged) into a global client-aware prototype bank; a Transformer-decoder PCD attends over the target-modality bank, and a modality-index-aware router mixes E such experts (Eq. 4). Multimodal clients train the MoE by symmetric MSE reconstruction (Eq. 6); unimodal clients use the synthesized features for the task loss. Experiments on MIMIC-CXR, NIH Open-I, PadChest, and CheXpert cover nine I:T:M partitions under homogeneous and heterogeneous regimes, reporting multi-seed macro AUC against Zero/Unif filling, FeatImp, PmcmFL, and (with a virtual public client) CAR-MFL, plus t-SNE/UMAP qualitative checks and a single MoE-vs-single-PCD ablation.

Significance. Missing-modality multimodal FL is a realistic clinical constraint, and the paper targets it without requiring public multimodal data. The combination of an accumulated client-aware prototype bank with direction-aware MoE routing is a concrete architectural contribution relative to pure feature imputation (FeatImp) and class-prototype substitution (PmcmFL). Strengths include multi-seed Table 1 results across nine partitions in both homogeneous and heterogeneous regimes, explicit comparison to a public-data baseline when adapted, qualitative alignment evidence (Fig. 2), and a public code link. If the heterogeneous gains hold under broader scrutiny, the method is a useful practical advance for multi-center chest X-ray federations with incomplete modality availability.

major comments (2)
  1. [Abstract; Table 1] Abstract and §3 claim that ProMoE-FL 'consistently outperforms' SOTA in both homogeneous and heterogeneous settings. Table 1 supports this fully for heterogeneous partitions, but under homogeneous data ProMoE-FL is best in only 6/9 configurations and trails FeatImp in 0:4:6, 0:6:4, and 0:8:2 (text-heavy). The abstract should be tightened to match the table (e.g., 'best or competitive under homogeneous; consistently best under heterogeneous').
  2. [§2 Eqs. 4–6; §3 MoE ablation; Fig. 2] The load-bearing transfer claim—that MSE reconstruction on multimodal clients (Eq. 6) conditioned on an accumulated prototype bank and modality-aware MoE (Eq. 4) transfers to unimodal clients from different institutions—is supported mainly by end-task AUC and t-SNE/UMAP. The only MoE ablation is a single 8:0:2 heterogeneous comparison (79.68 vs 78.98). A short ablation isolating accumulation vs averaging of the prototype bank, and at least one more MoE-vs-single-PCD setting (e.g., 4:4:2 or 0:8:2), would make the central design choices more load-bearing rather than confounded with the full pipeline.
minor comments (5)
  1. [§2 Missing Modality Feature Synthesis; Implementation Details] Number of experts E is never stated in the main text (only that E PCDs are used); report E and any sensitivity.
  2. [Eq. (4)] Eq. (4) normalizes the weighted expert sum by its L2 norm; a one-sentence justification relative to unnormalized gating would help.
  3. [Fig. 1] Fig. 1 caption and panel labels are dense; clarify which blocks are frozen vs trainable on unimodal clients.
  4. [§1] Typo: 'Toaddresstheselimitations' (missing spaces) near the end of the introduction.
  5. [§3 Clinical Relevancy] PadChest Spanish reports are a nice cross-lingual stress; a brief note on whether the BERT encoder was multilingual or English-only would help interpret that setting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical FL method whose AUC gains are measured against external public benchmarks and independent baselines, not forced by definition or self-citation.

full rationale

ProMoE-FL is an engineering/method paper. Its central claims are comparative macro-AUC numbers on held-out MIMIC-CXR splits under nine client partitions (homogeneous and heterogeneous mixes of MIMIC-CXR, Open-I, PadChest, CheXpert). The prototype alignment loss (Eq. 3), the MoE reconstruction objective on multimodal clients (Eq. 6), and the gated synthesis for unimodal clients (Eqs. 4–5) are ordinary training objectives; none of them algebraically equals the reported AUC. Self-citations (CAR-MFL, FeatImp, Med-MMFL) supply baselines, datasets, and related prior work; they do not supply a uniqueness theorem or an ansatz that forces the measured gains. Qualitative t-SNE/UMAP figures are post-hoc visualizations of the same trained features, not tautological rewrites of the inputs. The paper is therefore self-contained against external benchmarks; circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 3 invented entities

Load-bearing content is architectural and empirical, not axiomatic physics. Free parameters are standard ML knobs (λ, optimizer, rounds, latent dim). Domain assumptions are the usual FL and multi-label CXR setup. Invented entities are the method modules themselves; they have no evidence outside this paper's experiments.

free parameters (4)
  • prototype loss weight λ
    Explicitly tuned on the grid {0.1, 0.5, 1, 5, 10}; relative weighting of L_proto vs L_task affects prototype quality and thus synthesis.
  • number of MoE experts E
    Architecture width of Φ_MoE; ablation contrasts MoE vs single PCD but main-table E is not fixed in the text, so performance depends on this design choice.
  • latent feature dimension d=256
    Chosen projection size for encoders and prototypes; shapes capacity of synthesis and prototype bank.
  • local epochs N=3, communication rounds=30, lr=1e-4
    Training schedule and Adam step size chosen by authors; affect convergence under non-IID partitions.
assumptions (4)
  • domain assumption FedAvg aggregation of encoders, classifier, projection heads, and MoE parameters is sufficient for stable multimodal FL under the studied non-IID partitions.
    Stated in §2 Federated Training; no alternative aggregators tested.
  • ad hoc to paper Class-conditioned client prototypes accumulated into a global bank supply complementary target-modality priors that improve synthesis under client heterogeneity.
    Core design claim in §2 Prototype Construction; accumulation (not averaging) is a paper-specific choice.
  • domain assumption MSE reconstruction of latent features on multimodal clients is an adequate training signal for clinically useful missing-modality features.
    Eq. 6; standard feature-imputation assumption shared with FeatImp-style work.
  • domain assumption Macro AUC on multi-label CXR findings is a valid proxy for method quality under missing modalities.
    Evaluation protocol §3 following CAR-MFL splits.
invented entities (3)
  • client-aware global prototype bank (accumulated, not averaged)
    purpose: Provide federation-wide, client-specific modality priors that the decoder attends to when synthesizing missing features.
    Defined in §2; validated only via this paper's ablations/results, not independent external measurement.
  • Prototype-Conditioned Decoder (PCD) as Transformer decoder expert
    purpose: Let available-modality features attend over target-modality prototypes to reconstruct missing latent vectors.
    Fig. 1(c); architectural invention of the paper.
  • Modality-Aware Router R(h_i, P_j, i, j) for direction-aware MoE gating
    purpose: Route synthesis through shared experts conditioned on instance context and (from,to) modality indices without combinatorial parameter growth.
    Eq. 4 and Fig. 1(d); paper-specific routing design.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities." pith.science (2026). https://pith.science/paper/AWHYVVJC

@misc{pith2026260706633,
  author       = {Pith},
  title        = {Pith review of: ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AWHYVVJC}},
  note         = {Machine review of arXiv:2607.06633}
}
read the original abstract

In this paper, we address the problem of multimodal federated learning with missing modality. Existing methods utilize an additional public dataset or perform naive feature synthesis that is based solely on the available modality. To address these limitations, we propose ProMoE-FL, a Prototype-conditioned Mixture-of-Experts framework for robust missing-modality feature synthesis in multimodal federated learning. ProMoE-FL builds a global client-aware prototype bank that captures clinically meaningful modality priors across institutions. Our Mixture of Experts is conditioned on these prototypes and modality indices to enable direction-aware expert routing for dynamically synthesizing missing features. We perform extensive quantitative and qualitative evaluations on four public chest X-ray datasets (MIMIC-CXR, NIH Open-I, PadChest, and CheXpert) and demonstrate that ProMoE-FL consistently outperforms state-of-the-art methods in both homogeneous as well as the more challenging heterogeneous settings.

Figures

Figures reproduced from arXiv: 2607.06633 by the authors.

Figure 1
Figure 1. Overview of the proposed ProMoE-FL framework. from multimodal clients can degrade performance under client heterogeneity, where cross-modal relationships vary across sites and available modalities pro￾vide incomplete priors. To mitigate this, we introduce a client-aware global pro￾totype bank encoding federation-wide modality priors, conditioning the synthe￾sis process on target-modality prototypes. Furthermore, to … view at source ↗
Figure 2
Figure 2. Qualitative Analysis of the methods. (a–b) t-SNE of learnt features with class centroids. (c) Gating distributions across experts. (d–e) UMAP of rare pathological classes. semantic mapping. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 35 canonical work pages

  1. [1]

    Nature Medicine28(9), 1773–1784 (2022)

    Acosta, J., Falcone, G., Rajpurkar, P., Topol, E.: Multimodal biomedical ai. Nature Medicine28(9), 1773–1784 (2022)

    work page 2022

  2. [2]

    J Med Internet Res26, e59505 (Sep 2024)

    AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.A., Damseh, R., Sheikh, J.: Multimodal large language models in health care: Applications, challenges, and future outlook. J Med Internet Res26, e59505 (Sep 2024)

    work page 2024

  3. [3]

    In: ICML (2024)

    Bao, G., Zhang, Q., Miao, D., Gong, Z., Hu, L., Liu, K., Liu, Y., Shi, C.: Multi- modal federated learning with missing modality via prototype mask and contrast. In: ICML (2024)

    work page 2024

  4. [4]

    Medical Image Anal- ysis66, 101797 (2020)

    Bustos, A., Pertusa, A., Salinas, J.M., de la Iglesia-Vayá, M.: Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical Image Anal- ysis66, 101797 (2020)

    work page 2020

  5. [5]

    Computerized medical imaging and graphics : the official journal of the Com- puterized Medical Imaging Society113, 102342 (2024)

    Chen, J., Pan, R.: Medical report generation based on multimodal federated learn- ing. Computerized medical imaging and graphics : the official journal of the Com- puterized Medical Imaging Society113, 102342 (2024)

    work page 2024

  6. [6]

    Chen, Y., Liu, C., Huang, W., Cheng, S., Arcucci, R., Xiong, Z.: Generative text-guided 3d vision-language pretraining for unified medical image segmentation (2023)

    work page 2023

  7. [7]

    arXiv preprint arXiv:2602.04416 (2026)

    Chhetri, A., Niroula, B., Shrestha, P., Shrestha, Y.R., Anderson, L.A., Gyawali, P.K., Bazzani, L., Bhattarai, B.: Med-mmfl: A multimodal federated learning benchmark in healthcare. arXiv preprint arXiv:2602.04416 (2026)

    work page arXiv 2026

  8. [8]

    In: ICCV (2025)

    Dai, R., Li, C., Yan, Y., Mo, L., Qin, K., He, T.: Unbiased missing-modality mul- timodal learning. In: ICCV (2025)

    work page 2025

Show all 35 references
  1. [9]

    Journal of the American Medical Informatics Association23(2), 304–310 (2016)

    Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiol- ogy examinations for distribution and retrieval. Journal of the American Medical Informatics Association23(2), 304–310 (2016)

    2016

  2. [10]

    arXiv preprint arXiv:1810.04805 (2018) 10 A

    Devlin,J.,Chang,M.W.,Lee,K.,Toutanova,K.:Bert:Pre-trainingofdeepbidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) 10 A. Chhetri et al

    2018 arXiv

  3. [11]

    Circulation [Online]101(23), e215–e220 (2000)

    Goldberger, A., Amaral, L., Glass, L., Hausdorff, J., Ivanov, P.C., Mark, R., Stan- ley, H.E.: Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic signals. Circulation [Online]101(23), e215–e220 (2000)

    2000

  4. [12]

    In: CVPR

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)

    2016

  5. [13]

    In: AAAI (2019)

    Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: AAAI (2019)

    2019

  6. [14]

    Scientific data6(1), 317 (2019)

    Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data6(1), 317 (2019)

    2019

  7. [15]

    arXiv preprint arXiv:1412.6980 (2014)

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  8. [16]

    arXiv preprint arXiv:1908.06683 (2019)

    Lau, K., Adler, J., Sjölund, J.: A unified representation network for segmentation with missing modalities. arXiv preprint arXiv:1908.06683 (2019)

    1908 arXiv

  9. [17]

    Information Fusion122, 103219 (2025)

    Le, H.Q., Thwal, C.M., Qiao, Y., Tun, Y.L., Nguyen, M.N., Huh, E.N., Hong, C.S.: Cross-modal prototype based multimodal federated learning under severely missing modality. Information Fusion122, 103219 (2025)

    2025

  10. [18]

    In: CHIL (2024)

    Lee, H., Lee, D.Y., Kim, W., Kim, J.H., Kim, T., Kim, J., Sunwoo, L., Choi, E.: Vision-language generative model for view-specific chest x-ray generation. In: CHIL (2024)

    2024

  11. [19]

    In: CVPR (2021)

    Li, Q., He, B., Song, D.: Model-contrastive federated learning. In: CVPR (2021)

    2021

  12. [20]

    In: AAAI (2021)

    Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: Smil: Multimodal learning with severely missing modality. In: AAAI (2021)

    2021

  13. [21]

    In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)

    McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.: Communication- efficient learning of deep networks from decentralized data. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS). pp. 1273–1282. PMLR (2017)

    2017

  14. [22]

    In: MIUA (2025)

    Poudel, P., Chhetri, A., Gyawali, P., Leontidis, G., Bhattarai, B.: Multimodal federated learning with missing modalities through feature imputation network. In: MIUA (2025)

    2025

  15. [23]

    In: MICCAI (2024)

    Poudel, P., Shrestha, P., Amgain, S., Shrestha, Y.R., Gyawali, P., Bhattarai, B.: CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities . In: MICCAI (2024)

    2024

  16. [24]

    Qayyum, A., Ahmad, K., Ahsan, M.A., Al-Fuqaha, A., Qadir, J.: Collaborative federatedlearningforhealthcare:Multi-modalcovid-19diagnosisattheedge.IEEE Open Journal of the Computer Society3, 172–184 (2022)

    2022

  17. [25]

    SN Computer Science4, 1–12 (2023)

    Sachin, D.N., Annappa, B., Ambasange, S., Tony, A.E.: A multimodal contrastive federated learning for digital healthcare. SN Computer Science4, 1–12 (2023)

    2023

  18. [26]

    arXiv preprint arXiv:2402.05294 (2024)

    Saha, P., Mishra, D., Wagner, F., Kamnitsas, K., Noble, J.A.: Examining modality incongruityinmultimodalfederatedlearningformedicalvisionandlanguage-based disease detection. arXiv preprint arXiv:2402.05294 (2024)

    2024 arXiv

  19. [27]

    AAAI39(27), 28331–28339 (Apr 2025)

    Saha, P., Mishra, D., Wagner, F., Kamnitsas, K., Noble, J.A.: Incongruent multi- modal federated learning for medical vision and language-based multi-label disease detection. AAAI39(27), 28331–28339 (Apr 2025)

    2025

  20. [28]

    Seibold, C., Reiß, S., Sarfraz, M.S., Stiefelhagen, R., Kleesiek, J.: Breaking with Fixed Set Pathology Recognition Through Report-Guided Contrastive Training, p. 690–700. Springer Nature Switzerland (2022) ProMoE-FL 11

    2022

  21. [29]

    arXiv preprint arXiv:2312.06224 (2023)

    Shrestha, P., Amgain, S., Khanal, B., Linte, C., Bhattarai, B.: Medical vision language pretraining: A survey. arXiv preprint arXiv:2312.06224 (2023)

    2023 arXiv

  22. [30]

    In: ECCV

    Sun, G., Mendieta, M., Dutta, A., Li, X., Chen, C.: Towards multi-modal trans- formers in federated learning. In: ECCV. p. 229–246. Springer-Verlag, Berlin, Hei- delberg (2024)

    2024

  23. [31]

    arXiv preprint arXiv:2310.09650 (2023)

    Thrasher, J., Devkota, A., Siwakotai, P., Chivukula, R., Poudel, P., Hu, C., Bhat- tarai, B., Gyawali, P.: Multimodal federated learning in healthcare: A review. arXiv preprint arXiv:2310.09650 (2023)

    2023 arXiv

  24. [32]

    In: CVPR (2024)

    Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., Carneiro, G.: Multi-modal learning with missing modality via shared-specific feature modelling. In: CVPR (2024)

    2024

  25. [33]

    You, K., Gu, J., Ham, J., Park, B., Kim, J., Hong, E.K., Baek, W., Roh, B.: CXR- CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training, p. 101–111. Springer Nature Switzerland (2023)

    2023

  26. [34]

    In: ICLR (2023)

    Yu, Q., Liu, Y., Wang, Y., Xu, K., Liu, J.: Multimodal federated learning via contrastive representation ensemble. In: ICLR (2023)

    2023

  27. [35]

    In: Zong, C., Xia, F., Li, W., Navigli, R

    Zhao, J., Li, R., Jin, Q.: Missing modality imagination network for emotion recog- nition with uncertain missing modalities. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) ACL. pp. 2608–2618 (Aug 2021)

    2021

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.