The reviewed record of science sign in
Pith

arxiv: 2505.22483 · v2 · pith:AX4G7K4T · submitted 2025-05-28 · cs.LG · cs.AI· cs.CV

A Closer Look at Multimodal Representation Collapse

Abhra Chaudhuri , Anjan Dutta , Tu Bui , Serban Georgescu This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:AX4G7K4Trecord.jsonopen to challenge →

classification cs.LG cs.AIcs.CV
keywords modalitycollapsefeaturesmultimodalpredictivefusionmodalitiesabhrac
0
0 comments X
read the original abstract

We aim to develop a fundamental understanding of modality collapse, a recently observed empirical phenomenon wherein models trained for multimodal fusion tend to rely only on a subset of the modalities, ignoring the rest. We show that modality collapse happens when noisy features from one modality are entangled, via a shared set of neurons in the fusion head, with predictive features from another, effectively masking out positive contributions from the predictive features of the former modality and leading to its collapse. We further prove that cross-modal knowledge distillation implicitly disentangles such representations by freeing up rank bottlenecks in the student encoder, denoising the fusion-head outputs without negatively impacting the predictive features from either modality. Based on the above findings, we propose an algorithm that prevents modality collapse through explicit basis reallocation, with applications in dealing with missing modalities. Extensive experiments on multiple multimodal benchmarks validate our theoretical claims. Project page: https://abhrac.github.io/mmcollapse/.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LLM Features Can Hurt GNNs: Concatenation Interference on Homophilous Graph Benchmarks

    cs.LG 2026-06 accept novelty 7.0

    Pure concatenation of LLM features degrades GNN accuracy on homophilous datasets, with Delta_sig metric predicting when the drop occurs better than homophily.

  2. MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment

    cs.CV 2026-04 unverdicted novelty 6.0

    MiMIC mitigates visual modality collapse and semantic misalignment in universal multimodal retrieval via fusion-in-decoder architecture and robust single-modality training.

  3. Token-Efficient Multimodal Reasoning via Image Prompt Packaging

    cs.CV 2026-04 unverdicted novelty 6.0

    IPPg embeds text into images to reduce multimodal model inference costs by 35.8-91% with competitive accuracy on many VQA and code benchmarks.

  4. StateXDiff: Cell State-Contextualized Multimodal Diffusion for Single-Cell Perturbation Prediction

    q-bio.GN 2026-05 unverdicted novelty 5.0

    StateXDiff integrates transcriptomic profiles with inferred protein features via a conditional diffusion model and mechanism-aware drug templates to predict single-cell drug perturbation responses under unseen cell li...

  5. Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusion

    cs.CV 2026-01 unverdicted novelty 5.0

    A regularization method enforces diverse intra-modal embeddings and bounded inter-modal drift to improve both multimodal fusion and unimodal robustness.

  6. ModalImmune: Immunity Driven Unlearning via Self Destructive Training

    cs.LG 2026-02 unverdicted novelty 4.0

    ModalImmune enforces modality immunity in multimodal models by controlled collapse of input channels during training using adaptive regularizers and meta-optimization.