Pith. sign in

REVIEW 2 major objections 6 minor 59 references

Continuous Quality Spectrum Unifies Missing and Corrupted Modalities

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 22:25 UTC pith:4Q67PP4K

load-bearing objection Unified continuous-degradation framework for multimodal missingness — solid concept, but generalization claims to fog/rain/snow lack the evidence to back them up. the 2 major comments →

arxiv 2607.06943 v1 pith:4Q67PP4K submitted 2026-07-08 cs.CV

General Incomplete Multimodal Learning via Dynamic Quality Perception

Xiangyu Meng , Shicai Wei This is my paper
classification cs.CV
keywords missinggimlqualitydegradationlearningmodalitiesmultimodalnoise
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the two main forms of multimodal incompleteness — entire modalities going missing versus modalities being present but corrupted by noise — are not separate problems but endpoints of a single continuous quality spectrum ranging from 1 (fully clean) to 0 (completely absent). The authors propose GIML, a framework that unifies both within one stage by explicitly learning to estimate where each modality falls on this spectrum through controlled noise injection during training, then using those estimates to adaptively weight modalities during fusion. A companion module disentangles task-relevant semantics from noise-induced uncertainty by parameterizing each modality embedding as a Gaussian distribution whose mean carries semantics and whose variance tracks degradation. The paper claims this supervised, continuous quality estimation generalizes to unseen corruption types and intensities because the model learns the relationship between degradation severity and modality reliability rather than memorizing specific corruption patterns, outperforming prior methods that treat missing and corrupted modalities as sequential, separate problems.

Core claim

The central mechanism is a two-part pipeline. First, a Noise-Semantic Decoupled module parameterizes each modality embedding as a Gaussian distribution where the mean encodes task-relevant semantics and the variance encodes degradation-induced uncertainty, enforced by a KL-divergence prior that aligns the variance with the known noise intensity used during controlled training-time injection. Second, a Noise-aware Quality Estimator learns a direct, supervised mapping from this variance to the actual degradation level via mean-squared-error loss on injected noise intensities, producing calibrated quality scores that drive inverse-variance fusion weighting. The combination means that as a modal

What carries the argument

The load-bearing object is the continuous quality coefficient w_i^(v) in [0,1] that replaces the conventional binary missing indicator. It is computed from a supervised mapping: noise injection at known intensity η → Gaussian variance σ in the representation → learned estimator f_t that maps σ back to a predicted degradation level η̂ → inverse-variance fusion weight ω. The entire chain is trained end-to-end with direct MSE supervision on the degradation level, making quality estimation an explicitly calibrated rather than implicitly inferred quantity.

Load-bearing premise

The framework assumes that the variance produced by the semantic-noise decoupling module maintains a learnable, monotonic relationship to true degradation intensity even under noise types and intensities never encountered during training. If this variance saturates or decouples from actual information loss under novel corruptions, the quality estimates become unreliable and the adaptive fusion weights will be miscalibrated.

What would settle it

If a modality is corrupted by a noise type whose effect on the learned representation variance does not monotonically track the actual information loss — for instance, a corruption that inflates variance without degrading semantics, or one that suppresses variance while destroying information — then the quality estimator would assign incorrect fusion weights, and the semantic decoupling would fail to protect task performance.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The manuscript proposes General Incomplete Multimodal Learning (GIML), a framework that unifies inter-modality missing and intra-modality degradation by modeling modality conditions as continuous quality degradation. The method introduces two components: a Noise-Semantic Decoupled (NSD) module that parameterizes modality embeddings as Gaussian distributions to separate semantics from noise, and a Noise-aware Quality Estimator (NQE) that learns to predict degradation intensity from the NSD variance via controlled noise injection. The estimated quality then guides adaptive fusion. Experiments span five datasets with diverse modality combinations, and the paper evaluates robustness to unseen noise intensities and types.

Significance. The problem of jointly handling intra-modality corruption and inter-modality missing is well-motivated, as prior two-stage approaches (T2DR, TMDC) treat these separately and risk optimization conflicts. The continuous degradation formulation is a clean conceptual contribution. The experimental coverage is broad: five datasets, bimodal and trimodal settings, unseen intensities (Table 5), unseen noise types including Mask→Gaussian transfer (Table 6) and realistic corruptions (Table 7), and ablations isolating NQE and NSD (Tables 10–13). The Spearman correlation analysis (Table 8) provides a falsifiable check on whether the estimated degradation tracks true corruption severity. Code is publicly available, which supports reproducibility.

major comments (2)
  1. Circularity between NSD regularization and NQE supervision. Eq. (8) regularizes the variance σ_i^(v) to align with the degradation intensity η_i^(v) via KL divergence, and Eq. (9)–(10) train the NQE to predict η from σ. This creates a near-circular dependency: σ is explicitly pushed toward η during training, and the NQE then learns the σ→η̂ mapping under direct MSE supervision. The high Spearman correlations in Table 8 (0.991 for Mask) therefore partly reflect supervised fitting rather than evidence that the model has learned a generalizable quality estimator. The paper should clarify what independent signal the NQE provides beyond the already-regularized σ, or demonstrate that the NQE generalizes to noise types where the KL regularization in Eq. (8) was not applied. Without this, the contribution of NQE as a separate module is unclear.
  2. Generalization to unseen corruption types lacks mechanism specification. Table 7 evaluates fog, rain, snow, and Mask+Gaussian corruptions, but the NQE is trained only on Mask noise with a defined η. For structured corruptions like fog (which affects spatial frequency content rather than pixel-level masking), no ground-truth η is defined. The paper does not specify how quality is estimated for these corruptions at inference time. If σ does not track degradation severity for these structured corruptions, the fusion weights from Eq. (11) will be miscalibrated, and the performance gains in Table 7 could stem from NSD's semantic robustness alone rather than NQE's quality-guided fusion. The paper should either (a) provide Spearman correlation evidence for these additional corruption types, or (b) run an ablation on Table 7 settings with NQE disabled (uniform weights) to isolate whether NQE or
minor comments (6)
  1. Eq. (11): The fusion weight formula uses η̂ in the numerator and denominator, but the relationship between ω and the continuous coefficient w_i^(v) defined in Eq. (4) is not explicitly stated. Clarify how ω maps to w in the fused representation in Eq. (4).
  2. Table 2: Several cells have formatting issues where values appear concatenated without proper spacing (e.g., '72.9372.42' and '50.7848.50'). These should be separated into distinct Acc/F1 columns.
  3. Section 3.1: The notation δ_i^v is introduced in Eq. (1) as a Bernoulli indicator but the transition to the continuous coefficient w_i^(v) in Eq. (4) could benefit from an explicit statement that w replaces δ.
  4. Table 4: The intra ratio notation (r_a, r_v, r_t) is used but the mapping of subscripts to modality names (audio, visual, text) is not explicitly stated in the table caption.
  5. Section 3.2: The reparameterization trick is cited as [44], but the standard reference is Kingma and Welling (2013) or Rezende et al. (2014). Reference [44] appears to be the authors' own prior work; please verify this is the intended citation.
  6. Table 13: The 'w/o L_mse' variant reports ρ(η̂, Δf) of 0.018/0.527, which is a dramatic drop from 0.991/0.991. This suggests the NQE's correlation is almost entirely driven by the MSE supervision, reinforcing the circularity concern in the first major comment. The authors should discuss this explicitly.

Circularity Check

1 steps flagged

No significant circularity; the σ→η̂ mapping is supervised but not self-definitional, and generalization to unseen corruptions is empirically tested rather than assumed by construction.

specific steps
  1. fitted input called prediction [§3.3, Eqs. 8–10, Table 8]
    "L_reg = (1/N) Σ KL[ N(μ_i^(v), (σ_i^(v))²I) || N(0, (η_i^(v))²I) ] ... η̂_i^(v) = f_t^(v)(σ_i^(v)) ... L_mse = (1/N) Σ (η̂_i^(v) - η_i^(v))² ... Table 8: ρ(η̂, Δf) = 0.991 / 0.973 (Mask / Gaussian)"

    The variance σ is regularized toward η (Eq. 8), and η̂ is predicted from σ and supervised by η (Eqs. 9–10). The high Spearman correlations in Table 8 thus partly reflect that σ is explicitly shaped to match η during training. However, this is not circular by construction: η̂ is not defined as η; it is a learned mapping from σ via a separate network f_t, and the correlation is measured against Δf (cosine distance between clean and corrupted features), an independent quantity not used in the loss. The supervision is standard regression, not a tautology. The generalization to unseen noise types (Table 7) is an empirical claim tested on fog/rain/snow, not assumed by the training equations. This is a mild fitted-input concern, not a self-definitional reduction.

full rationale

The paper's core mechanism is not circular. The NQE learns a mapping σ→η̂ via supervised MSE (Eq. 10), and σ is regularized toward η via KL divergence (Eq. 8). While this means the high correlations in Table 8 partly reflect supervised fitting, the mapping is not definitional: η̂ is produced by a separate network f_t, not set equal to η, and the evaluation metric (Spearman correlation with Δf, an independent cosine-distance measure) is not the training target. The generalization claims to unseen noise types (Table 7: fog, rain, snow) are empirical results, not consequences of the training equations. The paper does not invoke any self-citation chain or uniqueness theorem to force its conclusions. The framework is evaluated against external baselines (TMDC, T2DR) on multiple datasets with code provided. The mild concern—that σ is shaped to match η during training, so Table 8's correlations are not fully independent evidence of generalization—warrants a score of 2, but does not undermine the paper's central claims, which rest on comparative experiments rather than on the correlation values alone.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The framework introduces two new architectural modules (NQE, NSD) that are empirically validated. The key free parameters are the loss weights β1-β3, of which only β1 is explicitly tuned. The ground-truth noise intensity η is a constructed signal used for supervision, and its applicability to complex corruptions (fog, rain) is assumed but not rigorously defined. The axioms are domain assumptions specific to the multimodal learning problem rather than ad hoc mathematical constructions.

free parameters (4)
  • β1 = 4.0 (selected from {2.0, 3.0, 4.0, 5.0} in Table 9)
    Balances unimodal classification loss L_cls against fusion loss L_cls^f. Tuned on CREMA-D validation.
  • β2 = Not specified
    Weight for the KL regularization term L_reg. Value not stated in the paper.
  • β3 = Not specified
    Weight for the MSE loss L_mse in NQE training. Value not stated in the paper.
  • Noise intensity η_i^(v) = Continuous in [0,1] for mask; discrete levels for Gaussian
    The ground-truth degradation intensity used to supervise NQE. For mask noise it is the mask rate; for Gaussian it is the variance. Its definition for fog/rain (Table 7) is not stated.
axioms (4)
  • domain assumption Modality degradation can be represented as a continuous scalar quantity η that monotonically reflects information loss from clean to completely absent.
    This is the foundational assumption of GIML, introduced in §3.1 (Eq. 3-5). It enables the unified treatment of intra- and inter-modality missingness.
  • domain assumption The variance σ of a probabilistic embedding can be regularized to align with degradation intensity η, thereby capturing noise-induced uncertainty separately from the semantic mean μ.
    Invoked in §3.2 (Eq. 8). The KL divergence prior N(0, (η)^2 I) assumes variance tracks degradation. This is a modeling choice, not a derived result.
  • standard math Inverse-variance weighting (following ARL [46]) is the appropriate principle for computing adaptive modality fusion weights from estimated degradation.
    Used in §3.3 (Eq. 11) to compute ω from ˆη. This is a standard statistical principle applied to the specific context.
  • domain assumption Controlled noise injection during training provides sufficient supervision for the NQE to generalize to unseen noise types and intensities at test time.
    This underpins the generalization claims in §4.4-4.5. The paper provides empirical support (Tables 5-7) but no theoretical guarantee.
invented entities (2)
  • Noise-aware Quality Estimator (NQE) independent evidence
    purpose: Maps the variance σ produced by NSD to a calibrated degradation estimate ˆη via supervised learning with controlled noise injection.
    The NQE is a trained module whose predictions are evaluated against ground-truth η (Table 8 Spearman correlations) and whose ablation (Table 10) shows performance impact. It is falsifiable: if the mapping does not generalize, Tables 5-7 would show degradation.
  • Noise-Semantic Decoupled (NSD) module independent evidence
    purpose: Represents modality features as Gaussian distributions where mean captures semantics and variance captures noise, enforced by KL regularization.
    The NSD is a architectural component whose effectiveness is tested via ablation (Table 11) and semantic separability analysis (Table 12). It is falsifiable: removing it degrades performance under Gaussian noise.

pith-pipeline@v1.1.0-glm · 21061 in / 3339 out tokens · 358979 ms · 2026-07-09T22:25:47.325765+00:00 · methodology

0 comments
read the original abstract

Multimodal learning robust to missing modalities is essential for real-world applications. Existing methods mainly focus on inter-modality missing, where entire modalities are absent, while overlooking intra-modality degradation, where modalities are present but severely corrupted. In practice, these two types of missing often coexist, making existing approaches ineffective. To address this limitation, we propose General Incomplete Multimodal Learning (GIML), a unified framework that simultaneously handles both inter-modality missing and intra-modality degradation through dynamic quality perception. Specifically, GIML models heterogeneous missing patterns as continuous modality information degradation, enabling degradation-aware adaptive fusion. To achieve reliable quality perception, we introduce a Noise-aware Quality Estimator that learns the mapping from corrupted features to noise intensity through controlled noise injection. Furthermore, we propose a Noise-Semantic Decoupled module that separates semantic information from noise interference. This improves robustness and generalization to unseen corruption patterns. Extensive experiments across datasets with diverse modality types demonstrate the effectiveness and generality of GIML. Code is available at: https://github.com/Yu-Five/GIML.

Figures

Figures reproduced from arXiv: 2607.06943 by Shicai Wei, Xiangyu Meng.

Figure 1
Figure 1. Figure 1: Overall framework of the proposed GIML. It consists of two components: a Noise-Semantic Decoupled (NSD) module and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 59 canonical work pages · 6 internal anchors

  1. [1]

    Look, listen and learn

    Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. InProceedings of the IEEE international conference on computer vision, pages 609–617, 2017. 5

    work page 2017

  2. [2]

    Cav-mae sync: Improving contrastive audio-visual mask au- toencoders via fine-grained alignment

    Edson Araujo, Andrew Rouditchenko, Yuan Gong, Saurab- hchand Bhati, Samuel Thomas, Brian Kingsbury, Leonid Karlinsky, Rogerio Feris, James R Glass, and Hilde Kuehne. Cav-mae sync: Improving contrastive audio-visual mask au- toencoders via fine-grained alignment. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18794–18803, 2025. 1, 2

    work page 2025

  3. [3]

    Deep adversarial learning for multi- modality missing data completion

    Lei Cai, Zhengyang Wang, Hongyang Gao, Dinggang Shen, and Shuiwang Ji. Deep adversarial learning for multi- modality missing data completion. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1158–1166, 2018. 1, 2

    work page 2018

  4. [4]

    Multi-modal gated mixture of local-to-global experts for dy- namic image fusion

    Bing Cao, Yiming Sun, Pengfei Zhu, and Qinghua Hu. Multi-modal gated mixture of local-to-global experts for dy- namic image fusion. InProceedings of the IEEE/CVF in- ternational conference on computer vision, pages 23555– 23564, 2023. 2

    work page 2023

  5. [5]

    Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014. 5

    work page 2014

  6. [6]

    A Closer Look at Multimodal Representation Collapse

    Abhra Chaudhuri, Anjan Dutta, Tu Bui, and Serban Georgescu. A closer look at multimodal representation col- lapse.arXiv preprint arXiv:2505.22483, 2025. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Unbiased missing-modality multimodal learning

    Ruiting Dai, Chenxi Li, Yandong Yan, Lisi Mo, Ke Qin, and Tao He. Unbiased missing-modality multimodal learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 24507–24517, 2025. 1, 2

    work page 2025

  8. [8]

    A study of dropout- induced modality bias on robustness to missing video frames for audio-visual speech recognition

    Yusheng Dai, Hang Chen, Jun Du, Ruoyu Wang, Shihao Chen, Haotian Wang, and Chin-Hui Lee. A study of dropout- induced modality bias on robustness to missing video frames for audio-visual speech recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27445–27455, 2024. 1, 2

    work page 2024

  9. [9]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. InProceedings of the 2019 conference of the North American chapter of the asso- ciation for computational linguistics: human language tech- nologies, volume 1 (long and short papers), pages 4171– 4186, 2019. 6

    work page 2019

  10. [10]

    Unreliable uncertainty estimates with monte carlo dropout.arXiv preprint arXiv:2512.14851,

    Aslak Djupsk ˚as, Alexander Johannes Stasik, and Signe Riemer-Sørensen. Unreliable uncertainty estimates with monte carlo dropout.arXiv preprint arXiv:2512.14851,

    work page arXiv

  11. [11]

    Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural networks

    PMXYS Gupta, KKSTJ Kautz, et al. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural networks. InCVPR, page 3, 2016. 5

    work page 2016

  12. [12]

    Fusemoe: Mixture-of-experts transformers for flexi- modal fusion.Advances in Neural Information Processing Systems, 37:67850–67900, 2024

    Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, and Suchi Saria. Fusemoe: Mixture-of-experts transformers for flexi- modal fusion.Advances in Neural Information Processing Systems, 37:67850–67900, 2024. 2

    work page 2024

  13. [13]

    Variational Bayesian Last Layers

    James Harrison, John Willes, and Jasper Snoek. Variational bayesian last layers.arXiv preprint arXiv:2404.11599, 2024. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    Hemis: Hetero-modal image segmen- tation

    Mohammad Havaei, Nicolas Guizard, Nicolas Chapados, and Yoshua Bengio. Hemis: Hetero-modal image segmen- tation. InInternational conference on medical image com- puting and computer-assisted intervention, pages 469–477. Springer, 2016. 1, 2

    work page 2016

  15. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6

    work page 2016

  16. [16]

    Cl-moe: Enhancing multi- modal large language model with dual momentum mixture- of-experts for continual visual question answering

    Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. Cl-moe: Enhancing multi- modal large language model with dual momentum mixture- of-experts for continual visual question answering. InPro- ceedings of the Computer Vision and Pattern Recognition Conference, pages 19608–19617, 2025. 2

    work page 2025

  17. [17]

    RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition

    Yuehan Jin, Xiaoqing Liu, Yiyuan Yang, Zhiwen Yu, Tong Zhang, and Kaixiang Yang. Rohydr: Robust hybrid diffu- sion recovery for incomplete multimodal emotion recogni- tion.arXiv preprint arXiv:2505.17501, 2025. 1, 2

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models

    Linglin Jing, Yuting Gao, Zhigang Wang, Wang Lan, Yi- wen Tang, Wenhai Wang, Kaipeng Zhang, and Qingpei Guo. Evomoe: Expert evolution in mixture of experts for multimodal large language models.arXiv preprint arXiv:2505.23830, 2025. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [19]

    Integrating cross-modality hallucinated mri with ct to aid mediastinal lung tumor segmentation

    Jiang Jue, Hu Jason, Tyagi Neelam, Rimner Andreas, Berry L Sean, Deasy O Joseph, and Veeraraghavan Harini. Integrating cross-modality hallucinated mri with ct to aid mediastinal lung tumor segmentation. InInternational con- ference on medical image computing and computer-assisted intervention, pages 221–229. Springer, 2019. 1, 2

    work page 2019

  20. [20]

    Crossfuse: A novel cross atten- tion mechanism based infrared and visible image fusion ap- proach.Information Fusion, 103:102147, 2024

    Hui Li and Xiao-Jun Wu. Crossfuse: A novel cross atten- tion mechanism based infrared and visible image fusion ap- proach.Information Fusion, 103:102147, 2024. 2

    work page 2024

  21. [21]

    Simmlm: A simple framework for multi-modal learning with missing modality

    Sijie Li, Chen Chen, and Jungong Han. Simmlm: A simple framework for multi-modal learning with missing modality. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 24068–24077, 2025. 2

    work page 2025

  22. [22]

    Xiao Li, Lin Lei, Yuli Sun, and Gangyao Kuang. Dynamic- hierarchical attention distillation with synergetic instance se- lection for land cover classification using missing hetero- geneity images.IEEE Transactions on Geoscience and Re- mote Sensing, 60:1–16, 2021. 1

    work page 2021

  23. [23]

    Multibench: Multiscale benchmarks for multimodal representation learning.Advances in neural in- formation processing systems, 2021(DB1):1, 2021

    Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning.Advances in neural in- formation processing systems, 2021(DB1):1, 2021. 1

    work page 2021

  24. [24]

    T2dr: A two-tier deficiency-resistant framework for incomplete multimodal learning

    Han Lin, Xiu Tang, Huan Li, Wenxue Cao, Sai Wu, Chang Yao, Lidan Shou, and Gang Chen. T2dr: A two-tier deficiency-resistant framework for incomplete multimodal learning. InFindings of the Association for Computational Linguistics: ACL 2025, pages 8602–8616, 2025. 1, 2, 6

    work page 2025

  25. [25]

    Intra-class probabilistic embeddings for uncer- tainty estimation in vision-language models.arXiv preprint arXiv:2511.22019, 2025

    Zhenxiang Lin, Maryam Haghighat, Will Browne, and Dim- ity Miller. Intra-class probabilistic embeddings for uncer- tainty estimation in vision-language models.arXiv preprint arXiv:2511.22019, 2025. 2

    work page arXiv 2025

  26. [26]

    Face anti-spoofing via ad- versarial cross-modality translation.IEEE Transactions on Information Forensics and Security, 16:2759–2772, 2021

    Ajian Liu, Zichang Tan, Jun Wan, Yanyan Liang, Zhen Lei, Guodong Guo, and Stan Z Li. Face anti-spoofing via ad- versarial cross-modality translation.IEEE Transactions on Information Forensics and Security, 16:2759–2772, 2021. 1, 2

    work page 2021

  27. [27]

    Energy-based out-of-distribution detection.Advances in neural information processing systems, 33:21464–21475,

    Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection.Advances in neural information processing systems, 33:21464–21475,

    work page

  28. [28]

    Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers

    Zhengyao Lv, Tianlin Pan, Chenyang Si, Zhaoxi Chen, Wangmeng Zuo, Ziwei Liu, and Kwan-Yee K Wong. Re- thinking cross-modal interaction in multimodal diffusion transformers.arXiv preprint arXiv:2506.07986, 2025. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  29. [29]

    Sentiment analysis on multi-view social data

    Teng Niu, Shiai Zhu, Lei Pang, and Abdulmotaleb El Sad- dik. Sentiment analysis on multi-view social data. InInter- national conference on multimedia modeling, pages 15–27. Springer, 2016. 5

    work page 2016

  30. [30]

    Balanced multimodal learning via on-the-fly gradient modulation

    Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8238–8247,

    work page

  31. [31]

    Face spoofing detection through visual codebooks of spectral temporal cubes.IEEE Transactions on image processing, 24(12):4726–4740, 2015

    Allan Pinto, Helio Pedrini, William Robson Schwartz, and Anderson Rocha. Face spoofing detection through visual codebooks of spectral temporal cubes.IEEE Transactions on image processing, 24(12):4726–4740, 2015. 1

    work page 2015

  32. [32]

    Recursive joint cross- modal attention for multimodal fusion in dimensional emo- tion recognition

    R Gnana Praveen and Jahangir Alam. Recursive joint cross- modal attention for multimodal fusion in dimensional emo- tion recognition. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 4803–4813, 2024. 2

    work page 2024

  33. [33]

    Probabilistic face embeddings

    Yichun Shi and Anil K Jain. Probabilistic face embeddings. InProceedings of the IEEE/CVF international conference on computer vision, pages 6902–6911, 2019. 2, 4

    work page 2019

  34. [34]

    Ogp- net: Optical guidance meets pixel-level contrastive distilla- tion for robust multi-modal and missing modality segmen- tation

    Aniruddh Sikdar, Jayant Teotia, and Suresh Sundaram. Ogp- net: Optical guidance meets pixel-level contrastive distilla- tion for robust multi-modal and missing modality segmen- tation. InProceedings of the AAAI Conference on Artificial Intelligence, pages 6922–6930, 2025. 1, 2

    work page 2025

  35. [35]

    D3d: Distilled 3d networks for video action recognition

    Jonathan Stroud, David Ross, Chen Sun, Jia Deng, and Rahul Sukthankar. D3d: Distilled 3d networks for video action recognition. InProceedings of the IEEE/CVF winter con- ference on applications of computer vision, pages 625–634,

    work page

  36. [36]

    Redcore: Relative advantage aware cross-modal representation learning for missing modalities with imbal- anced missing rates

    Jun Sun, Xinxin Zhang, Shoukang Han, Yu-Ping Ruan, and Taihao Li. Redcore: Relative advantage aware cross-modal representation learning for missing modalities with imbal- anced missing rates. InProceedings of the AAAI Conference on Artificial Intelligence, pages 15173–15182, 2024. 1, 2

    work page 2024

  37. [37]

    Generative multimodal mod- els are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal mod- els are in-context learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14398–14409, 2024. 1

    work page 2024

  38. [38]

    Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models

    Aishwarya Venkataramanan, Paul Bodesheim, and Joachim Denzler. Probabilistic embeddings for frozen vision- language models: Uncertainty quantification with gaus- sian process latent variable models.arXiv preprint arXiv:2505.05163, 2025. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  39. [39]

    Learnable cross- modal knowledge distillation for multi-modal learning with missing modality

    Hu Wang, Congbo Ma, Jianpeng Zhang, Yuan Zhang, Jodie Avery, Louise Hull, and Gustavo Carneiro. Learnable cross- modal knowledge distillation for multi-modal learning with missing modality. InInternational Conference on Medi- cal Image Computing and Computer-Assisted Intervention, pages 216–226. Springer, 2023. 1, 2

    work page 2023

  40. [40]

    What makes train- ing multi-modal classification networks hard? InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695–12705, 2020

    Weiyao Wang, Du Tran, and Matt Feiszli. What makes train- ing multi-modal classification networks hard? InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695–12705, 2020. 2

    work page 2020

  41. [41]

    Mmanet: Margin- aware distillation and modality-aware regularization for in- complete multimodal learning

    Shicai Wei, Chunbo Luo, and Yang Luo. Mmanet: Margin- aware distillation and modality-aware regularization for in- complete multimodal learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20039–20049, 2023. 1

    work page 2023

  42. [42]

    Shicai Wei, Yang Luo, Xiaoguang Ma, Peng Ren, and Chunbo Luo. MSH-Net: Modality-shared hallucination with joint adaptation distillation for remote sensing image clas- sification using missing modalities.IEEE Transactions on Geoscience and Remote Sensing, 61:1–15, 2023. 2

    work page 2023

  43. [43]

    Gradient decoupled learning with unimodal regularization for multimodal remote sensing classification.IEEE Trans- actions on Geoscience and Remote Sensing, 62:1–12, 2024

    Shicai Wei, Chunbo Luo, Xiaoguang Ma, and Yang Luo. Gradient decoupled learning with unimodal regularization for multimodal remote sensing classification.IEEE Trans- actions on Geoscience and Remote Sensing, 62:1–12, 2024. 1

    work page 2024

  44. [44]

    Ro- bust multimodal learning via representation decoupling

    Shicai Wei, Yang Luo, Yuji Wang, and Chunbo Luo. Ro- bust multimodal learning via representation decoupling. In European Conference on Computer Vision, pages 38–54. Springer, 2024. 1, 2, 4

    work page 2024

  45. [45]

    Boosting mul- timodal learning via disentangled gradient learning

    Shicai Wei, Chunbo Luo, and Yang Luo. Boosting mul- timodal learning via disentangled gradient learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22879–22888, 2025. 2

    work page 2025

  46. [46]

    Improving mul- timodal learning via imbalanced learning

    Shicai Wei, Chunbo Luo, and Yang Luo. Improving mul- timodal learning via imbalanced learning. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 2250–2259, 2025. 2, 5

    work page 2025

  47. [47]

    Unbiased dynamic multimodal fusion

    Shicai Wei, Kaijie Zhang, Luyi Chen, Tao He, and Guiduo Duan. Unbiased dynamic multimodal fusion. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6239–6249, 2026. 2

    work page 2026

  48. [48]

    Be- yond simple fusion: Adaptive gated fusion for robust multi- modal sentiment analysis.arXiv preprint arXiv:2510.01677,

    Han Wu, Yanming Sun, Yunhe Yang, and Derek F Wong. Be- yond simple fusion: Adaptive gated fusion for robust multi- modal sentiment analysis.arXiv preprint arXiv:2510.01677,

    work page arXiv

  49. [49]

    Leveraging knowledge of modality experts for incomplete multimodal learning

    Wenxin Xu, Hexin Jiang, and Xuefeng Liang. Leveraging knowledge of modality experts for incomplete multimodal learning. InProceedings of the 32nd ACM International Conference on Multimedia, pages 438–446, 2024. 2

    work page 2024

  50. [50]

    Rui Yang, Jie Wang, Guoping Wu, and Bin Li. Uncertainty- based offline variational bayesian reinforcement learning for robustness under diverse data corruptions.Advances in Neu- ral Information Processing Systems, 37:39748–39783, 2024. 2

    work page 2024

  51. [51]

    Multi-attention recurrent network for human communication comprehen- sion

    Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehen- sion. InProceedings of the AAAI conference on artificial intelligence, 2018. 5

    work page 2018

  52. [52]

    Rate-in: Information-driven adaptive dropout rates for improved inference-time uncer- tainty estimation

    Tal Zeevi, Ravid Shwartz-Ziv, Yann LeCun, Lawrence H Staib, and John A Onofrey. Rate-in: Information-driven adaptive dropout rates for improved inference-time uncer- tainty estimation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 20757–20766, 2025. 2

    work page 2025

  53. [53]

    Towards ro- bust multimodal sentiment analysis with incomplete data

    Haoyu Zhang, Wenbin Wang, and Tianshu Yu. Towards ro- bust multimodal sentiment analysis with incomplete data. Advances in Neural Information Processing Systems, 37: 55943–55974, 2024. 2

    work page 2024

  54. [54]

    Provable dynamic fusion for low-quality multimodal data

    Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. InInterna- tional conference on machine learning, pages 41753–41769. PMLR, 2023. 5

    work page 2023

  55. [55]

    mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmenta- tion

    Yao Zhang, Nanjun He, Jiawei Yang, Yuexiang Li, Dong Wei, Yawen Huang, Yang Zhang, Zhiqiang He, and Yefeng Zheng. mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmenta- tion. InInternational conference on medical image com- puting and computer-assisted intervention, pages 107–117. Springer, 2022. 1, 2

    work page 2022

  56. [56]

    Incomplete multi-view clustering via diffusion contrastive generation

    Yuanyang Zhang, Yijie Lin, Weiqing Yan, Li Yao, Xinhang Wan, Guangyuan Li, Chao Zhang, Guanzhou Ke, and Jie Xu. Incomplete multi-view clustering via diffusion contrastive generation. InProceedings of the AAAI Conference on Arti- ficial Intelligence, pages 22650–22658, 2025. 1, 2

    work page 2025

  57. [57]

    Proxy-driven robust multimodal sen- timent analysis with incomplete data

    Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yim- ing Tang, and Ning An. Proxy-driven robust multimodal sen- timent analysis with incomplete data. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22123–22138,

    work page

  58. [58]

    Hyper-modality enhancement for multimodal sentiment analysis with missing modalities

    Yan Zhuang, LIU Minhao, Wei Bai, Yanru Zhang, Wei Li, Ji- awen Deng, and Fuji Ren. Hyper-modality enhancement for multimodal sentiment analysis with missing modalities. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2

    work page

  59. [59]

    Tmdc: A two-stage modality denoising and complementation framework for multimodal sentiment anal- ysis with missing and noisy modalities.arXiv preprint arXiv:2511.10325, 2025

    Yan Zhuang, Minhao Liu, Yanru Zhang, Jiawen Deng, and Fuji Ren. Tmdc: A two-stage modality denoising and complementation framework for multimodal sentiment anal- ysis with missing and noisy modalities.arXiv preprint arXiv:2511.10325, 2025. 1, 2, 6

    work page arXiv 2025