Pith. sign in

REVIEW 4 major objections 5 minor 72 references

MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MIDAS claims that incomplete multimodal sentiment inputs can be handled by disentangling each modality into shared and exclusive Gaussian latents and fusing them by posterior uncertainty, without imputing missing data.

desk verdict Sound architecture, weak evidence: single-run comparisons don't support the SOTA claim, but the paper is coherent and deserves refereeing. read the letter →

arxiv 2608.09986 v1 pith:OHTAPGMK submitted 2026-08-06 cs.AI cs.LG

Yuhua Wen , Yingying Zhou , Qifei Li , Yingming Gao , Zhengqi Wen , Jianhua Tao , Ya Li This is my paper · ORCID
classification cs.AIcs.LG
keywords multimodalsentimentanalysisincompletemodalitiesmutualinformationdisentangledrepresentationlearningvariationalinferenceuncertainty-awarefusionmissingdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MIDAS tackles multimodal sentiment analysis when one or more input channels—text, audio, or video—are missing or corrupted. The paper's claim is that a single model can stay accurate from fully intact input up to heavy missingness by restructuring each modality into shared and private Gaussian latent factors, then fusing them by their learned uncertainty rather than treating all channels equally. The authors argue this beats both data-imputation methods, which reconstruct missing inputs first, and coordination methods, which try to keep representations consistent across modality combinations. Across three public datasets and missing rates from 0% to 90%, MIDAS reports consistent gains over ten baselines. If true, it would make sentiment systems usable in real deployments where sensors fail, connections are poor, or privacy filters remove parts of the signal.

What carries the argument

The load-bearing mechanism is a mutual-information minimax over disentangled Gaussian latents, paired with reliability-modulated attention. Each modality is represented by shared and exclusive multivariate Gaussian posteriors; the objective minimizes mutual information within a modality between the shared and exclusive factors via a variational upper bound, and maximizes mutual information across modalities between shared factors via a Jensen–Shannon divergence estimator with a critic. The variance of each posterior is converted by differential entropy and a temperature-scaled sigmoid into a reliability weight, and the weights scale the key vectors inside a Transformer-style attention block so uncertain tokens contribute less to fusion. This is what carries the claim that the model can maintain semantic alignment and robust prediction when modalities are partly erased.

What would settle it

The claim would be falsified by an ablation that replaces the uncertainty weights with uniform attention and finds the averaged accuracy across missing rates from 0.0 to 0.9 changes by less than noise; then the posterior-variance weighting is not what produces the reported robustness.

Watch

Extended reading notes

Core claim

The central discovery, as the paper states it, is that incomplete multimodal inputs are best handled by reorganizing the latent space rather than filling in missing data or imposing geometric constraints. Each modality is encoded as a pair of multivariate Gaussians—a shared factor meant to carry modality-invariant sentiment semantics and an exclusive factor meant to carry modality-specific detail and noise. A minimax mutual-information objective minimizes the dependence between shared and exclusive factors within a modality while maximizing the dependence among shared factors across modalities, so the shared space becomes semantically aligned. Prediction and reconstruction losses anchor the latents to the task. Fusion then uses the posterior variance of each latent token as a reliability weight inside attention, so tokens whose distributions are broad—taken as a sign of missing or unreliable information—are downweighted. The paper's experiments on MOSI, MOSEI, and CH-SIMS show MIDAS outperforming all compared methods on most metrics averaged across missing levels, with the largest margins on the Chinese CH-SIMS dataset.

Load-bearing premise

The load-bearing premise is that the variance of the learned Gaussian posterior is a faithful signal of how unreliable that token is, so that downweighting high-variance tokens improves fusion instead of discarding useful information.

Editorial extensions

If this is right

  • A single MIDAS checkpoint trained on mixed missing rates can be deployed under heterogeneous sensor failures without retraining per missing pattern, since the model is evaluated on all rates with one selected checkpoint.
  • Downweighting unreliable tokens during fusion should make predictions degrade more slowly as missingness increases, which is the smooth performance curve the paper reports.
  • Because shared latents are forced to align across modalities, the model can lean on whichever channels remain, so severe loss in one channel should not be able to dominate or silently corrupt the final prediction.
  • The uncertainty weights come from the variational posterior itself, so no auxiliary uncertainty estimator or Monte Carlo sampling is needed at inference.
  • The method's parameter count stays low beyond the text encoder while giving the reported accuracy, suggesting that robust handling of missingness does not require a larger model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the same uncertainty weighting could transfer to other multimodal tasks—emotion recognition or human-computer interaction—where a channel can be unreliable for reasons other than missingness, such as loud acoustic noise or blurred video, provided the variance estimate remains calibrated to that noise.
  • My inference: the reliability-weight mechanism is only as good as the learned variance; if the encoder's variance collapses or becomes scale-invariant across inputs, the attention modulation degenerates. A cheap probe would be to compare prediction accuracy under uniform weights versus learned weights on held-out missing patterns.
  • My inference: the paper's benchmark fixes the same missing rate across all modalities. Asymmetric missing patterns—say, 80% of audio missing but only 10% of text—are the harder real-world case, and the method's disentanglement objective is likely where any advantage would show or fail.
  • My inference: the claim that posterior variance tracks reliability could be tested by correlating per-token variance with whether that token's input region was actually masked; a strong correlation would make the mechanism auditable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MIDAS, a framework for incomplete multimodal sentiment analysis. Each modality is encoded into a pair of multivariate Gaussian latent variables, factorized into shared and exclusive components. A mutual-information minimax objective is used to disentangle these components and align shared spaces across modalities, and an uncertainty-aware fusion module computes reliability weights from posterior variance and modulates attention keys. The authors train a single model under stochastic feature masking with rates 0.0–0.9 and evaluate on MOSI, MOSEI, and CH-SIMS, reporting aggregate robustness metrics, ablations, an efficiency analysis, and a qualitative case study.

Significance. If the claims are supported, the paper makes a useful contribution: it addresses incomplete multimodal sentiment analysis without imputation, combines a variational latent formulation with MI-based disentanglement, and uses intrinsic posterior uncertainty for fusion. The paper also releases code, evaluates a single unified model across missing rates, compares against both complete and incomplete baselines, and includes component-level ablations and a complexity analysis. These are genuine strengths. However, the central empirical claim of 'strong and consistent performance gains' is not yet established because all comparisons are single-run, the theoretical bound in Eq. (9) is not a valid variational upper bound as written, and the reliability assumption behind the uncertainty weighting is asserted rather than validated. The contribution is therefore conditional on resolving these load-bearing issues.

major comments (4)
  1. [Section V-A, Tables II and III] The central claim of strong and consistent improvements rests on single-run comparisons. No seed variance, confidence intervals, or paired significance tests are reported anywhere, and several margins over the strongest baseline are very small: on MOSI, Acc-2 improves by 0.84 over EMT-DLFR but Acc-5 is worse (35.12 vs 35.19), and MAE is worse than Self-MM/TETFN/TFR-Net (1.074 vs 1.065); on MOSEI, the Acc-2 gain over EMT-DLFR is only 0.41. In MSA benchmarks, seed-level deviations are typically of the same order as these margins, so the reported numbers do not by themselves establish that MIDAS outperforms the baselines. Please report at least 3–5 seeds per method and dataset, provide confidence intervals or paired tests for the aggregated metrics, and discuss per-missing-rate variability; the abstract and Section I should then be reworded to match the strength of the evidence.
  2. [Section III-C, Eq. (9)] The derivation of the variational upper bound is not valid as written. Starting from Eq. (8), the KL-divergence terms can upper-bound I(Z_s_m; X_m) and I(Z_e_m; X_m), but subtracting an expected log-likelihood term does not guarantee an upper bound on I(Z_s_m; Z_e_m); when the reconstruction likelihood is high, the RHS can become negative and even unbounded below. In addition, the expectation notation in Eq. (9), 'E_{p(Z_s,Z_e)} E_{q(Z_s|X)q(Z_e|X)}', is ambiguous because X is not specified. The authors should state the precise variational identity being used, or clearly identify Eq. (9) as a heuristic objective rather than a bound, since the theoretical grounding of L_dis depends on this step.
  3. [Section III-D, Eqs. (15)–(17)] The core assumption that posterior variance of the variational latents is a valid reliability indicator for fusion is asserted without calibration or empirical validation. Under random feature masking, the variance may reflect encoder sensitivity, initialization, or optimization state rather than true predictive uncertainty. If high variance does not track low reliability, the reliability-modulated attention in Eq. (17) can downweight informative tokens and the advantage of the uncertainty mechanism would vanish. The authors should provide evidence that the estimated uncertainties are calibrated, for example by showing that uncertainty increases with missing rate and correlates with per-sample prediction error, or by comparing against an oracle reliability weighting that uses the true missingness or error.
  4. [Section III-C, Eq. (7)] The simplification that I(Z_s_m; Z_e_m | X_m) = 0 is an independence assumption, not a consequence of the variational factorization, and it is not stated explicitly as an assumption. Conditioning on X_m does not by itself make the shared and exclusive latents conditionally independent. If this conditional-independence assumption fails, the third term in Eq. (7) cannot be dropped and the subsequent expansion in Eq. (8) is incomplete. Please state this assumption explicitly and justify it, or keep the conditional mutual-information term and explain how it is handled.
minor comments (5)
  1. [Section III-C, after Eq. (9)] The sentence 'Here, p(X_m) is typically chosen as a standard Gaussian prior' appears to be a typo: the priors should be p(Z_s_m) and p(Z_e_m), not p(X_m).
  2. [Table VI] The F1 value listed for EMT-DLFR (69.39/67.58) is identical to that of TFR-Net and differs from the EMT-DLFR F1 reported in Table II (70.87/70.18); this appears to be a transcription error and should be corrected.
  3. [Fig. 2 and Section I] There are several typos, including 'encocding' and 'vatiational' in the Fig. 2 caption, 'as as a measure' in Section I, and 'uncertainity' in the Index Terms; a careful proofreading pass is needed.
  4. [Section III-D, Eq. (17)] The notation in Eq. (17) is unclear: W is defined as a 6-dimensional vector, but the expression Q(√W K)^T requires a precise definition of how the vector is expanded across the key dimension and how the square root is applied; please clarify.
  5. [Section V-C, 'Training Stability and Co-optimization'] The claim that the five objectives 'act in a complementary rather than adversarial manner' is presented without quantitative evidence; adding loss curves or gradient-conflict measurements would make this statement verifiable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is self-contained and the reported gains are empirical.

full rationale

MIDAS's derivation chain does not reduce to its inputs. Variational modeling (Eqs. 3-5) and the MI disentanglement objective are standard variational bounds with explicitly stated assumptions (conditional independence after Eq. 7; JSD estimator from Deep InfoMax in Eq. 11). The uncertainty-aware fusion uses posterior differential entropy (Eq. 15) as a heuristic reliability weight (Eq. 16) and modulates attention keys (Eq. 17); this is a design choice, not a fitted prediction renamed as a result. Reported metrics in Tables II and III are held-out test measurements, not quantities forced by construction. The paper's self-citations (e.g., [9], [28]) appear in related work and baseline descriptions and are not load-bearing; no uniqueness theorem or prior result by the same authors is invoked to justify the central method. Concerns about single-run comparisons and lack of significance tests concern statistical evidence, which is a correctness risk rather than circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 3 invented entities

The model introduces latent factors and a reliability weighting mechanism, all validated only through downstream accuracy. The loss weights and temperature are hand-chosen. The main domain assumptions, that posterior variance tracks reliability and that feature masking proxies missing modalities, are not independently verified.

free parameters (6)
  • Loss weight alpha (disentanglement) = 0.05
    Chosen by hand and kept constant across datasets, as reported in Table I.
  • Loss weight beta (alignment) = 1.0
    Chosen by hand, Table I.
  • Loss weight gamma (prediction) = 0.1
    Chosen by hand, Table I.
  • Loss weight lambda (reconstruction) = 1.0
    Chosen by hand, Table I.
  • Temperature tau for uncertainty weighting = 10
    Chosen by hand to control the sigmoid steepness in Eq. (16).
  • Latent dimension d = 128
    Architectural choice used for all datasets, Table I.
assumptions (4)
  • standard math Mutual information identities and variational bounds, including the chain rule, interaction information, DKL upper bound, and JSD estimator, hold as stated.
    Used in Section III-C to derive Eqs. (6) through (11); these are standard results from information theory.
  • domain assumption The variational posteriors factorize as q(Z^s_m|X_m) q(Z^e_m|X_m), so I(Z^s_m;Z^e_m|X_m) = 0.
    Stated in Section III-C.1 to simplify Eq. (7); the independence holds by the sampling scheme in Eq. (5).
  • ad hoc to paper Posterior variance of the latent Gaussians is a valid reliability indicator for fusion.
    Assumed in Section III-D and Eqs. (15) and (16); no calibration or external validation is provided.
  • domain assumption Random feature masking, with zeros for audio and visual and [UNK] for text, simulates real-world incomplete multimodal inputs.
    Introduced in Section III-A and used in all experiments; whole-modality absence is not tested.
invented entities (3)
  • Shared latent factor z^s_m
    purpose: Captures modality-invariant semantic content for alignment and fusion.
    Introduced in Section III-B; the only evidence is the model's own performance on benchmarks, with no external falsifiable prediction.
  • Exclusive latent factor z^e_m
    purpose: Retains modality-specific details and is encouraged to be independent of the shared factor.
    Same as the shared factor; validated only through downstream accuracy and ablations.
  • Uncertainty weight w_j
    purpose: Modulates attention keys to downweight unreliable tokens during fusion.
    Derived from learned posterior variances in Eq. (16); no ground-truth uncertainty comparison is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis." pith.science (2026). https://pith.science/paper/OHTAPGMK

@misc{pith2026260809986,
  author       = {Pith},
  title        = {Pith review of: MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OHTAPGMK}},
  note         = {Machine review of arXiv:2608.09986}
}
read the original abstract

Most existing multimodal sentiment analysis approaches assume access to complete multimodal inputs. However, real-world applications frequently encounter incomplete or corrupted modalities, posing a critical challenge. Although several methods have been proposed to tackle this issue, they mainly rely on data imputation and heuristic coordination constraints, which fail to effectively extract and leverage task-relevant information from the incomplete multimodal data. To address this challenge, we propose a unified framework termed Mutual Information Disentanglement with uncertainty-Aware fuSion (MIDAS), which effectively restructures multimodal representations under incomplete conditions. MIDAS adopts a variational modeling strategy to represent each modality with multivariate Gaussian latent variables and further decomposes them into shared and exclusive factors. To obtain reliable representations, we design a minimax objective that minimizes the mutual information between shared and exclusive spaces for stable disentanglement, while maximizing the mutual information among shared spaces across modalities to enhance semantic alignment. In addition, an uncertainty-aware fusion mechanism is introduced, where posterior variance is leveraged as a reliability indicator to adaptively weight latent features during fusion, ensuring robust integration even when modalities are incomplete. Extensive experiments on three widely used datasets show that MIDAS achieves strong and consistent performance gains over competitive baselines across a wide range of incomplete settings, demonstrating its effectiveness and robustness for incomplete data scenarios.

Figures

Figures reproduced from arXiv: 2608.09986 by the authors.

Figure 1
Figure 1. Illustration of incomplete multimodal sentiment analysis in real-world [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall architecture of the MIDAS for incomplete multimodal sentiment analysis. It consist of modality encocding, vatiational modeling, MI [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The framework of the proposed Uncertainty-aware Fusion. Posterior uncertainty estimates derived from latent variances are mapped to reliability [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance curves of varying missing rates on three datasets. Each row corresponds to a dataset (MOSI, MOSEI, CH-SIMS), and each column [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Visualization of example predictions by MIDAS and MISA under varying missing rates (10%, 50%, and 90%) from the MOSI dataset. Note: The [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 56 canonical work pages

  1. [62]

    Feddcsr: Federated cross-domain sequential recommendation via disentangled representa- tion learning,

    H. Zhang, D. Zheng, X. Yang, J. Feng, and Q. Liao, “Feddcsr: Federated cross-domain sequential recommendation via disentangled representa- tion learning,” inProc. 2024 SIAM Int. Conf. Data Min. (SDM), 2024, pp. 535–543

    work page 2024

  2. [63]

    Variational interaction information maximization for cross-domain disentanglement,

    H. Hwang, G.-H. Kim, S. Hong, and K.-E. Kim, “Variational interaction information maximization for cross-domain disentanglement,”Proc. Int. Conf. Neural Inf. Process. Syst., vol. 33, pp. 22 479–22 491, 2020

    work page 2020

  3. [23]

    Embracing unimodal aleatoric uncertainty for robust multimodal fusion,

    Z. Gao, X. Jiang, X. Xu, F. Shen, Y . Li, and H. T. Shen, “Embracing unimodal aleatoric uncertainty for robust multimodal fusion,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2024, pp. 26 876– 26 885

    work page 2024

  4. [58]

    Cold fusion: Calibrated and ordinal latent dis- tribution fusion for uncertainty-aware multimodal emotion recognition,

    M. K. Tellamekala, S. Amiriparian, B. W. Schuller, E. Andr ´e, T. Gies- brecht, and M. Valstar, “Cold fusion: Calibrated and ordinal latent dis- tribution fusion for uncertainty-aware multimodal emotion recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 2, pp. 805–822, 2024

    work page 2024

  5. [1]

    Multimodal sentiment analysis: a survey of methods, trends, and challenges,

    R. Das and T. D. Singh, “Multimodal sentiment analysis: a survey of methods, trends, and challenges,”ACM Comput. Surv., vol. 55, no. 13s, pp. 1–38, 2023

    work page 2023

  6. [2]

    Multi- modal sentiment analysis: A systematic review of history, datasets, mul- timodal fusion methods, applications, challenges and future directions,

    A. Gandhi, K. Adhvaryu, S. Poria, E. Cambria, and A. Hussain, “Multi- modal sentiment analysis: A systematic review of history, datasets, mul- timodal fusion methods, applications, challenges and future directions,” Inf. Fusion, vol. 91, pp. 424–444, 2023

    work page 2023

  7. [3]

    Deep multimodal learn- ing with missing modality: A survey,

    R. Wu, H. Wang, H.-T. Chen, and G. Carneiro, “Deep multimodal learn- ing with missing modality: A survey,”arXiv preprint arXiv:2409.07825, 2024

    arXiv 2024

  8. [4]

    A review of multimodal-based emotion recognition techniques for cyberbullying detection in online social media platforms,

    S. Wang, A. S. Shibghatullah, T. J. Iqbal, and K. H. Keoy, “A review of multimodal-based emotion recognition techniques for cyberbullying detection in online social media platforms,”Neural Comput. Appl., vol. 36, no. 35, pp. 21 923–21 956, 2024

    work page 2024

Show all 72 references
  1. [5]

    Emotion recognition framework using multiple modalities for an effective human–computer interaction,

    A. Moin, F. Aadil, Z. Ali, and D. Kang, “Emotion recognition framework using multiple modalities for an effective human–computer interaction,” J. Supercomput., vol. 79, no. 8, pp. 9320–9349, 2023

    2023

  2. [6]

    Ecmer: edge-cloud collaborative personalized multimodal emotion recognition framework in the internet of vehicles,

    P. Zhang, M. Fu, R. Zhao, D. Wu, H. Zhang, Z. Yang, and R. Wang, “Ecmer: edge-cloud collaborative personalized multimodal emotion recognition framework in the internet of vehicles,”IEEE Netw., vol. 37, no. 4, pp. 192–199, 2023

    2023

  3. [7]

    Multimodal reconstruct and align net for missing modality problem in sentiment analysis,

    W. Luo, M. Xu, and H. Lai, “Multimodal reconstruct and align net for missing modality problem in sentiment analysis,” inInt. Conf. Multimedia Model.Springer, 2023, pp. 411–422

    2023

  4. [8]

    Transformer-based feature recon- struction network for robust multimodal sentiment analysis,

    Z. Yuan, W. Li, H. Xu, and W. Yu, “Transformer-based feature recon- struction network for robust multimodal sentiment analysis,” inProc. 29th ACM Int. Conf. Multimedia, 2021, pp. 4400–4407

    2021

  5. [9]

    Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis,

    L. Sun, Z. Lian, B. Liu, and J. Tao, “Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis,”IEEE Trans. Affective Comput., vol. 15, no. 1, pp. 309–325, 2023

    2023

  6. [10]

    Contrastive learn- ing based modality-invariant feature acquisition for robust multimodal emotion recognition with missing modalities,

    R. Liu, H. Zuo, Z. Lian, B. W. Schuller, and H. Li, “Contrastive learn- ing based modality-invariant feature acquisition for robust multimodal emotion recognition with missing modalities,”IEEE Trans. Affective Comput., vol. 15, no. 4, pp. 1856–1873, 2024

    2024

  7. [11]

    Noise imitation based adver- sarial training for robust multimodal sentiment analysis,

    Z. Yuan, Y . Liu, H. Xu, and K. Gao, “Noise imitation based adver- sarial training for robust multimodal sentiment analysis,”IEEE Trans. Multimedia, vol. 26, pp. 529–539, 2023

    2023

  8. [12]

    Distribution-consistent modal recovering for incomplete multimodal learning,

    Y . Wang, Z. Cui, and Y . Li, “Distribution-consistent modal recovering for incomplete multimodal learning,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023, pp. 22 025–22 034

    2023

  9. [13]

    Missing modality imagination network for emotion recognition with uncertain missing modalities,

    J. Zhao, R. Li, and Q. Jin, “Missing modality imagination network for emotion recognition with uncertain missing modalities,” inProc. 59th Annu. Meeting Assoc. Comput. Linguistic and 11th Inter. Joint Conf. Natural Lang. Process, 2021, pp. 2608–2618

    2021

  10. [14]

    Modality translation- based multimodal sentiment analysis under uncertain missing modali- ties,

    Z. Liu, B. Zhou, D. Chu, Y . Sun, and L. Meng, “Modality translation- based multimodal sentiment analysis under uncertain missing modali- ties,”Inf. Fusion, vol. 101, p. 101973, 2024

    2024

  11. [15]

    A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities,

    M. Li, D. Yang, Y . Lei, S. Wang, S. Wang, L. Su, K. Yang, Y . Wang, M. Sun, and L. Zhang, “A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities,” in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 9, 2024, pp. 10 074–10 082

    2024

  12. [16]

    Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities,

    M. Li, D. Yang, X. Zhao, S. Wang, Y . Wang, K. Yang, M. Sun, D. Kou, Z. Qian, and L. Zhang, “Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2024, pp. 12 458– 12 468

    2024

  13. [17]

    Towards robust multimodal sentiment analysis with incomplete data,

    H. Zhang, W. Wang, and T. Yu, “Towards robust multimodal sentiment analysis with incomplete data,” inProc. Int. Conf. Neural Inf. Process. Syst., 2024

    2024

  14. [18]

    Disentangled rep- resentation learning,

    X. Wang, H. Chen, S. Tang, Z. Wu, and W. Zhu, “Disentangled rep- resentation learning,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 12, pp. 9677–9696, 2024

    2024

  15. [19]

    Misa: Modality-invariant and-specific representations for multimodal sentiment analysis,

    D. Hazarika, R. Zimmermann, and S. Poria, “Misa: Modality-invariant and-specific representations for multimodal sentiment analysis,” inProc. 28th ACM Int. Conf. Multimedia, 2020, pp. 1122–1131

    2020

  16. [20]

    Disentangled representation learning for multimodal emotion recognition,

    D. Yang, S. Huang, H. Kuang, Y . Du, and L. Zhang, “Disentangled representation learning for multimodal emotion recognition,” inProc. 30th ACM Inter. Conf. Multimedia, 2022, pp. 1642–1651

    2022

  17. [21]

    Mul- timodal sentiment analysis with mutual information-based disentangled representation learning,

    H. Sun, Z. Niu, H. Wang, X. Yu, J. Liu, Y .-W. Chen, and L. Lin, “Mul- timodal sentiment analysis with mutual information-based disentangled representation learning,”IEEE Trans. Affective Comput., 2025

    2025

  18. [22]

    Proxy- driven robust multimodal sentiment analysis with incomplete data,

    A. Zhu, M. Hu, X. Wang, J. Yang, Y . Tang, and N. An, “Proxy- driven robust multimodal sentiment analysis with incomplete data,” in Proc. 63rd Annu. Meeting Assoc. Comput. Linguistics (Volume 1: Long Papers), 2025, pp. 22 123–22 138

    2025

  19. [24]

    Robust multimodal learning via representation decoupling,

    S. Wei, Y . Luo, Y . Wang, and C. Luo, “Robust multimodal learning via representation decoupling,” inProc. Eur. Conf. Comput. Vis.Springer, 2024, pp. 38–54

    2024

  20. [25]

    Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis,

    W. Yu, H. Xu, Z. Yuan, and J. Wu, “Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis,” inProc. AAAI Conf. Artif. Intell., vol. 35, no. 12, 2021, pp. 10 790–10 797

    2021

  21. [26]

    Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis,

    W. Han, H. Chen, and S. Poria, “Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis,” inProc. 2021 Conf. Empir. Methods Natural Lang. Process., Nov. 2021, pp. 9180–9192. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, A...

    2021

  22. [27]

    Tetfn: A text enhanced transformer fusion network for multimodal sentiment analysis,

    D. Wang, X. Guo, Y . Tian, J. Liu, L. He, and X. Luo, “Tetfn: A text enhanced transformer fusion network for multimodal sentiment analysis,”Pattern Recognit., vol. 136, 2023

    2023

  23. [28]

    Dash- fusion: Dual-stream alignment with hierarchical bottleneck fusion for multimodal sentiment analysis,

    Y . Wen, Q. Li, Y . Zhou, Y . Gao, Z. Wen, J. Tao, and Y . Li, “Dash- fusion: Dual-stream alignment with hierarchical bottleneck fusion for multimodal sentiment analysis,”IEEE Trans. Neural Netw. Learn. Syst., vol. 36, no. 10, pp. 17 941–17 952, 2025

    2025

  24. [29]

    Improving cross-modal semantic alignment with cross-modal joint semantic trans- former for multimodal sentiment analysis,

    W. Zhang, J. Ding, H. Liu, T. Han, Y . Liu, and L. T. Yang, “Improving cross-modal semantic alignment with cross-modal joint semantic trans- former for multimodal sentiment analysis,”Big Data Min. Anal., 2026

    2026

  25. [30]

    Generative adversarial networks,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial networks,” Commun. ACM, vol. 63, no. 11, pp. 139–144, 2020

    2020

  26. [31]

    Incomplete multimodality-diffused emotion recognition,

    Y . Wang, Y . Li, and Z. Cui, “Incomplete multimodality-diffused emotion recognition,”Proc. Int. Conf. Neural Inf. Process. Syst., vol. 36, pp. 17 117–17 128, 2023

    2023

  27. [32]

    Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis,

    H. Zhang, Y . Wang, G. Yin, K. Liu, Y . Liu, and T. Yu, “Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis,” inProc. 2023 Conf. Empir. Methods Natural Lang. Process., 2023, pp. 756–767

    2023

  28. [33]

    Gcnet: Graph completion network for incomplete multimodal learning in conversation,

    Z. Lian, L. Chen, L. Sun, B. Liu, and J. Tao, “Gcnet: Graph completion network for incomplete multimodal learning in conversation,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 7, pp. 8419–8432, 2023

    2023

  29. [34]

    Deep learning and the information bottleneck principle,

    N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” inProc. 2015 IEEE Inf. Theory Wkshp. (ITW). IEEE, 2015, pp. 1–5

    2015

  30. [35]

    Deep variational information bottleneck,

    A. A. Alemi, I. Fischer, J. V . Dillon, and K. Murphy, “Deep variational information bottleneck,”arXiv preprint arXiv:1612.00410, 2016

    2016 arXiv

  31. [36]

    Learning deep representations by mutual information estimation and maximization,

    R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y . Bengio, “Learning deep representations by mutual information estimation and maximization,” inProc. Int. Conf. Learn. Representations, 2018

    2018

  32. [37]

    Learning representations by maximizing mutual information across views,

    P. Bachman, R. D. Hjelm, and W. Buchwalter, “Learning representations by maximizing mutual information across views,”Proc. Int. Conf. Neural Inf. Process. Syst., vol. 32, 2019

    2019

  33. [38]

    On mutual information maximization for representation learning,

    M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic, “On mutual information maximization for representation learning,” arXiv preprint arXiv:1907.13625, 2019

    1907 arXiv

  34. [39]

    Mutual information neural estimation,

    M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y . Bengio, A. Courville, and D. Hjelm, “Mutual information neural estimation,” inProc. Int. Conf. Mach. Learn.PMLR, 2018, pp. 531–540

    2018

  35. [40]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  36. [41]

    Club: A contrastive log-ratio upper bound of mutual information,

    P. Cheng, W. Hao, S. Dai, J. Liu, Z. Gan, and L. Carin, “Club: A contrastive log-ratio upper bound of mutual information,” inProc. Int. Conf. Mach. Learn.PMLR, 2020, pp. 1779–1788

    2020

  37. [42]

    Multimodal representations learning based on mutual information maximization and minimization and identity embedding for multimodal sentiment analysis,

    J. Zheng, S. Zhang, X. Wang, and Z. Zeng, “Multimodal representations learning based on mutual information maximization and minimization and identity embedding for multimodal sentiment analysis,”arXiv preprint arXiv:2201.03969, 2022

    2022 arXiv

  38. [43]

    A survey of uncertainty in deep neural networks,

    J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscheret al., “A survey of uncertainty in deep neural networks,”Artif. Intell. Rev., vol. 56, no. Suppl 1, pp. 1513–1589, 2023

    2023

  39. [44]

    What uncertainties do we need in bayesian deep learning for computer vision?

    A. Kendall and Y . Gal, “What uncertainties do we need in bayesian deep learning for computer vision?”Proc. Int. Conf. Neural Inf. Process. Syst., vol. 30, 2017

    2017

  40. [45]

    Transforming neural-net output levels to probability distributions,

    J. Denker and Y . LeCun, “Transforming neural-net output levels to probability distributions,”Proc. Int. Conf. Neural Inf. Process. Syst., vol. 3, 1990

    1990

  41. [46]

    R. M. Neal,Bayesian learning for neural networks. Springer Science & Business Media, 2012, vol. 118

    2012

  42. [47]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning,

    Y . Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” inProc. Int. Conf. Mach. Learn.PMLR, 2016, pp. 1050–1059

    2016

  43. [48]

    Simple and scalable predictive uncertainty estimation using deep ensembles,

    B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,”Proc. Int. Conf. Neural Inf. Process. Syst., vol. 30, 2017

    2017

  44. [49]

    Energy-based out-of-distribution detection,

    W. Liu, X. Wang, J. Owens, and Y . Li, “Energy-based out-of-distribution detection,”Proc. Int. Conf. Neural Inf. Process. Syst., vol. 33, pp. 21 464–21 475, 2020

    2020

  45. [50]

    Uncertainty-aware contrastive distillation for incremental semantic segmentation,

    G. Yang, E. Fini, D. Xu, P. Rota, M. Ding, M. Nabi, X. Alameda-Pineda, and E. Ricci, “Uncertainty-aware contrastive distillation for incremental semantic segmentation,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 2, pp. 2567–2581, 2023

    2023

  46. [51]

    Uncertainty- aware action decoupling transformer for action anticipation,

    H. Guo, N. Agarwal, S.-Y . Lo, K. Lee, and Q. Ji, “Uncertainty- aware action decoupling transformer for action anticipation,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2024, pp. 18 644– 18 654

    2024

  47. [52]

    Relative uncertainty learning for facial expression recognition,

    Y . Zhang, C. Wang, and W. Deng, “Relative uncertainty learning for facial expression recognition,”Proc. Int. Conf. Neural Inf. Process. Syst., vol. 34, pp. 17 616–17 627, 2021

    2021

  48. [53]

    Trusted multi-view classifica- tion with dynamic evidential fusion,

    Z. Han, C. Zhang, H. Fu, and J. T. Zhou, “Trusted multi-view classifica- tion with dynamic evidential fusion,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 2, pp. 2551–2566, 2023

    2023

  49. [54]

    Uncertainty- aware graph neural networks: A multihop evidence fusion approach,

    Q. Chen, S. Li, Y . Liu, S. Pan, G. I. Webb, and S. Zhang, “Uncertainty- aware graph neural networks: A multihop evidence fusion approach,” IEEE Trans. Neural Netw. Learn. Syst., 2025

    2025

  50. [55]

    Uncertainty-aware hierarchical aggregation network for medical image segmentation,

    T. Zhou, Y . Zhou, G. Li, G. Chen, and J. Shen, “Uncertainty-aware hierarchical aggregation network for medical image segmentation,”IEEE Trans. Circuits Syst. Video Technol., vol. 34, no. 8, pp. 7440–7453, 2024

    2024

  51. [56]

    Ua-fusion: Uncertainty- aware multimodal data fusion framework for 3d object detection of autonomous vehicles,

    Z. Shao, H. Wang, Y . Cai, L. Chen, and Y . Li, “Ua-fusion: Uncertainty- aware multimodal data fusion framework for 3d object detection of autonomous vehicles,”IEEE Trans. Instrum. Meas., 2025

    2025

  52. [57]

    Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference,

    M. Subedar, R. Krishnan, P. L. Meyer, O. Tickoo, and J. Huang, “Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 6301–6310

    2019

  53. [59]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2018

    2018

  54. [60]

    Probabilistic embeddings for cross-modal retrieval,

    S. Chun, S. J. Oh, R. S. De Rezende, Y . Kalantidis, and D. Larlus, “Probabilistic embeddings for cross-modal retrieval,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 8415–8424

    2021

  55. [61]

    Auto-encoding variational bayes,

    D. P. Kingma and M. Welling, “Auto-encoding variational bayes,”arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  56. [64]

    Multimodal senti- ment intensity analysis in videos: Facial gestures and verbal messages,

    A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Multimodal senti- ment intensity analysis in videos: Facial gestures and verbal messages,” IEEE Intell. Syst., vol. 31, no. 6, pp. 82–88, 2016

    2016

  57. [65]

    Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,

    A. Bagher Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,” inProc. 56th Annu. Meeting Assoc. Comput. Linguistics, Jul. 2018, pp. 2236–2246

    2018

  58. [66]

    CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine- grained annotation of modality,

    W. Yu, H. Xu, F. Meng, Y . Zhu, Y . Ma, J. Wu, J. Zou, and K. Yang, “CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine- grained annotation of modality,” inProc. 58th Annu. Meeting Assoc. Comput. Linguistics, Jul. 2020, pp. 3718–3727

    2020

  59. [67]

    Cross-modal enhancement network for multimodal sentiment analysis,

    D. Wang, S. Liu, Q. Wang, Y . Tian, L. He, and X. Gao, “Cross-modal enhancement network for multimodal sentiment analysis,”IEEE Trans. Multimedia, vol. 25, pp. 4909–4921, 2023

    2023

  60. [68]

    Covarep - a collaborative voice analysis repository for speech technologies,

    G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer, “Covarep - a collaborative voice analysis repository for speech technologies,” in Proc. IEEE 40th Int. Conf. Acoust. Speech Signal Process., 2014, pp. 960–964

    2014

  61. [69]

    librosa: Audio and music signal analysis in python,

    B. McFee, C. Raffel, D. Liang, D. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python,” in Proc. 14th Scipy. Conf.SciPy, 2015, p. 18

    2015

  62. [70]

    Openface 2.0: Facial behavior analysis toolkit,

    T. Baltrusaitis, A. Zadeh, Y . C. Lim, and L.-P. Morency, “Openface 2.0: Facial behavior analysis toolkit,” inProc. IEEE 13th Int. Conf. Autom. Face Gesture Recognit., 2018, pp. 59–66

    2018

  63. [71]

    M-sena: An integrated platform for multimodal sentiment analysis,

    H. Mao, Z. Yuan, H. Xu, W. Yu, Y . Liu, and K. Gao, “M-sena: An integrated platform for multimodal sentiment analysis,” inProc. 60th Annu. Meeting Assoc. Comput. Linguistic, 2022, pp. 204–213

    2022

  64. [72]

    Adam: A method for stochastic optimization,

    D. P. Kingma, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.