Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts

Guowei Zhong; Mingzhen Wu; Peng Chen; Ronghua Liang; Ruohong Huan

REVIEW 4 major objections 5 minor 1 cited by

Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CIDer claims that one compact model, trained once, can stay accurate when arbitrary tokens disappear from language, audio, or visual streams and when test word or label distributions shift.

desk verdict A genuinely useful missing-modality framework whose central OOD claim currently rests on an unseen, possibly IID-preserving dataset split; worth reviewing carefully but needs a clear description of the repartitioning before the OOD numbers can be trusted. read the letter →

arxiv 2506.10452 v1 pith:7TOEWVQG submitted 2025-06-12 cs.CV cs.CLcs.LGcs.MM

Guowei Zhong , Ruohong Huan , Mingzhen Wu , Ronghua Liang , Peng Chen This is my paper · ORCID

classification cs.CVcs.CLcs.LGcs.MM

keywords multimodalemotionrecognitionmissingmodalitiesrandommodalityfeatureout-of-distributiongeneralizationself-distillationcausalinferencecounterfactualreasoningtransformer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that one compact model can handle two failure modes at once: arbitrary token loss in any of the language, audio, or visual streams, and test-time shifts in word and label distributions. It proposes CIDer, which combines a Model-Specific Self-Distillation (MSSD) module for incomplete inputs and a Model-Agnostic Causal Inference (MACI) module for out-of-distribution inputs. The paper also defines a generalized missing-modality task called RMFM and repartitions CMU-MOSI and CMU-MOSEI into new OOD splits. If the claims hold, practical emotion-recognition systems could be smaller, faster to train, and more reliable under real-world degradation.

What carries the argument

The load-bearing machinery is the two-module CIDer architecture. MSSD is a weight-sharing twin network whose complete-view stream acts as teacher for the incomplete-view stream, with SmoothL1 loss on reconstructed unimodal features, KL divergence on attention maps, and cosine similarity on joint representations. MACI is the causal module: during training, a Multimodal Causal Module (MCM) forms class-specific representations from joint features and per-class mean tokens, then averages over class priors to implement backdoor adjustment $P(Y|do(L))$; during testing, counterfactual texts are built by masking words whose inter-class frequency variation is below a threshold, and the counterfactual prediction is subtracted from the main prediction. Two supporting components keep the model compact: WSAM aligns long audio and visual sequences to word level, and the Multimodal Composite Transformer (MCT) computes intra- and inter-modal attention from shared matrices.

What would settle it

Compare the unigram word distribution of the new OOD training and test splits; if their divergence is no larger than the divergence between the original IID training and test splits, then the 'OOD' condition is not a real distribution shift and the OOD comparison collapses.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a weight-sharing twin network trained once can preserve emotion-recognition accuracy when arbitrary tokens are dropped from any modality at missing rates up to 100%, and that a small causal module can then remove label and language biases on shifted test data. MSSD performs hierarchical self-distillation from a complete-view stream to an incomplete-view stream, aligning reconstructed low-level features, attention maps, and joint representations. MACI uses a causal graph with backdoor adjustment during training to counter label bias, and subtracts predictions made from fine-grained counterfactual texts during testing to counter language bias. The paper further claims that MACI adds about 37K learnable parameters and improves OOD accuracy when plugged into existing MER models such as DLF and MPLMM.

Load-bearing premise

The OOD results stand on the assumption that the newly repartitioned test sets genuinely shift word and label distributions away from training, yet the paper never describes how the repartitioning was done.

Editorial extensions

If this is right

A single CIDer trained once with random missing rates can be tested across the full 0-100% range under RMFM, traditional RMFM, RMM, TMFM, STMFM, and SMM without retraining at each rate.
Plugging MACI into existing MER models such as DLF and MPLMM improves their OOD performance with about 37K additional parameters.
On the repartitioned CMU-MOSI and CMU-MOSEI OOD splits, CIDer reports the best results on most metrics for complete inputs and for the combined RMFM-plus-OOD setting, with roughly 248K parameters and 10.4 seconds per training epoch.
The hierarchical distillation aligns reconstructed features, attention maps, and joint representations so that the same weight-sharing architecture serves as both teacher and student.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If RMFM becomes a standard evaluation protocol, missing-modality studies could be compared on one missing-rate axis, making one-to-all training the default instead of per-rate retraining.
The counterfactual-text recipe (top-100 frequent words, coefficient-of-variation threshold 0.1) looks transferable to other sequence tasks where label-language correlation drifts, such as review sentiment or toxicity detection.
MACI's portability across two very different base models suggests OOD debiasing may be separable from fusion architecture; testing it on larger pretrained multimodal encoders would show whether the 37K-parameter module scales.
The MSSD design implies that any future modality-missing type can be handled by the same three-level distillation, so the authors' missing-scenario enumeration is an instantiation rather than a closed list.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth a look. The RMFM task is a modest but real generalization of the usual missing-modality definitions, and the MSSD module—weight-shared twin networks doing hierarchical distillation—is clean and practical. MACI is also genuinely model-agnostic and adds very few parameters, which is a nice engineering property. The experimental coverage is extensive, the code is public, and the cross-dataset results give some independent support that the method generalizes beyond IID. This is not a paper that reorganizes the field, but it is a solid, useful contribution to robust multimodal emotion recognition.

The soft spot is exactly where the stress-test note lands. The OOD datasets are supposedly created by repartitioning CMU-MOSI and CMU-MOSEI, but the paper never says how train/validation/test word or label distributions are made to differ. The split counts in Table I are essentially identical to the IID splits, so a random reshuffle would produce the same numbers and still be described as "repartitioned." If that is what happened, the OOD columns in Tables IV and XVII–XVIII are just IID results relabeled. That is a load-bearing problem because the headline OOD-robustness claim depends on those columns. The causal adjustment in Eq. (20) also looks underspecified: conditioning on the mediator M while adjusting for C is not standard backdoor adjustment, and the paper does not defend the step. These issues are addressable, but they have to be addressed.

Smaller concerns: tau and the counterfactual selection thresholds are not reported for the main results, and there are no error bars, so the leaderboard margins are a bit fragile. The cross-dataset experiments help, but they are not the same as the missing OOD construction.

Who gets value from this? Anyone working on MER under missing inputs who wants a compact, trainable baseline, and anyone building debiasing modules they want to bolt onto existing models. A serious referee should see it, but the review has to force the authors to document the OOD repartitioning procedure and show that the train/test distributions actually differ. Without that, the central claim is unverified.

Referee Report

4 major / 5 minor

Summary. The paper proposes CIDer, a multimodal emotion recognition (MER) framework that jointly handles random modality feature missing and out-of-distribution (OOD) inputs. The framework combines a Model-Specific Self-Distillation (MSSD) module for robustness to the newly introduced Random Modality Feature Missing (RMFM) task and a Model-Agnostic Causal Inference (MACI) module for OOD generalization. The authors also introduce a generalized RMFM definition, a word-level alignment module (WSAM), a multimodal fusion transformer (MCT), and repartitioned OOD versions of CMU-MOSI and CMU-MOSEI. Experiments compare CIDer with several baselines across IID, OOD, RMFM, RMM, TMFM, STMFM, and cross-dataset settings, reporting state-of-the-art or competitive results with fewer parameters and short training time.

Significance. If the OOD evaluation is valid, the paper makes a practically relevant contribution: a compact MER model that is robust to arbitrary token-level feature loss and to word/label distribution shifts, with a model-agnostic debiasing module (MACI) that can be plugged into existing methods. The public code, the extensive per-missing-rate tables, and the generalization of the missing-modality definition to RMFM are strengths. However, the central OOD claim currently rests on an undocumented repartitioning procedure and on causal equations whose validity is questionable; until those are resolved, the significance of the OOD results cannot be assessed from the manuscript alone.

major comments (4)

[Section III-C, Eq. (20)] The construction of the 'repartitioned MER OOD datasets' is not described. The text only states that the authors 'repartitioned the original CMU-MOSI and CMU-MOSEI datasets,' with no algorithm, criterion, or distributional check to ensure that train and test differ in word distribution and, consequently, label distribution. Table I shows that the new OOD splits have almost identical cardinalities to the IID splits (e.g., MOSI 1284/229/686 for both IID and OOD), so a random reshuffle would preserve the IID word-label correlation and would not produce an OOD test set. This is load-bearing because the OOD columns in Tables IV, XVII, and XVIII are the only evidence for the paper's OOD robustness claim. The authors should specify the repartitioning rule, report distributional statistics (e.g., word-frequency shift, label-prior shift, KL divergence) for both datasets, and release the splits; without this, the OOD experiments cannot be interpreted as distribution-shift tests.
[Section III-C, Eq. (20)] The backdoor adjustment formula P(Y|do(L)) = sum_c P(Y|M=f(L,A,V), c)P(c) is not a standard backdoor adjustment for the graph in Fig. 5. In that graph, M is a descendant/mediator of L (since M=f(L,A,V)), so conditioning on M does not give the total causal effect of L on Y; the standard adjustment would use P(Y|L,c) after adjusting for the confounder C. The derivation of Eq. (20) should be supplied, or the claim should be reframed as estimating a different target quantity (e.g., a controlled direct effect). Without a valid derivation, the label-bias-mitigation mechanism of MCM is not established.
[Section III-C, Eq. (25)] The debiasing strength tau in Eq. (25) is a free parameter, but the manuscript does not specify how it is selected. If tau is tuned on the OOD test set, the MACI gains in Table IV partly reflect test-set fitting; if it is fixed, its value and sensitivity should be reported. Similarly, the counterfactual word-selection thresholds (CV_w >= 0.1 and top-100 frequency) are presented without justification or ablation. Since these choices directly control the magnitude of the counterfactual subtraction, their treatment affects the validity of the OOD claim.
[Section III-C, Eqs. (24) and footnote 4] The test-time assumption of uniform class priors P(i)=1/cls conflicts with the paper's own definition of OOD in footnote 1, which states that OOD includes 'the resulting deviations in label distribution' caused by word-distribution shifts. If the test label prior actually deviates from uniform under the new OOD splits, then using uniform P(i) in Eq. (24) is an unsupported assumption. The authors should either justify the uniform prior as an interventional distribution, report the actual test label distributions, or include a sensitivity analysis over P(i).

minor comments (5)

[Section IV-A] There is a typo in the first sentence of the OOD scenario paragraph: 'OO) scenarios' should be 'OOD scenarios'.
[Section IV-A] The paper claims the new OOD datasets provide 'word-level aligned and unaligned versions,' but the main OOD experiments in Table IV only use unaligned sequences; no aligned OOD results are reported. This discrepancy should be clarified.
[Appendix] The text refers to 'Tabel XI' in the appendix, which should be 'Table XI'.
[Section III-C] The notation for counterfactual text is inconsistent: the text writes S_cf = <w_1, [MASK], [MASK], ..., w_Tl>, but the word positions that are not masked are not explicitly defined. Please clarify which words remain after applying the CV and frequency thresholds.
[Section VI] The conclusion states that MACI 'can be independently applied to enhance the OOD generalization capabilities of existing MER models,' but the only evidence for this is the DLF+MACI and MPLMM+MACI rows in Table IV; no statistical significance or variance information is provided, so the robustness of this claim is not established.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CIDer's predictions are not equivalent to its inputs; the OOD dataset construction is under-specified but not shown to be circular.

full rationale

The paper's central derivations are explicit: MSSD defines reconstruction, attention, and joint-representation distillation losses (Eqs. 17-19) comparing complete and incomplete views, and MACI defines backdoor-adjusted training (Eq. 20) and counterfactual subtraction (Eq. 25). Neither reduces to a fitted parameter renamed as a prediction. The MCM class-prior and counterfactual-word selection use training-set statistics, but this is standard training; the test sets are held out. The new OOD datasets are described in Section IV-A only as 'we repartitioned the original CMU-MOSI and CMU-MOSEI datasets' with no repartitioning algorithm, so the OOD results are not independently verifiable and may not actually involve distribution shift; that is an experimental-validity and reproducibility concern, not circularity. Self-citations [4], [10], and [15] appear only in related work and do not carry the derivation. Overall, no prediction in the paper is equivalent by definition or self-citation to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central performance claim depends on tuned hyperparameters (distillation weights, counterfactual selection thresholds, and tau) and on a causal graph that is asserted rather than derived. The RMFM task and OOD datasets are introduced, but the OOD split construction is undocumented.

free parameters (3)

alpha, beta, gamma (distillation loss weights) = Not reported (tuned via Optuna over [0.0, 1.0])
These weights in Eq. (30) balance the reconstruction, attention, and joint-distillation losses. They are optimized on the validation split, making them data-fitted.
Counterfactual word selection thresholds (CV_w >= 0.1, top-100 frequency) = 0.1 and 100
These thresholds in Eq. (27) decide which words are retained in the counterfactual text. They are hand-chosen and not swept or justified.
tau (debiasing strength) = Not reported
tau in Eq. (25) controls how much counterfactual prediction is subtracted. It is a free hyperparameter whose value is not listed in Table II.

assumptions (4)

domain assumption The causal graph in Fig. 5 correctly represents the data-generating process: label bias C affects language L and label Y, and language L affects Y through the multimodal representation M.
The entire MACI debiasing procedure rests on this graph, but the paper provides no causal discovery or sensitivity analysis.
ad hoc to paper The backdoor adjustment formula P(Y|do(L)) = sum_c P(Y|M=f(L,A,V),c)P(c) is valid even though M is a mediator/descendant of L.
Standard backdoor adjustment conditions on confounders of the treatment, not on a mediator that also depends on L. The paper does not reconcile this.
ad hoc to paper At test time, uniform class priors P(i)=1/cls are assumed.
Stated in footnote 4: 'Since the category labels are not available, we assume an equal number of instances across all categories.' This changes the MCM output and is a modeling assumption.
domain assumption Pre-extracted BERT, COVAREP, and Facet features are sufficient representations of language, audio, and vision.
The method consumes these fixed features; errors or incompleteness in these upstream features would propagate into the model.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts." pith.science (2026). https://pith.science/paper/7TOEWVQG

@misc{pith2026250610452,
  author       = {Pith},
  title        = {Pith review of: Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7TOEWVQG}},
  note         = {Machine review of arXiv:2506.10452}
}

read the original abstract

Recent advancements in Multimodal Emotion Recognition (MER) face challenges in addressing both modality missing and Out-Of-Distribution (OOD) data simultaneously. Existing methods often rely on specific models or introduce excessive parameters, which limits their practicality. To address these issues, we propose a novel robust MER framework, Causal Inference Distiller (CIDer), and introduce a new task, Random Modality Feature Missing (RMFM), to generalize the definition of modality missing. CIDer integrates two key components: a Model-Specific Self-Distillation (MSSD) module and a Model-Agnostic Causal Inference (MACI) module. MSSD enhances robustness under the RMFM task through a weight-sharing self-distillation approach applied across low-level features, attention maps, and high-level representations. Additionally, a Word-level Self-aligned Attention Module (WSAM) reduces computational complexity, while a Multimodal Composite Transformer (MCT) facilitates efficient multimodal fusion. To tackle OOD challenges, MACI employs a tailored causal graph to mitigate label and language biases using a Multimodal Causal Module (MCM) and fine-grained counterfactual texts. Notably, MACI can independently enhance OOD generalization with minimal additional parameters. Furthermore, we also introduce the new repartitioned MER OOD datasets. Experimental results demonstrate that CIDer achieves robust performance in both RMFM and OOD scenarios, with fewer parameters and faster training compared to state-of-the-art methods. The implementation of this work is publicly accessible at https://github.com/gw-zhong/CIDer.

Figures

Figures reproduced from arXiv: 2506.10452 by the authors.

**Figure 2.** The overall framework of CIDer. Specifically, CIDer consists of two primary modules: the MSSD module and the MACI module. These modules are [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The Word-level Self-aligned Attention Module (WSAM), taking the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: The causal graph for MER. During training, label bias is mitigated [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: The Multimodal Causal Module (MCM). we compute the mean features Li 3 , Ai , and Vi , and encode them using modality-shared MLPs: H {l,a,v} i = MLP{l,a,v} ({L, A, V }i ) ∈ R d (21) where MLP{l,a,v}(·) consists of three linear layers interspersed with two ReLU activatio…

**Figure 7.** Figure 7: The phenomenon of mixed sentences in the MER OOD datasets [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: The distribution of joint representations in the embedding space. Red dots correspond to negative samples, green dots represent positive samples, and [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 9.** Figure 9: An example of MACI correcting prediction. The category highlighted [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

EmoScene: A Dual-space Dataset for Controllable Affective Image Generation
cs.CV 2026-04 reject novelty 6.0 of 10

EmoScene contributes 1.2M images annotated with discrete emotions, continuous VAD scores, perceptual attributes, and captions, plus a cross-attention modulation that shifts generated images toward requested affective targets.

Reference graph

Works this paper leans on

41 extracted references · 38 canonical work pages · cited by 1 Pith paper

[1]

Tensor fusion network for multimodal sentiment analysis,

A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor fusion network for multimodal sentiment analysis,” inProc. Conf. Empirical Methods Natural Lang. Process., 2017, pp. 1103–1114

work page 2017
[2]

Efficient low-rank multimodal fusion with modality- specific factors,

Z. Liu, Y . Shen, V . B. Lakshminarasimhan, P. P. Liang, A. B. Zadeh, and L.-P. Morency, “Efficient low-rank multimodal fusion with modality- specific factors,” inProc. Assoc. Comput. Linguistics, 2018, pp. 2247– 2256

work page 2018
[3]

Words can shift: Dynamically adjusting word representations using nonverbal behaviors,

Y . Wang, Y . Shen, Z. Liu, P. P. Liang, A. Zadeh, and L.-P. Morency, “Words can shift: Dynamically adjusting word representations using nonverbal behaviors,” inProc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, pp. 7216–7223

work page 2019
[4]

Trisat: Trimodal represen- tation learning for multimodal sentiment analysis,

R. Huan, G. Zhong, P. Chen, and R. Liang, “Trisat: Trimodal represen- tation learning for multimodal sentiment analysis,”IEEE/ACM Trans. Audio Speech Lang. Process., 2024

work page 2024
[5]

Distinguishing visually similar images: Triplet contrastive learning framework for image-text retrieval,

P. Ouyang, J. Chen, Q. Ma, Z. Wang, and C. Bai, “Distinguishing visually similar images: Triplet contrastive learning framework for image-text retrieval,” inProc. IEEE Int. Conf. Multimedia Expo. IEEE, 2024, pp. 1–6

work page 2024
[6]

Counterfactual reasoning for out-of-distribution multimodal sentiment analysis,

T. Sun, W. Wang, L. Jing, Y . Cui, X. Song, and L. Nie, “Counterfactual reasoning for out-of-distribution multimodal sentiment analysis,” in Proc. ACM Int. Conf. Multimedia, 2022, pp. 15–23

work page 2022
[7]

Found in translation: Learning robust joint representations by cyclic translations between modalities,

H. Pham, P. P. Liang, T. Manzini, L.-P. Morency, and B. P ´oczos, “Found in translation: Learning robust joint representations by cyclic translations between modalities,” inProc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, pp. 6892–6899. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13 TABLE XI COMPARISON WITH STATE-OF-THE-ART METHODS. M...

work page 2019
[8]

Transmodality: An end2end fusion method with transformer for multimodal sentiment analysis,

Z. Wang, Z. Wan, and X. Wan, “Transmodality: An end2end fusion method with transformer for multimodal sentiment analysis,” inProc. Web Conf., 2020, pp. 2514–2520

work page 2020

Show all 41 references

[9]

Ctfn: Hierarchical learning for multimodal sentiment analysis using coupled- translation fusion network,

J. Tang, K. Li, X. Jin, A. Cichocki, Q. Zhao, and W. Kong, “Ctfn: Hierarchical learning for multimodal sentiment analysis using coupled- translation fusion network,” inProc. Assoc. Comput. Linguistics, 2021, pp. 5301–5311

2021
[10]

Unimf: A unified multi- modal framework for multimodal sentiment analysis in missing modal- ities and unaligned multimodal sequences,

R. Huan, G. Zhong, P. Chen, and R. Liang, “Unimf: A unified multi- modal framework for multimodal sentiment analysis in missing modal- ities and unaligned multimodal sequences,”IEEE Trans. Multimedia, vol. 26, pp. 5753–5768, 2023

2023
[11]

Transformer-based feature recon- struction network for robust multimodal sentiment analysis,

Z. Yuan, W. Li, H. Xu, and W. Yu, “Transformer-based feature recon- struction network for robust multimodal sentiment analysis,” inProc. ACM Int. Conf. Multimedia, 2021, pp. 4400–4407

2021
[12]

Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis,

L. Sun, Z. Lian, B. Liu, and J. Tao, “Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis,”IEEE Trans. Affective Comput., vol. 15, no. 1, pp. 309–325, 2023

2023
[13]

Gcnet: Graph completion network for incomplete multimodal learning in conversation,

Z. Lian, L. Chen, L. Sun, B. Liu, and J. Tao, “Gcnet: Graph completion network for incomplete multimodal learning in conversation,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 7, pp. 8419–8432, 2023

2023
[14]

Towards multimodal sentiment analysis debiasing via bias purification,

D. Yang, M. Li, D. Xiao, Y . Liu, K. Yang, Z. Chen, Y . Wang, P. Zhai, K. Li, and L. Zhang, “Towards multimodal sentiment analysis debiasing via bias purification,” inProc. Eur. Conf. Comput. Vis.Springer, 2024, pp. 464–481

2024
[15]

Muldef: A model-agnostic debiasing framework for robust multimodal sentiment analysis,

R. Huan, G. Zhong, P. Chen, and R. Liang, “Muldef: A model-agnostic debiasing framework for robust multimodal sentiment analysis,”IEEE Trans. Multimedia, 2024

2024
[16]

General debiasing for multimodal sentiment analysis,

T. Sun, J. Ni, W. Wang, L. Jing, Y . Wei, and L. Nie, “General debiasing for multimodal sentiment analysis,” inProc. ACM Int. Conf. Multimedia, 2023, pp. 5861–5869

2023
[17]

Bcd-mm: Mul- timodal sentiment analysis model with dual-bias-aware feature learning and attention mechanisms,

L. Ma, J. Li, D. Shao, J. Yan, J. Wang, and Y . Yan, “Bcd-mm: Mul- timodal sentiment analysis model with dual-bias-aware feature learning and attention mechanisms,”IEEE Access, 2024

2024
[18]

Noise imitation based adver- sarial training for robust multimodal sentiment analysis,

Z. Yuan, Y . Liu, H. Xu, and K. Gao, “Noise imitation based adver- sarial training for robust multimodal sentiment analysis,”IEEE Trans. Multimedia, vol. 26, pp. 529–539, 2023

2023
[19]

Meta noise adaption framework for multimodal sentiment analysis with feature noise,

Z. Yuan, B. Zhang, H. Xu, and K. Gao, “Meta noise adaption framework for multimodal sentiment analysis with feature noise,”IEEE Trans. Multimedia, vol. 26, pp. 7265–7277, 2024

2024
[20]

Towards robust multimodal sentiment analysis with incomplete data,

H. Zhang, W. Wang, and T. Yu, “Towards robust multimodal sentiment analysis with incomplete data,” inProc. Adv. Neural Inf. Process. Syst., 2024

2024
[21]

Tag-assisted multimodal sentiment analysis under uncertain missing modalities,

J. Zeng, T. Liu, and J. Zhou, “Tag-assisted multimodal sentiment analysis under uncertain missing modalities,” inProc. Int. ACM SIGIR Conf. Res. Develop Inf. Ret., 2022, pp. 1545–1554

2022
[22]

Robust multimodal sentiment analysis via tag encoding of uncertain missing modalities,

J. Zeng, J. Zhou, and T. Liu, “Robust multimodal sentiment analysis via tag encoding of uncertain missing modalities,”IEEE Trans. Multimedia, vol. 25, pp. 6301–6314, 2022

2022
[23]

Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities,

J. Zeng, J. Zhou, and T. Liu, “Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities,” inProc. Conf. Empirical Methods Natural Lang. Process., 2022, pp. 2924–2934

2022
[24]

Distribution-consistent modal recovering for incomplete multimodal learning,

Y . Wang, Z. Cui, and Y . Li, “Distribution-consistent modal recovering for incomplete multimodal learning,” inProc. IEEE Int. Conf. Comput. Vis., 2023, pp. 22 025–22 034

2023
[25]

Missmodal: Increasing robustness to missing modality in multimodal sentiment analysis,

R. Lin and H. Hu, “Missmodal: Increasing robustness to missing modality in multimodal sentiment analysis,”Trans. Assoc. Comput. Linguistics, vol. 11, pp. 1686–1702, 2023

2023
[26]

Modality translation- based multimodal sentiment analysis under uncertain missing modali- ties,

Z. Liu, B. Zhou, D. Chu, Y . Sun, and L. Meng, “Modality translation- based multimodal sentiment analysis under uncertain missing modali- ties,”Inf. Fusion, vol. 101, p. 101973, 2024

2024
[27]

Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition,

Z. Guo, T. Jin, and Z. Zhao, “Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition,” inProc. Assoc. Comput. Linguistics, 2024, pp. 1726–1736. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 14 TABLE XIII COMPARISON WITH ST...

2024
[28]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inProc. Conf. North Amer. Chapter Assoc. Comput. Linguistics, 2019, pp. 4171– 4186

2019
[29]

Empirical evaluation of gated recurrent neural networks on sequence modeling,

J. Chung, C. Gulcehre, K. Cho, and Y . Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,”arXiv preprint arXiv:1412.3555, 2014

2014 arXiv
[30]

Layer normalization,

J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,”arXiv preprint arXiv:1607.06450, 2016

2016 arXiv
[31]

Multimodal senti- ment intensity analysis in videos: Facial gestures and verbal messages,

A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Multimodal senti- ment intensity analysis in videos: Facial gestures and verbal messages,” IEEE Intell. Syst., vol. 31, no. 6, pp. 82–88, 2016

2016
[32]

Multimodal language analysis in the wild: Cmu-mosei dataset and in- terpretable dynamic fusion graph,

A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and in- terpretable dynamic fusion graph,” inProc. Assoc. Comput. Linguistics, 2018, pp. 2236–2246

2018
[33]

Co- varep—a collaborative voice analysis repository for speech technolo- gies,

G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer, “Co- varep—a collaborative voice analysis repository for speech technolo- gies,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process.IEEE, 2014, pp. 960–964

2014
[34]

Adam: A method for stochastic optimization,

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014

2014 arXiv
[35]

Optuna: A next-generation hyperparameter optimization framework,

T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” inProc. ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2019, pp. 2623– 2631

2019
[36]

Decoupled multimodal distilling for emo- tion recognition,

Y . Li, Y . Wang, and Z. Cui, “Decoupled multimodal distilling for emo- tion recognition,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit., 2023, pp. 6631–6640

2023
[37]

Glomo: Global-local modal fusion for multimodal sentiment analysis,

Y . Zhuang, Y . Zhang, Z. Hu, X. Zhang, J. Deng, and F. Ren, “Glomo: Global-local modal fusion for multimodal sentiment analysis,” inProc. ACM Int. Conf. Multimedia, 2024, pp. 1800–1809

2024
[38]

Dlf: Disentangled- language-focused multimodal sentiment analysis,

P. Wang, Q. Zhou, Y . Wu, T. Chen, and J. Hu, “Dlf: Disentangled- language-focused multimodal sentiment analysis,” inProc. AAAI Conf. Artif. Intell., 2025

2025
[39]

Integrating multimodal information in large pretrained transformers,

W. Rahman, M. K. Hasan, S. Lee, A. Zadeh, C. Mao, L.-P. Morency, and E. Hoque, “Integrating multimodal information in large pretrained transformers,” inProc. Assoc. Comput. Linguistics, vol. 2020, 2020, p. 2359

2020
[40]

Connection- ist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Connection- ist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” inProc. Int. Conf. Mach. Learn., 2006, pp. 369–376

2006
[41]

Bridging the gap for test- time multimodal sentiment analysis,

Z. Guo, T. Jin, W. Xu, W. Lin, and Y . Wu, “Bridging the gap for test- time multimodal sentiment analysis,” inProc. AAAI Conf. Artif. Intell., 2025. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 15 TABLE XIV COMPARISON WITH STATE-OF-THE-ART METHODS. Models CMU-MOSE...

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Tensor fusion network for multimodal sentiment analysis,

A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor fusion network for multimodal sentiment analysis,” inProc. Conf. Empirical Methods Natural Lang. Process., 2017, pp. 1103–1114

work page 2017

[2] [2]

Efficient low-rank multimodal fusion with modality- specific factors,

Z. Liu, Y . Shen, V . B. Lakshminarasimhan, P. P. Liang, A. B. Zadeh, and L.-P. Morency, “Efficient low-rank multimodal fusion with modality- specific factors,” inProc. Assoc. Comput. Linguistics, 2018, pp. 2247– 2256

work page 2018

[3] [3]

Words can shift: Dynamically adjusting word representations using nonverbal behaviors,

Y . Wang, Y . Shen, Z. Liu, P. P. Liang, A. Zadeh, and L.-P. Morency, “Words can shift: Dynamically adjusting word representations using nonverbal behaviors,” inProc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, pp. 7216–7223

work page 2019

[4] [4]

Trisat: Trimodal represen- tation learning for multimodal sentiment analysis,

R. Huan, G. Zhong, P. Chen, and R. Liang, “Trisat: Trimodal represen- tation learning for multimodal sentiment analysis,”IEEE/ACM Trans. Audio Speech Lang. Process., 2024

work page 2024

[5] [5]

Distinguishing visually similar images: Triplet contrastive learning framework for image-text retrieval,

P. Ouyang, J. Chen, Q. Ma, Z. Wang, and C. Bai, “Distinguishing visually similar images: Triplet contrastive learning framework for image-text retrieval,” inProc. IEEE Int. Conf. Multimedia Expo. IEEE, 2024, pp. 1–6

work page 2024

[6] [6]

Counterfactual reasoning for out-of-distribution multimodal sentiment analysis,

T. Sun, W. Wang, L. Jing, Y . Cui, X. Song, and L. Nie, “Counterfactual reasoning for out-of-distribution multimodal sentiment analysis,” in Proc. ACM Int. Conf. Multimedia, 2022, pp. 15–23

work page 2022

[7] [7]

Found in translation: Learning robust joint representations by cyclic translations between modalities,

H. Pham, P. P. Liang, T. Manzini, L.-P. Morency, and B. P ´oczos, “Found in translation: Learning robust joint representations by cyclic translations between modalities,” inProc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, pp. 6892–6899. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13 TABLE XI COMPARISON WITH STATE-OF-THE-ART METHODS. M...

work page 2019

[8] [8]

Transmodality: An end2end fusion method with transformer for multimodal sentiment analysis,

Z. Wang, Z. Wan, and X. Wan, “Transmodality: An end2end fusion method with transformer for multimodal sentiment analysis,” inProc. Web Conf., 2020, pp. 2514–2520

work page 2020