Pith. sign in

REVIEW 1 major objections 5 minor 44 references

AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment

T0 review · 1 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AlignMamba claims that adding optimal-transport-based token matching and a distribution-alignment loss before a Mamba backbone lets a linear-complexity model match or beat Transformer-based multimodal fusion, reporting higher accuracy on…

desk verdict Credible Mamba-fusion architecture with believable efficiency numbers and honest ablations, but the incomplete-fusion protocol is never described, leaving the headline robustness claim unverifiable as written. read the letter →

arxiv 2412.00833 v1 pith:QJCVSLHV submitted 2024-12-01 cs.CV cs.AI

Yan Li , Yifei Xing , Xiangyuan Lan , Xin Li , Haifeng Chen , Dongmei Jiang This is my paper · ORCID
classification cs.CVcs.AI
keywords multimodalfusionMambastatespacemodelsoptimaltransportmaximummeandiscrepancycross-modalalignmentincompletesentimentanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AlignMamba sets out to fix a specific weakness of Mamba-based multimodal fusion: Mamba's sequential scanning processes tokens in order, so it misses cross-modal correspondences between unscanned tokens. The paper proposes two complementary alignment mechanisms applied before the Mamba backbone: a local, optimal-transport-based module that matches tokens across modalities, and a global Maximum Mean Discrepancy loss that aligns whole modality distributions. On CMU-MOSI and CMU-MOSEI sentiment benchmarks, the resulting model reports higher accuracy than previous multimodal fusion methods while using less GPU memory and much less inference time. A reader should care because the method points to a way of getting Transformer-level cross-modal modeling at linear-complexity cost.

What carries the argument

The load-bearing components are the relaxed optimal transport matrix (a nearest-neighbor assignment under cosine distance, following the relaxed OT of Kusner et al. [12]) that projects audio and video tokens onto the language token grid, and the squared Maximum Mean Discrepancy with a Gaussian kernel, used as an auxiliary loss to align the projected distributions with language. The third component is the time-priority interleaving of the aligned unimodal tokens into a single sequence for the Mamba selective scan, which lets the sequential model see cross-modal neighbors at each timestep.

What would settle it

Compare AlignMamba against its own two variants (local-only and global-only) on the incomplete-fusion benchmark while varying how missing inputs are constructed — masking with zeros, removing tokens, and learned recovery — and check whether the reported 73.8% accuracy at 70% missing on CMU-MOSI holds under the same input-construction protocol used by GCNet and IMDer.

Watch

Extended reading notes

Core claim

The central discovery is that explicit cross-modal alignment before a Mamba backbone lets a state space model match or exceed Transformer-based multimodal fusion performance without quadratic attention cost. AlignMamba treats the audio and video feature sequences as discrete distributions and learns a relaxed optimal transport plan that maps each audio and video token to its nearest language token under cosine distance, producing aligned sequences at the language length; an MMD loss then pulls the aligned distributions together in a reproducing kernel Hilbert space. The aligned sequences are interleaved by timestep and fed to Mamba, whose selective scan then processes neighbors from different modalities in sequence. The paper reports that on CMU-MOSI this recipe raises binary classification accuracy to 86.9% (a 0.9% absolute gain over the best prior result it compares against) while reducing GPU memory by 20.3% and inference time by 83.3% relative to Transformer baselines at 6.4k tokens, and that on incomplete fusion with 10-70% modality dropout it retains higher average accuracy and a smaller drop than recovery-based baselines.

Load-bearing premise

The incomplete-fusion experiments assume a specific way of presenting partially missing inputs to AlignMamba, but the paper never states whether missing modalities are zero-padded, dropped, or filled by a recovery module; if the comparison baselines use explicit recovery and AlignMamba simply ignores or zeros the missing stream, the reported robustness could be an artifact of that unevaluated choice.

Editorial extensions

If this is right

  • On CMU-MOSI and CMU-MOSEI, AlignMamba reports the highest binary accuracy and F1 among the compared methods for both complete and incomplete fusion, with average incomplete accuracy of 79.9% and 81.6% respectively.
  • The dual alignment (OT local + MMD global) is the source of the gain: removing either component lowers accuracy on both datasets, and removing the local module costs more (2.3% on MOSI) than removing the global one (1.1%).
  • Replacing the Mamba fusion with single-stream or multi-stream Mamba variants without explicit alignment hurts performance, meaning naive Mamba scanning does not by itself capture cross-modal correspondences.
  • At 6.4k tokens, AlignMamba uses 8.53 GB GPU memory versus 10.7 GB for single-stream and 20.3 GB for multi-stream Transformers, and 6.05 s inference time versus 36.13 s and 48.61 s; at 1024 tokens it needs 46.7G FLOPs versus 101.6G and 203.2G.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper leaves open is training with modality dropout so the alignment modules experience missing inputs during training; such a protocol would clarify whether the reported incomplete-fusion robustness comes from the alignment losses or from an unstated input-construction choice.
  • Because the OT alignment anchors audio and video to the language token grid, the method implicitly assumes language is the most informative modality; for tasks without a dominant modality, an anchor-selection or symmetric alignment rule would be needed, which the paper does not address.
  • The relaxed OT plan is essentially a row-wise nearest-neighbor assignment, so the local alignment module can be seen as a sparse, interpretable attention that could be transplanted into other backbones or used to initialize fusion without extra parameters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes AlignMamba, a Mamba-based multimodal fusion method that adds two alignment mechanisms before feeding unimodal features into a Mamba backbone: an optimal-transport-inspired local alignment module that matches tokens across modalities, and an MMD-based global alignment loss that aligns feature distributions. The method is evaluated on CMU-MOSI and CMU-MOSEI for complete multimodal fusion and for incomplete fusion under 10%–70% missing modality rates, reporting improved accuracy and large reductions in GPU memory and inference time relative to Transformer baselines. The paper also includes ablations of the alignment components and a qualitative transport-plan analysis.

Significance. If the reported results are reproducible, AlignMamba offers a practical way to bring the linear-complexity benefits of Mamba to multimodal fusion while mitigating Mamba's limited cross-modal interaction through explicit token-level and distribution-level alignment. The efficiency gains (e.g., 83.3% inference-time reduction at 6.4k tokens) are substantial and well aligned with the paper's motivation. However, the paper's central empirical claims, especially the incomplete-fusion robustness results in Table 1, rest on an unstated missing-modality handling procedure, and the complete-fusion results lack essential training details. The conceptual contribution is clear, but the experimental verification is currently incomplete. The paper would be strengthened by a precise description of how missing modalities are represented, the exact training protocol, and the values of the key hyperparameters (lambda, sigma, architecture sizes).

major comments (1)
  1. [Section 3.2] The term 'optimal transport' is used loosely. The relaxation in Eq. (4) removes the column-sum constraint, and the closed-form solution in Eq. (5) is a nearest-neighbor assignment of each row to its minimal-cost column, not a coupling between the two marginal distributions. This is a valid algorithmic choice, but the paper's claim to be 'grounded in Optimal Transport' is overstated. The authors should either clarify that they use a one-sided assignment relaxation or discuss the relationship to OT more precisely. This issue does not invalidate the empirical results, but it affects the framing of the method.
minor comments (5)
  1. [Section 3.3] Equation (8) uses the notation T for the number of samples in both sequences, but the aligned sequences have length T_l while the original audio and video sequences may have different lengths. The notation should be adjusted to avoid ambiguity.
  2. [Section 3.2] In Eq. (4), the relaxed constraints are written with a single row-sum constraint; it would be clearer to also explicitly state the non-negativity constraint and the range of indices, as done in Eq. (2).
  3. [Section 4.4] The modality ablation for 'w/o Language' mentions that 'we only align the audio modality with the video modality,' but the corresponding procedure is not described in Section 3. This is related to the missing-modality protocol issue and should be clarified.
  4. [Section 4.3] Figures 3 and 4 are referenced but the actual plots are not visible in the manuscript text; please ensure the figures are included and labeled with clear axes and legends.
  5. [Section 4.5.1] The A-distance results in Table 4 are computed on the model's own learned representations. While this is a reasonable diagnostic, the caption or text should note that this measures internal consistency rather than an external property of the data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AlignMamba's SOTA and efficiency claims are empirical, benchmarked against external methods, and none of the method's equations reduce to its inputs.

full rationale

The central claims are empirical. Table 2 measures accuracy/F1 against external baselines (ICCN, MulT, MTMD, etc.), and Table 1 does the same for incomplete fusion; neither metric is a fitted parameter reused as a prediction. The OT local alignment (Eqs. 1-6) is a standard relaxed optimal-transport mapping borrowing from external ref [12], and the MMD global loss (Eqs. 7-10) is the textbook kernel MMD; neither is defined in terms of the reported accuracies. Ablations in Table 3 show that removing each proposed component degrades downstream accuracy, which is independent evidence of contribution rather than a circular validation. Efficiency claims are direct measurements of FLOPs, GPU memory, and inference time, consistent with Mamba's known linear complexity. Table 4's A-distance is an internal diagnostic computed on the model's own representations, so it only confirms that the explicitly optimized alignment reduces a related distributional discrepancy; it is not load-bearing for the SOTA claim and no equation-level reduction to the method's output is present. The incomplete-fusion protocol for feeding missing modalities into Eq. (11) and Eq. (5) is under-specified in the manuscript, which undermines reproducibility and comparability with recovery-based baselines, but this is an experimental-reporting gap, not circularity. There are no load-bearing self-citations; all cited prior work (Mamba, relaxed OT, MMD, benchmark baselines) is external.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. The load-bearing additions are design choices: the language-anchor convention, the one-sided relaxation of transport, and the timestep-interleaved sequence. The most important unstated procedural assumption is how missing modalities are fed to the network in the incomplete-fusion experiments.

free parameters (3)
  • lambda (alignment loss weight) = not reported
    Eq. 12 combines Ltask and Lalign with a weight lambda; the paper reports no value or tuning procedure, so the balance between task and alignment objectives is a free choice.
  • sigma (Gaussian kernel bandwidth) = not reported
    The MMD kernel in Eq. 9 depends on sigma; no value or selection rule is given, and results may be sensitive to it.
  • architecture sizes (Mamba layers, hidden dimension d) = not reported
    The unimodal feature dimension and number of Mamba layers are never specified, which prevents reproduction of the efficiency and accuracy numbers.
assumptions (4)
  • domain assumption Language is treated as the anchor; audio and video are aligned to it.
    Section 3.1 states that Xl is the anchor for both OT and MMD, but no experiment compares alternative anchors.
  • ad hoc to paper The one-sided relaxation of the OT constraints still produces useful alignment.
    Eq. 4 removes the incoming sum constraint and Eq. 5 returns a per-row argmin. This is not a transport plan, yet the paper calls it OT-based alignment.
  • standard math Gaussian MMD captures the distributional difference that matters for fusion.
    Section 3.3 relies on standard RKHS theory; the choice of kernel bandwidth sigma and equal token weights is not justified.
  • domain assumption Interleaving tokens by timestep lets the Mamba scan capture cross-modal dependencies.
    Section 3.4, Eq. 11 asserts this ordering enables both intra- and inter-modal dependencies; no ablation isolates this ordering.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment." pith.science (2026). https://pith.science/paper/QJCVSLHV

@misc{pith2026241200833,
  author       = {Pith},
  title        = {Pith review of: AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QJCVSLHV}},
  note         = {Machine review of arXiv:2412.00833}
}
read the original abstract

Cross-modal alignment is crucial for multimodal representation fusion due to the inherent heterogeneity between modalities. While Transformer-based methods have shown promising results in modeling inter-modal relationships, their quadratic computational complexity limits their applicability to long-sequence or large-scale data. Although recent Mamba-based approaches achieve linear complexity, their sequential scanning mechanism poses fundamental challenges in comprehensively modeling cross-modal relationships. To address this limitation, we propose AlignMamba, an efficient and effective method for multimodal fusion. Specifically, grounded in Optimal Transport, we introduce a local cross-modal alignment module that explicitly learns token-level correspondences between different modalities. Moreover, we propose a global cross-modal alignment loss based on Maximum Mean Discrepancy to implicitly enforce the consistency between different modal distributions. Finally, the unimodal representations after local and global alignment are passed to the Mamba backbone for further cross-modal interaction and multimodal fusion. Extensive experiments on complete and incomplete multimodal fusion tasks demonstrate the effectiveness and efficiency of the proposed method.

Figures

Figures reproduced from arXiv: 2412.00833 by the authors.

Figure 1
Figure 1. Transformer leverages attention mechanisms to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. AlignMamba enhances multimodal Mamba by incorporating token-level alignment and distribution-level alignment, enabling [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Inference time comparison with varying lengths. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: GPU memory usage comparison with varying lengths. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 5
Figure 5. Figure 5: The learned optimal transport plan. We only show the transport plan between video and language modalities for brevity. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 28 canonical work pages

  1. [1]

    Deep canonical correlation analysis

    Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In Interna- tional conference on machine learning , pages 1247–1255. PMLR, 2013. 6

    work page 2013

  2. [2]

    Fusion-mamba for cross-modality object detection

    Wenhao Dong, Haodong Zhu, Shaohui Lin, Xiaoyan Luo, Yunhang Shen, Xuhui Liu, Juan Zhang, Guodong Guo, and Baochang Zhang. Fusion-mamba for cross-modality object detection. arXiv preprint arXiv:2404.09146, 2024. 1, 3, 7

    arXiv 2024

  3. [3]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 1, 3

    arXiv 2023

  4. [4]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R ´e. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing sys- tems, 34:572–585, 2021. 1, 3

    work page 2021

  5. [5]

    On the parameterization and initialization of diagonal state space models

    Albert Gu, Karan Goel, Ankit Gupta, and Christopher R ´e. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971–35983, 2022. 1, 3

    work page 2022

  6. [6]

    Improving mul- timodal fusion with hierarchical mutual information maxi- mization for multimodal sentiment analysis

    Wei Han, Hui Chen, and Soujanya Poria. Improving mul- timodal fusion with hierarchical mutual information maxi- mization for multimodal sentiment analysis. arXiv preprint arXiv:2109.00412, 2021. 5, 6

    arXiv 2021

  7. [7]

    Misa: Modality-invariant and-specific representations for multimodal sentiment analysis

    Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM international conference on multimedia , pages 1122–1131, 2020. 5, 6

    work page 2020

  8. [8]

    Pan-mamba: Effective pan-sharpening with state space model

    Xuanhua He, Ke Cao, Keyu Yan, Rui Li, Chengjun Xie, Jie Zhang, and Man Zhou. Pan-mamba: Effective pan-sharpening with state space model. arXiv preprint arXiv:2402.12192, 2024. 1, 3

    arXiv 2024

Show all 44 references
  1. [9]

    Self-supervised uni- modal label generation strategy using recalibrated modality representations for multimodal sentiment analysis

    Yewon Hwang and Jong-Hwan Kim. Self-supervised uni- modal label generation strategy using recalibrated modality representations for multimodal sentiment analysis. In Find- ings of the Association for Computational Linguistics: EACL 2023, pages 35–46, 2023. 6

    2023

  2. [10]

    Aobert: All- modalities-in-one bert for multimodal sentiment analysis.In- formation Fusion, 92:37–45, 2023

    Kyeonghun Kim and Sanghyun Park. Aobert: All- modalities-in-one bert for multimodal sentiment analysis.In- formation Fusion, 92:37–45, 2023. 6

    2023

  3. [11]

    Vilt: Vision- and-language transformer without convolution or region su- pervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In International conference on machine learning, pages 5583–5594. PMLR, 2021. 1, 2

    2021

  4. [12]

    From word embeddings to document distances

    Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Wein- berger. From word embeddings to document distances. In International conference on machine learning , pages 957–

  5. [13]

    Cross-attentional audio-visual fusion for weakly- supervised action localization

    Jun-Tae Lee, Mihir Jain, Hyoungwoo Park, and Sungrack Yun. Cross-attentional audio-visual fusion for weakly- supervised action localization. In International conference on learning representations, 2020. 1

    2020

  6. [14]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  7. [15]

    Visualbert: A simple and perfor- mant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and perfor- mant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. 1, 2

    1908 arXiv

  8. [16]

    Decoupled multi- modal distilling for emotion recognition

    Yong Li, Yuanzhi Wang, and Zhen Cui. Decoupled multi- modal distilling for emotion recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6631–6640, 2023. 5, 6

    2023

  9. [17]

    Mambadfuse: A mamba-based dual-phase model for multi-modality image fusion

    Zhe Li, Haiwei Pan, Kejia Zhang, Yuhua Wang, and Fengming Yu. Mambadfuse: A mamba-based dual-phase model for multi-modality image fusion. arXiv preprint arXiv:2404.08406, 2024. 1, 3

    2024 arXiv

  10. [18]

    Gcnet: Graph completion network for incomplete mul- timodal learning in conversation

    Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Gcnet: Graph completion network for incomplete mul- timodal learning in conversation. IEEE Transactions on pat- tern analysis and machine intelligence , 45(7):8419–8432,

  11. [19]

    Mtmamba: Enhancing multi- task dense scene understanding by mamba-based decoders

    Baijiong Lin, Weisen Jiang, Pengguang Chen, Yu Zhang, Shu Liu, and Ying-Cong Chen. Mtmamba: Enhancing multi- task dense scene understanding by mamba-based decoders. arXiv preprint arXiv:2407.02228, 2024. 3

    2024 arXiv

  12. [20]

    Multi-task momentum distil- lation for multimodal sentiment analysis

    Ronghao Lin and Haifeng Hu. Multi-task momentum distil- lation for multimodal sentiment analysis. IEEE Transactions on Affective Computing, 2023. 5, 6

    2023

  13. [21]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485,

    arXiv

  14. [22]

    Robomamba: Mul- timodal state space model for efficient robot reasoning and manipulation

    Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Mul- timodal state space model for efficient robot reasoning and manipulation. arXiv preprint arXiv:2406.04339, 2024. 1

    2024 arXiv

  15. [23]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019. 1, 2

    2019

  16. [24]

    Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis

    Sijie Mai, Ying Zeng, Shuangjia Zheng, and Haifeng Hu. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis. IEEE Transactions on Affec- tive Computing, 2022. 5, 6

    2022

  17. [25]

    Found in translation: Learn- ing robust joint representations by cyclic translations be- tween modalities

    Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnab´as P´oczos. Found in translation: Learn- ing robust joint representations by cyclic translations be- tween modalities. In Proceedings of the AAAI conference on artificial intelligence, pages 6892–6899, 2019. 6

    2019

  18. [26]

    Vl-mamba: Ex- ploring state space models for multimodal learning

    Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. Vl-mamba: Ex- ploring state space models for multimodal learning. arXiv preprint arXiv:2403.13600, 2024. 1, 3

    2024 arXiv

  19. [27]

    Integrating multimodal information in large pre- 9 trained transformers

    Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. Integrating multimodal information in large pre- 9 trained transformers. In Proceedings of the conference. Asso- ciation for Computational Linguistics. Meeting , pa...

    2020

  20. [28]

    Learning relationships between text, audio, and video via deep canonical correlation for multimodal lan- guage analysis

    Zhongkai Sun, Prathusha Sarma, William Sethares, and Yingyu Liang. Learning relationships between text, audio, and video via deep canonical correlation for multimodal lan- guage analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8992–8999, 2020. 5, 6

    2020

  21. [29]

    Lxmert: Learning cross- modality encoder representations from transformers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross- modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019. 1, 2

    1908 arXiv

  22. [30]

    Multimodal transformer for unaligned multimodal language sequences

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, page 655...

    2019

  23. [31]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 1, 2

    2017

  24. [32]

    Topics in optimal transportation

    C ´edric Villani. Topics in optimal transportation. American Mathematical Soc., 2021. 3

    2021

  25. [33]

    On deep multi-view representation learning

    Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International conference on machine learning, pages 1083–

  26. [34]

    Incomplete multimodality-diffused emotion recognition

    Yuanzhi Wang, Yong Li, and Zhen Cui. Incomplete multimodality-diffused emotion recognition. Advances in Neural Information Processing Systems, 36, 2024. 5, 6

    2024

  27. [35]

    Disentangled representation learning for multimodal emotion recognition

    Dingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du, and Lihua Zhang. Disentangled representation learning for multimodal emotion recognition. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1642– 1651, 2022. 6

    2022

  28. [36]

    Confede: Contrastive feature decomposition for multimodal sentiment analysis

    Jiuding Yang, Yakun Yu, Di Niu, Weidong Guo, and Yu Xu. Confede: Contrastive feature decomposition for multimodal sentiment analysis. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7617–7630, 2023. 5, 6

    2023

  29. [37]

    Cm-bert: Cross- modal bert for text-audio sentiment analysis

    Kaicheng Yang, Hua Xu, and Kai Gao. Cm-bert: Cross- modal bert for text-audio sentiment analysis. In Proceed- ings of the 28th ACM international conference on multime- dia, pages 521–528, 2020. 6

    2020

  30. [38]

    Learning modality-specific representations with self-supervised multi- task learning for multimodal sentiment analysis

    Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. Learning modality-specific representations with self-supervised multi- task learning for multimodal sentiment analysis. In Proceed- ings of the AAAI conference on artificial intelligence , pages 10790–10797, 2021. 5, 6

    2021

  31. [39]

    Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos

    Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259, 2016. 5

    2016 arXiv

  32. [40]

    Multimodal lan- guage analysis in the wild: Cmu-mosei dataset and inter- pretable dynamic fusion graph

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal lan- guage analysis in the wild: Cmu-mosei dataset and inter- pretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational...

    2018

  33. [41]

    Multimodal chain-of- thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023. 1

    2023 arXiv

  34. [42]

    Cobra: Extending mamba to multi-modal large language model for efficient inference

    Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint arXiv:2403.14520, 2024. 1, 3, 7

    2024 arXiv

  35. [43]

    Missing modal- ity imagination network for emotion recognition with un- certain missing modalities

    Jinming Zhao, Ruichen Li, and Qin Jin. Missing modal- ity imagination network for emotion recognition with un- certain missing modalities. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguis- tics and the 11th International Joint Conference...

    2021

  36. [44]

    Multi-channel weight-sharing autoencoder based on cascade multi-head attention for multimodal emo- tion recognition

    Jiahao Zheng, Sen Zhang, Zilu Wang, Xiaoping Wang, and Zhigang Zeng. Multi-channel weight-sharing autoencoder based on cascade multi-head attention for multimodal emo- tion recognition. IEEE Transactions on Multimedia , 25: 2213–2225, 2022. 1 10

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.