Pith. sign in

REVIEW 6 major objections 6 minor 32 references

Partitioner Guided Modal Learning Framework

T0 review · 6 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a trainable modal partitioner, which splits each modality's representation into uni-modal and paired-modal components that are learned separately and then decoded back together, improves multimodal learning across…

desk verdict A plausible new multimodal training module with a real capacity confound and an unvalidated partition semantics; worth refereeing with revisions demanded. read the letter →

arxiv 2507.11661 v1 pith:FGCC3BP4 submitted 2025-07-15 cs.CL cs.AI

Guimin Hu , Yi Xin , Lijie Hu , Zhihong Zhu , Hasti Seifi This is my paper · ORCID
classification cs.CLcs.AI
keywords multimodallearningmodalpartitioneruni-modalfeaturespaired-modalmodalitylazinessrepresentationsentimentanalysisemotionrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a learned multimodal representation is not a single block: part of it carries information learnable from that modality alone, and part carries information that only emerges through cross-modal interaction. PgM is a trainable module that splits each modality's representation into those two parts with a cumulative-softmax gate, trains a dedicated Transformer learner on each part, and reconstructs the original representation from the two pieces. The claim is that this separated training improves downstream multimodal performance and can be dropped into existing multimodal models. The paper reports gains over simple fusion baselines on sentiment analysis, emotion recognition, cross-modal retrieval, and image-text classification, and improved results for four published multimodal models.

What carries the argument

The central mechanism is the cumulative-softmax gate, defined as $\mathrm{cumsum}(\mathrm{softmax}(\cdot))$, which produces a binary-like mask that separates a modality vector into a leading uni-modal block and a trailing paired-modal block. The paper iterates this gating $N$ times ($N=3$ in experiments), letting the split adapt during training. The two learners are Transformer stacks with padding masks that set the opposite partition's positions to $-\infty$ in attention, so each learner sees only its own partition; the uni-paired decoder then concatenates the two learned parts and reconstructs the original modality. The split is supervised by classification losses (UFC, PFC) and a reconstruction loss (UPR).

What would settle it

Train PgM on MOSI and MELD with the learned cumulative-softmax gates replaced by fixed random binary masks of exactly the same shape and capacity, keeping all losses and learners unchanged; if accuracy does not drop substantially, the learned split is not what produces the reported gains.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the learned representation of each modality can be partitioned into a uni-modal component and a paired-modal component, and that learning these components separately—rather than feeding a single fused representation to the task head—yields better multimodal performance. The partitioner uses a cumulative-softmax gate to choose a split point per iteration, the two learners apply attention masks so each sees only its own part, and the decoder forces the two parts to together reconstruct the original representation. PgM is trained with three losses: uni-modal feature classification, paired-modal feature classification, and reconstruction. When attached to downstream tasks, it outperforms six basic fusion baselines on all four benchmarks and improves Self-MM, MMIM, UniMSE, and UniMEEC when it replaces their multimodal learning module.

Load-bearing premise

The whole result rests on the gates actually separating each modality's representation into semantically distinct uni-modal and paired-modal parts, even though no external label or ground truth verifies that the split means what the paper says it means.

Editorial extensions

If this is right

  • Adding PgM to an existing multimodal model should improve that model's scores on sentiment and emotion benchmarks; the paper shows this for Self-MM, MMIM, UniMSE, and UniMEEC.
  • The optimal balance between uni-modal and paired-modal features is task-specific: the paper's visualizations show text relies mostly on uni-modal features while audio relies more on paired-modal features, and these proportions change between sentiment and emotion tasks.
  • Because the partitioner is trained end-to-end with the task, the split can adjust to the downstream objective rather than being fixed.
  • Separate learning rates for modalities and partitions can mitigate modality laziness, the failure mode where one modality stops learning.
  • PgM works across text, vision, and audio, so it is a general add-on rather than a task-specific architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gates could serve as a diagnostic: the fraction of features assigned to the paired-modal partition measures how much a modality depends on cross-modal information for a given task, but the paper only reports these distributions without using them as a metric.
  • The same partitioner idea could extend to more than three modalities or to non-linguistic signals such as sensor and biometric data, which the paper lists as future work but does not test.
  • A natural stress test the paper does not run is to fix the gates to random binary masks and retrain; if performance holds, the learned split would not be the active ingredient.
  • The two-stage training schedule (pretrain PgM, then joint fine-tune) could likely be compressed into a single warm-up schedule, but the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. This paper introduces PgM, a partitioner-guided multimodal learning framework that segments each modality representation into uni-modal and paired-modal partitions via cumulative-softmax gates (Eqs. 1-3). Two Transformer-based learners process the masked partitions, a decoder reconstructs the original representation, and three losses (UFC, PFC, UPR; Eqs. 9-12) supervise learning. The framework is pretrained and then jointly fine-tuned with downstream tasks. Experiments on MOSI, MELD, Wikipedia, and UPMC Food 101 report large gains over simple fusion baselines, and integration with Self-MM, MMIM, UniMSE, and UniMEEC yields consistent improvements. The paper also visualizes the learned distributions of uni-modal and paired-modal features.

Significance. If the central claim were established, PgM would be a useful model-agnostic module: it is applicable to four tasks, compatible with several existing models, and the partition idea is a plausible response to modality-laziness. The architecture is described in enough detail to reproduce, and the transfer experiments are a positive sign. However, the main empirical claim is currently stronger than the evidence: the control conditions do not isolate the partitioner mechanism, the PFC loss is self-referential, and no variance is reported. The paper would be substantially strengthened by capacity-matched baselines, sham-partition controls, multiple seeds, and an independent probe of partition semantics. In its current form the evidence supports the capacity of the added Transformer modules more directly than it supports the partitioning story.

major comments (6)
  1. [Section 4.4, Table 2, Table 5] The baselines in Table 2 (concatenation, add, element-wise maximum, linear fusion, MLP) add almost no parameters on top of frozen encoders, whereas PgM adds 22.36M trainable parameters (Table 5). The consistent gains in Table 2 therefore do not show that the learned uni-modal/paired-modal split is beneficial; they may reflect the extra Transformer capacity. A capacity-matched control (e.g., one Transformer stack of the same total size applied to the concatenated modality representations without the partitioner) and a sham-partition control (e.g., fixed or random gates) are needed before the causal claim in Section 4.5 can be supported.
  2. [Table 3] The '-w/o Modal Partitioner' row reports 48.58/50.69 ACC-2 and 44.31/47.46 F1 on MOSI, which is essentially the Single-Modal (Audio) baseline in Table 2 (48.58/50.36 ACC-2, 44.31/46.47 F1). Removing the partitioner also removes the uni-modal learner, paired-modal learner, and decoder by construction, so this ablation does not isolate the partitioner; it is a degenerate single-modality control. An ablation that removes only the partitioner while retaining the Transformer learners would be the meaningful control.
  3. [Section 3.3, Eqs. (9)-(10)] The PFC loss is trained with binary labels Ou and Op that are assigned by which partitioner branch produced the feature. Since the label is determined by path identity, a classifier can minimize LPFC perfectly without the partitions having any semantic relation to uni-modal versus cross-modal content. The paper therefore provides no independent verification that the split is meaningful. External probes (e.g., testing whether uni-modal features alone suffice for single-modality classification and whether paired-modal features improve cross-modal transfer) or a control with randomized partition labels would address this circularity.
  4. [Section 3.2, Eqs. (4)-(7)] The masking operation is dimensionally inconsistent as written. The mask M_z is defined as a vector over feature dimensions (e.g., [1,...,1,0,...,0] in the text), but it is added to QK^T, which has shape S by S for a sequence of length S. This makes the equations un-implementable in the standard Transformer formulation. Please clarify whether attention is over sequence positions or feature positions and give the exact shapes and broadcasting rules.
  5. [Section 3.2, Appendix B] The paired-modal learner operates on a single modality representation with a mask that excludes the uni-modal positions; it never attends to other modalities. Cross-modal interaction first appears in the downstream fusion Transformer in Appendix B, after PgM has produced its outputs. The features called paired-modal therefore cannot literally encode cross-modal interactions within PgM. This terminological gap should be addressed either by revising the interpretation of the two partitions or by adding cross-modal attention inside the framework.
  6. [Section 4.2, Section 4.3, Table 4] No experiment reports variance: all numbers are apparently single runs, and no random seeds or number of runs are given. Several transfer-table gains are small (e.g., MMIM +1.22 ACC-2 and +1.27 F1; UniMEEC +0.56 ACC), so without multiple seeds and error bars these improvements are not distinguishable from training noise. Please report mean and standard deviation over at least three seeds for the main results and the transfer experiments.
minor comments (6)
  1. [Eq. (3)] The symbols upperc and downerc are never defined; they presumably refer to upper(i)_m and downer(i)_m from Eq. (2). Please correct this notation.
  2. [Figure 3] Within a single modality the reported uni-modal and paired-modal percentages sum to more than 100% (e.g., text 60.54 + 45.61 = 106.15). If overlap is allowed, explain how the percentages are computed and why they may exceed 100%.
  3. [Table 3] The header contains the misspelling Precison@10, and the table formatting joins rows together; please fix both.
  4. [Table 1 and Section 4.1] The dataset name is inconsistent: Table 1 says UMPC Food 101 while Section 4.1 says UPMC Food 101. The correct name should be used throughout.
  5. [Footnote 1] The definition of cumsoftmax is truncated in the footnote; please provide a complete, numbered equation.
  6. [Section 4.3] One of the three advertised benefits is different learning rates across modalities and partitions, but the experiments set the same learning rate for the uni-modal and paired-modal learners (0.0001) and no experiment varies these rates. Please either demonstrate this benefit or remove it from the list of contributions.

Circularity Check

2 steps flagged · score 4.0 of 10

Task accuracy is independently measured, but the uni/paired-modal split is self-defined: the PFC loss trains on labels assigned by the partitioner's own branches, so the semantic partition claim is circular and the reported partition statistics are tautological.

  1. self definitional [Section 3.3, Eq. (10) (Paired-modal Feature Classification); interpreted in Section 4.8]
    "To ensure the paired-modal feature remains distinct from the uni-modal feature, we input the paired-modal feature of each iteration into a classifier designed to differentiate between uni-modal and paired-modal representations. ... Ou = [1 0; 1 0], Op = [0 1; 0 1] where Ou and Op denote the ground labels of uni-modal and paired-modal representation, respectively."

    The PFC classifier's ground-truth labels are not independent: Ou is attached to the output of the uni-modal learner and Op to the output of the paired-modal learner, and those two feature sets are already segregated by masks derived from the partitioner's own cumsoftmax gates (Eqs. 1-4). A classifier can separate them by path identity alone, so the loss cannot certify that one partition contains 'uni-modal' information and the other contains 'paired-modal' (cross-modal) information. The semantic distinction is enforced by construction rather than measured. Section 4.8 then reports percentages of these self-labeled partitions (e.g., '60.54% of features are uni-modal') as if they were empirically established contributions.

  2. self definitional [Section 3.1, Eqs. (1)-(3); Section 3.2]
    "The uni-modal features store information for uni-modal training, while the paired-modal features capture information for cross-modal interactions (Du et al., 2023). ... In the first iteration, um = pm = Im, m ∈ {m1, m2, · · ·, mN} denotes specific modality."

    Here 'paired-modal features' are defined as one side of a cumsoftmax split of the single-modality representation Im; no cross-modal input is involved at this stage, and the paired-modal learner takes query, key, and value from the same modal representation (Section 3.2). Thus the category 'paired-modal' is stipulated by the partitioner's own gates. Any later measurement of the 'paired-modal' partition, such as the percentages in Section 4.8, is a measurement of that stipulation, not of independently verified cross-modal content.

full rationale

The downstream task numbers are honest external evaluations: PgM is compared on MOSI, MELD, Wikipedia, and Food-101 accuracy/F1/MAP, and those outcomes are not derived from the partition labels. So the paper is not wholly circular. It also does not use unverified self-citations as load-bearing, nor does it fit a parameter and then call a closely related number a prediction. The circular content is in the conceptual claim about what the partitioner separates. The only loss aimed at giving the split meaning is PFC, whose labels are defined by the partitioner's own branches (Eq. 10), so 'uni-modal vs paired-modal' is true by construction, and the visualization in Section 4.8 is a tally of self-referential labels. Additionally, the ablation in Table 3 removes all learned modules at once and collapses to the Single-Modal (Audio) MOSI row of Table 2, so it cannot isolate the partition mechanism; this is a missing-control problem rather than a circular reduction, but it compounds the concern. The Limitations section acknowledges added parameters and simple fusion but does not acknowledge the self-referential labeling of the partitions. Overall score 4: the central empirical claims retain independent content, but the framework's scientific characterization of the uni/paired split is circular.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The framework's central decomposition assumption and the auxiliary-loss semantics are self-imposed; all remaining design choices are hand-set hyperparameters. The empirical claims are therefore supported only by the experiments, not by derivation.

free parameters (8)
  • alpha = 0.5
    Weight on the PgM pretraining objective in LD (Equation 13), chosen without reported sensitivity analysis.
  • beta = 1.0
    Weight on the downstream task loss in LD (Equation 13), hand-set.
  • iteration count N = 3
    Number of partitioner iterations in Section 3.1, hand-set.
  • pretraining epochs N1st = 20
    Duration of the first training stage, Section 4.3, hand-set.
  • joint training epochs N2st = 50
    Duration of the second training stage, Section 4.3, hand-set.
  • learning rate for uni and paired learners = 0.0001
    Learning rate for the two learners, Section 4.3, hand-set.
  • learning rate for decoder = 0.001
    Learning rate for the uni-paired decoder, Section 4.3, hand-set.
  • overall learning rate = 3e-4
    Global learning rate, Section 4.3, hand-set.
assumptions (5)
  • domain assumption Multimodal representations can be decomposed into uni-modal and paired-modal components with possible overlap.
    Stated in Section 3.1 and formalized in Equations (1)-(3).
  • ad hoc to paper Cumulative softmax gates approximate binary gates and segment neurons by threshold.
    Used to derive gates in Equation (1); no guarantee the segmentation is semantically correct.
  • ad hoc to paper The UFC and PFC classification losses make the partitions learn the intended semantics.
    Section 3.3; PFC labels come from the partitioner, so the semantics are self-referential.
  • domain assumption Frozen pretrained encoders give representations sufficient for partitioning and reconstruction.
    T5, ViT, and AST encoders are frozen; the partitioner and decoders operate on their outputs, Section 4.3 and Appendix C.1.
  • domain assumption The concatenation-and-FFN downstream fusion fairly uses the partitioned features.
    The Limitations section concedes the fusion remains simple and may underuse the partitions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Partitioner Guided Modal Learning Framework." pith.science (2026). https://pith.science/paper/FGCC3BP4

@misc{pith2026250711661,
  author       = {Pith},
  title        = {Pith review of: Partitioner Guided Modal Learning Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGCC3BP4}},
  note         = {Machine review of arXiv:2507.11661}
}
read the original abstract

Multimodal learning benefits from multiple modal information, and each learned modal representations can be divided into uni-modal that can be learned from uni-modal training and paired-modal features that can be learned from cross-modal interaction. Building on this perspective, we propose a partitioner-guided modal learning framework, PgM, which consists of the modal partitioner, uni-modal learner, paired-modal learner, and uni-paired modal decoder. Modal partitioner segments the learned modal representation into uni-modal and paired-modal features. Modal learner incorporates two dedicated components for uni-modal and paired-modal learning. Uni-paired modal decoder reconstructs modal representation based on uni-modal and paired-modal features. PgM offers three key benefits: 1) thorough learning of uni-modal and paired-modal features, 2) flexible distribution adjustment for uni-modal and paired-modal representations to suit diverse downstream tasks, and 3) different learning rates across modalities and partitions. Extensive experiments demonstrate the effectiveness of PgM across four multimodal tasks and further highlight its transferability to existing models. Additionally, we visualize the distribution of uni-modal and paired-modal features across modalities and tasks, offering insights into their respective contributions.

Figures

Figures reproduced from arXiv: 2507.11661 by the authors.

Figure 1
Figure 1. Illustration of modal partitioner. Modal parti [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of PgM: (a) Segmentation process of partitioner,(b)Architecture of PgM. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of uni-modal and paired-modal [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Overview of training framework with multi [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Loss variation curves during the second stage, [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 16 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Relja Arandjelovic and Andrew Zisserman. 2017. https://doi.org/10.1109/ICCV.2017.73 Look, listen and learn . In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 , pages 609--617. IEEE Computer Society

    doi:10.1109/iccv.2017.73 2017

  4. [4]

    Chenzhuang Du, Jiaye Teng, Tingle Li, Yichen Liu, Tianyuan Yuan, Yue Wang, Yang Yuan, and Hang Zhao. 2023. https://proceedings.mlr.press/v202/du23e.html On uni-modal feature learning in supervised multi-modal learning . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine...

    work page 2023

  5. [5]

    Yash Goyal, Tejas Khot, Douglas Summers - Stay, Dhruv Batra, and Devi Parikh. 2017. https://doi.org/10.1109/CVPR.2017.670 Making the V in VQA matter: Elevating the role of image understanding in visual question answering . In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 6325--6334....

    doi:10.1109/cvpr.2017.670 2017

  6. [6]

    Nitin Grover, Aviral Chharia, Rahul Upadhyay, and Luca Longo. 2023. https://doi.org/10.1109/TNSRE.2023.3237375 Schizo-net: A novel schizophrenia diagnosis framework using late fusion multimodal deep learning on electroencephalogram-based brain connectivity indices . IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:464--473

    arXiv 2023

  7. [7]

    Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis

    Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 9180--9192

    work page 2021

  8. [8]

    Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, and Yongbin Li. 2022. Unimse: Towards unified multimodal sentiment analysis and emotion recognition. arXiv preprint arXiv:2211.11256

    arXiv 2022

Show all 32 references
  1. [9]

    Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Lijie Hu, Hasti Seifi, and Jiayuan Xie. 2024. Unimeec: Towards unified multimodal emotion recognition and emotion cause. arXiv preprint arXiv:2404.00403

    2024 arXiv

  2. [10]

    Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. 2021. https://proceedings.neurips.cc/paper/2021/hash/5aa3405a3f865c10f420a4a7b55cbff3-Abstract.html What makes multi-modal learning better than single (provably) . In Advances in Neural Information ...

    2021

  3. [11]

    Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. 2022. https://proceedings.mlr.press/v162/huang22e.html Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably) . In International Conference on Machine Learning,...

    2022

  4. [12]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  5. [13]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888--12900. PMLR

    2022

  6. [14]

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694--9705

    2021

  7. [15]

    Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders S gaard, et al. 2024. Foodieqa: A multimodal dataset for fine-grained understanding of chinese food culture. In Proceedings of the 2024 Conference on Empirical ...

    2024

  8. [16]

    Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, Jinming Zhao, Ziyang Ma, Xie Chen, et al. 2024. Mer 2024: Semi-supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition. In Proceedings of the 2nd International W...

    2024

  9. [17]

    Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis - Philippe Morency, and Ruslan Salakhutdinov. 2022. https://doi.org/10.48550/ARXIV.2207.00056 Multiviz: An analysis benchmark for visualizing and understanding multimodal models . CoRR, abs/...

    2022 doi arXiv

  10. [18]

    Huisheng Mao, Ziqi Yuan, Hua Xu, Wenmeng Yu, Yihe Liu, and Kai Gao. 2022. M-sena: An integrated platform for multimodal sentiment analysis. arXiv preprint arXiv:2203.12441

    2022 arXiv

  11. [19]

    Juan-Manuel P \'e rez-R \'u a, Valentin Vielzeuf, St \'e phane Pateux, Moez Baccouche, and Fr \'e d \'e ric Jurie. 2019. Mfas: Multimodal fusion architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6966--6975

    2019

  12. [20]

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. https://doi.org/10.18653/v1/p19-1050 MELD: A multimodal multi-party dataset for emotion recognition in conversations . In Proceedings of the 57th Conference of the Associa...

    2019 doi

  13. [21]

    Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. 2010. A new approach to cross-modal multimedia retrieval. In Proceedings of the 18th ACM international conference on Multimedia, pages 251--260

    2010

  14. [22]

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15...

    2022

  15. [23]

    Zico Kolter, Louis - Philippe Morency, and Ruslan Salakhutdinov

    Yao - Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis - Philippe Morency, and Ruslan Salakhutdinov. 2019. https://doi.org/10.18653/v1/p19-1656 Multimodal transformer for unaligned multimodal language sequences . In Proceedings of the 57th Conference of the ...

    2019 doi

  16. [24]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  17. [25]

    Weiyao Wang, Du Tran, and Matt Feiszli. 2020. https://doi.org/10.1109/CVPR42600.2020.01271 What makes training multi-modal classification networks hard? In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , page...

    2020

  18. [26]

    Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. 2015. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1--6. IEEE

    2015

  19. [27]

    Jiayuan Xie, Mengqiu Cheng, Xinting Zhang, Yi Cai, Guimin Hu, Mengying Xie, and Qing Li. 2025. Explicitly guided difficulty-controllable visual question generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25552--25560

    2025

  20. [28]

    Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovativ...

    2021

  21. [29]

    Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis - Philippe Morency. 2016. https://doi.org/10.1109/MIS.2016.94 Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages . IEEE Intell. Syst. , 31(6):82--88

    2016 doi

  22. [30]

    Yi Zhang, Mingyuan Chen, Jundong Shen, and Chongjun Wang. 2022. https://doi.org/10.1609/AAAI.V36I8.20895 Tailor versatile multi-modal learning for multi-label emotion recognition . In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference ...

    2022 doi

  23. [31]

    Zhihong Zhu, Xuxin Cheng, Guimin Hu, Yaowei Li, Zhiqi Huang, and Yuexian Zou. 2024 a . Towards multi-modal sarcasm detection via disentangled multi-grained multi-modal distilling. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language ...

    2024

  24. [32]

    Zhihong Zhu, Xianwei Zhuang, Yunyan Zhang, Derong Xu, Guimin Hu, Xian Wu, and Yefeng Zheng. 2024 b . Tfcd: Towards multi-modal sarcasm detection via training-free counterfactual debiasing. In Proc. of IJCAI

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.