Pith. sign in

REVIEW 5 major objections 5 minor 52 references

A Trustworthy Method for Multimodal Emotion Recognition

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Multimodal emotion model hits 0.90 trusted F1 by fusing on uncertainty

desk verdict The trusted-SOTA claim is unsupported because the thresholds are fitted on test labels; the architecture is a reasonable application of evidential fusion, but the evaluation needs an overhaul. read the letter →

arxiv 2508.07625 v1 pith:C64LNZ5O submitted 2025-08-11 cs.CV

Junxiao Xue , Xiaozhen Liu , Jie Wang , Xuecheng Wu , Bin Wu This is my paper · ORCID
classification cs.CV
keywords multimodalemotionrecognitionevidentialdeeplearningconfidenceestimationtrustedevaluationDempster-Shaferfusionuncertaintycross-entropylossIEMOCAP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes trusted emotion recognition (TER), a model that trains each audio and video branch to output an evidential confidence value alongside its class prediction, fuses the branches with a Dempster-Shafer combination rule, and trains with a trusted cross-entropy loss that rewards correct high-confidence predictions. It also introduces trusted precision and trusted recall, a trusted threshold chosen where the trusted precision-recall curve crosses the line $\mathrm{TP}=\mathrm{TR}$, and trusted accuracy and trusted F1 as evaluation metrics. On the IEMOCAP and Music-video benchmarks, TER is reported to match or beat state-of-the-art accuracy while achieving the best trusted F1 scores: 0.7511 and 0.9035, respectively.

What carries the argument

The evidential confidence module (based on the Dirichlet/evidence framework in [12]) computes belief mass $b_c$ and uncertainty $u$ from softplus-transformed logits, turning each branch's output into a subjective opinion. The Combining Beliefs module applies the Dempster-Shafer combination rule (Eq. 4) to fuse the two modalities. The trusted cross-entropy loss (Eq. 5) and the trusted threshold selection (intersection of the trusted P-R curve with $\mathrm{TP}=\mathrm{TR}$) carry the argument: they convert uncertainty estimates into a training signal and an evaluation protocol. Together they let the model prefer 'I don't know' over guessing, and let the evaluator measure predictions only wher

What would settle it

Recompute the trusted metrics in Tables 5-7 using a single global threshold (e.g., $u=0.5$) or a threshold selected on a held-out validation split; if TER no longer leads in trusted F1 or its trusted F1 no longer exceeds its standard F1, the claimed trusted superiority is an artifact of per-model test-set threshold selection.

Watch

Extended reading notes

Core claim

At the center of TER is the replacement of the softmax classifier with an evidential output: a softplus layer produces Dirichlet evidence per class, from which a belief mass $b_c$ and uncertainty $u$ are computed. Each modality therefore reports not just a label but how unsure it is. A combining-beliefs module fuses the two modalities with the Dempster-Shafer combination rule (Eq. 4), which discounts conflicting evidence and lets the more confident modality dominate. The trusted cross-entropy loss (Eq. 5) includes the uncertainty term so the model is penalized for high-uncertainty wrong answers as well as ordinary classification error. For evaluation, the trusted threshold for each model is

Load-bearing premise

The trusted threshold is fit per model to the test-set trusted precision-recall curve, so the trusted metrics assume it is fair to choose each model's confidence cut-off with the test answer key in hand; if the cut-off were fixed or chosen on a validation split, the reported trusted-F1 gaps could shrink or disappear.

Editorial extensions

If this is right

  • High-confidence predictions from TER are claimed to be correct more often than the model's overall accuracy, making the model usable selectively in risk-sensitive settings such as screening or triage.
  • The trusted-loss retrofit is nearly drop-in: swapping softmax for softplus and adding the uncertainty term to the loss improved trusted F1 of existing state-of-the-art models on IEMOCAP by an average of about 20% (Table 7).
  • The trusted evaluation criterion shifts model selection from raw accuracy to reliability: a slightly less accurate model with a higher trusted F1 becomes preferable for deployment.
  • Because the method is end-to-end and uses no manual feature engineering, it can be applied directly to raw video and audio, and the modular confidence scheme extends naturally to additional modalities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported trusted metrics depend on fitting each model's threshold to test labels; a validation-set-selected threshold or a fixed global threshold would likely reduce the gap between TER and the baselines, so part of the claimed advantage may be a property of the evaluation protocol rather than the model alone.
  • Evidential confidence values from different architectures retrofitted with the same module may not be directly comparable, since no calibration step aligns their scales; the per-model thresholds in Table 7 mask this.
  • The same trust-aware loss and trusted-F1 objective could transfer to other multimodal tasks where silent wrong answers are costly, such as medical imaging or robot perception.
  • A cleaner head-to-head test would report trusted F1 at a predefined threshold (e.g., $u=0.5$) to decouple model quality from threshold optimization.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes TER, an end-to-end audio-visual emotion recognition model in which Video Swin Transformer and Multi-VGGish branches produce evidential outputs; a confidence module converts softplus evidence into belief masses and an uncertainty mass, and a combining-beliefs module fuses modalities via Dempster-Shafer combination. Training uses a 'trusted cross-entropy' loss that adds an uncertainty term. The main novelty is a trusted evaluation criterion: trusted precision/recall, trusted accuracy, and trusted F1 are computed from high-confidence predictions, with the confidence threshold chosen as the intersection of the trusted P-R curve with y=x. Experiments on IEMOCAP and Music-video report that TER reaches state-of-the-art trusted F1 (0.7511 and 0.9035) and high accuracy on Music-video (82.40%), and that the trusted loss improves SOTA models' trusted performance.

Significance. Confidence-based late fusion and reliability evaluation are timely for affective computing. The DS fusion module is a reasonable adaptation of prior work [12], and the paper explicitly aims to provide a new evaluation criterion. If the evaluation protocol were sound, the trusted F1 metric would be a useful complement to accuracy in high-stakes applications. The paper is clearly structured and reports ablations of modality, fusion, and loss components. However, the reported trusted metrics are invalidated by test-set threshold selection, and the classification SOTA comparison is too narrow; the current experimental evidence does not support the central claims. The proposed trusted loss and fusion ideas may still be valuable after a rigorous re-evaluation.

major comments (5)
  1. [Sec. 3.3.2, Fig. 4, Tables 5-7] The trusted threshold is chosen per model as the intersection of the trusted P-R curve with the y=x line, but the P-R curve is built from test predictions and test labels. No validation split or fixed threshold is used. Consequently every trusted metric in Tables 2-7 is evaluated at a test-fitted operating point. This is label leakage: the threshold itself is a fitted parameter. It explains the surprising result that TER's trusted F1 (0.7511) exceeds its ordinary F1 (0.6007) and that the gaps over bc-LSTM (0.5642) are large. Please select thresholds on a held-out validation set (or fix a priori), report all trusted metrics at that threshold, and show error bars over seeds.
  2. [Sec. 3.3.3 and 3.3.4, Eqs. (7)-(10)] Eq. (10) defines Trusted Accuracy as HT/(HT+HF), which is identical to Trusted Precision in Eq. (7). Thus 'Trusted Acc.' in Tables 2-7 is not a new metric and carries no information beyond trusted precision. The paper should either acknowledge this equivalence or define a genuinely different trusted accuracy, e.g., the fraction of all test samples that are both correct and high-confidence.
  3. [Sec. 4.3, Table 6] The Music-video SOTA comparison includes only CIM, COGMEN, and MMGCN, three conversational models that were developed for text-based emotion recognition in conversation. The authors state that they disable the text feature input for these models. Those models are therefore evaluated outside their design setting, and no audio-visual or AVCA baseline is included. This does not support the claim that TER is state-of-the-art on Music-video (82.40%). The comparison should include recent audio-visual methods (e.g., [50]-[52] cited in Section 2) trained on the same visual+audio features and protocol.
  4. [Sec. 3.2, Eq. (5)] The trusted cross-entropy loss includes a term -u_i log uhat_i for the ground-truth uncertainty u_i, but the paper never specifies how u_i is defined for labeled examples. With the usual one-hot labels u_i=0, this term vanishes and the loss reduces to CE on the belief masses; with u_i>0 it changes the target distribution. This underspecification affects the central training contribution in Table 4 and Fig. 5. Please define how u_i is constructed (label smoothing? estimated uncertainty?) and ablate the choice.
  5. [Sec. 4, Tables 2-7] All reported numbers are single runs with no standard deviations or significance tests. Differences such as TER Acc. 0.6014 vs DAG-ERC 0.6794 on IEMOCAP are large, but the claimed trusted-performance advantages need repeated runs, especially because the threshold selection step adds another source of variance. Report mean±std over at least 3-5 seeds and a paired significance test for the main claims.
minor comments (5)
  1. [Sec. 3.3.4, Sec. 4.4] Typos: 'Accuarcy' in the Section 3.3.4 heading and 'Performence' in the Section 4.4 heading.
  2. [Fig. 4 caption] Caption says 'branchs'; should be 'branches'.
  3. [Sec. 4.2.1] The phrase 'dissolution research' should be 'ablation study'.
  4. [Table 7 caption] The caption mentions 'six evaluation metrics,' but the table lists five performance metrics plus the threshold; the wording should be clarified.
  5. [Eq. (4)] The DS combination rule is presented in simplified form; please provide a direct citation or a short derivation to [12].

Circularity Check

1 steps flagged · score 6.0 of 10

Trusted SOTA claim rests on a test-fitted threshold: trusted metrics are computed at operating points chosen from the evaluation labels themselves.

  1. fitted input called prediction [Section 3.3.2 (Trusted Threshold), Eqs. (7)-(10); results in Tables 5 and 7]
    "A set of data on the (TR,TP) based on confidence can be obtained by adjusting different confidence thresholds. This data can be used to construct a Trusted P-R curve, as shown in Fig.4. To balance trusted precision and trusted recall, the y = x line was introduced, and the threshold corresponding to the intersection point with the P-R curve can be selected as the confidence threshold for the training model."

    The trusted threshold is selected per model as the intersection of the trusted precision-recall curve with the y=x line, and the same (TR,TP) data is then used in Eqs. (7)-(10) to compute trusted accuracy and trusted F1. The paper does not state that this threshold is chosen on a held-out validation set; the tables report per-model thresholds and trusted metrics on the same evaluation sets. Thus each model is allowed to choose a favorable operating point with access to the test labels, and the reported trusted F1 gaps (e.g., TER 0.7511 vs bc-LSTM 0.5642 on IEMOCAP) compare models at different test-fitted operating points. This also explains the otherwise surprising result that TER's trusted F1 exceeds its ordinary F1: a threshold tuned on ground-truth labels can always be moved to a favora

full rationale

The paper's architectural core (evidential confidence module and Dempster-Shafer combination) is inherited from external prior work by Han et al. [12], not from the present authors, and the standard classification results (e.g., 82.40% Acc on Music-video) are independent benchmark numbers. I do not find a self-citation chain or a definitional equivalence in the fusion or loss design. However, the headline 'trusted performance' evaluation is not an independent fixed-procedure metric. In Section 3.3.2, the trusted threshold is chosen per model as the intersection of the trusted P-R curve with y=x on the evaluation data; Eqs. (7)-(10) then compute trusted precision, recall, accuracy, and F1 at that chosen threshold. Since the threshold is a per-model free parameter fitted to the same labels used to compute the metric, the trusted SOTA comparisons in Tables 5 and 6, and the 'trusted improvement' claims in Table 7, compare models at different operating points selected with access to ground truth. This does not invalidate the conventional accuracy/F1 comparisons, but it means the central trusted-superiority claim reduces in part to a fitted threshold being presented as an evaluation criterion. Score 6 reflects that the standard classification claim retains independent content while the trusted-performance claim is substantially self-referential.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The paper's claims depend on (1) the transferred evidential framework from Han et al. [12] and (2) the test-set-fitted trusted threshold. The free parameter is the per-model trusted threshold reported in Table 7. No new physical or conceptual entities are postulated.

free parameters (1)
  • trusted threshold (uncertainty cutoff) = e.g., 0.5341, 0.4608, 0.6481, etc. in Table 7
    Selected per model as the intersection of the test-set Trusted P-R curve with y=x (Sec. 3.3.2, Fig. 4); all trusted metrics depend on it.
assumptions (2)
  • domain assumption Evidential deep learning formulation (softplus evidence, Dirichlet confidence, DS combination) from Han et al. [12] is valid for multimodal emotion features.
    Adopted in Sec. 3.1.3-3.1.4 with no re-derivation; the paper's entire confidence mechanism rests on it.
  • ad hoc to paper The test-set-derived trusted threshold is a fair and stable basis for comparing models.
    Introduced in Sec. 3.3.2; no validation set or fixed threshold is used, so the comparison is conditional on test labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Trustworthy Method for Multimodal Emotion Recognition." pith.science (2026). https://pith.science/paper/C64LNZ5O

@misc{pith2026250807625,
  author       = {Pith},
  title        = {Pith review of: A Trustworthy Method for Multimodal Emotion Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C64LNZ5O}},
  note         = {Machine review of arXiv:2508.07625}
}
read the original abstract

Existing emotion recognition methods mainly focus on enhancing performance by employing complex deep models, typically resulting in significantly higher model complexity. Although effective, it is also crucial to ensure the reliability of the final decision, especially for noisy, corrupted and out-of-distribution data. To this end, we propose a novel emotion recognition method called trusted emotion recognition (TER), which utilizes uncertainty estimation to calculate the confidence value of predictions. TER combines the results from multiple modalities based on their confidence values to output the trusted predictions. We also provide a new evaluation criterion to assess the reliability of predictions. Specifically, we incorporate trusted precision and trusted recall to determine the trusted threshold and formulate the trusted Acc. and trusted F1 score to evaluate the model's trusted performance. The proposed framework combines the confidence module that accordingly endows the model with reliability and robustness against possible noise or corruption. The extensive experimental results validate the effectiveness of our proposed model. The TER achieves state-of-the-art performance on the Music-video, achieving 82.40% Acc. In terms of trusted performance, TER outperforms other methods on the IEMOCAP and Music-video, achieving trusted F1 scores of 0.7511 and 0.9035, respectively.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 49 canonical work pages

  1. [12]

    Trusted multi-view classification with dynamic evidential fusion,

    Z. Han, C. Zhang, H. Fu, and J. T. Zhou, “Trusted multi-view classification with dynamic evidential fusion,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 2, pp. 2551–2566, 2022

    work page 2022

  2. [50]

    Multimodal graph learning with framelet-based stochastic configuration networks for emotion recognition in conversation,

    J. Shi, M. Li, Y . Chen, L. Cui, and L. Bai, “Multimodal graph learning with framelet-based stochastic configuration networks for emotion recognition in conversation,”Information Sciences, vol. 686, p. 121393, 2025

    work page 2025

  3. [52]

    Frameerc: Framelet transform based multimodal graph neural networks for emotion recognition in conversation,

    M. Li, J. Shi, L. Bai, C. Huang, Y . Jiang, K. Lu, S. Wang, and E. R. Hancock, “Frameerc: Framelet transform based multimodal graph neural networks for emotion recognition in conversation,”Pattern Recognition, vol. 161, p. 111340, 2025

    work page 2025

  4. [1]

    EEG emotion recognition using dynamical graph convolutional neural networks,

    T. Song, W. Zheng, P. Song, and Z. Cui, “EEG emotion recognition using dynamical graph convolutional neural networks,”IEEE Trans. Affective Comput., vol. 11, no. 3, pp. 532–541, 2018

    work page 2018

  5. [2]

    A comparison of the affectiva imotions facial expression analysis software with EMG for identifying facial expressions of emotion,

    L. Kulke, D. Feyerabend, and A. Schacht, “A comparison of the affectiva imotions facial expression analysis software with EMG for identifying facial expressions of emotion,”Front. Psychol., vol. 11, p. 329, 2020

    work page 2020

  6. [3]

    Utilizing deep learning towards multi- 14Big Data Mining and Analytics,xxxxxxx20xx, x(x): xxx-xxx modal bio-sensing and vision-based affective computing,

    T.-P. Jung, T. J. Sejnowskiet al., “Utilizing deep learning towards multi- 14Big Data Mining and Analytics,xxxxxxx20xx, x(x): xxx-xxx modal bio-sensing and vision-based affective computing,”IEEE Trans. Affective Comput., vol. 13, no. 1, pp. 96–107, 2019

    work page 2019

  7. [4]

    Emotion recognition using facial expressions,

    P. Tarnowski, M. Kołodziej, A. Majkowski, and R. J. Rak, “Emotion recognition using facial expressions,”Procedia Comput. Sci., vol. 108, pp. 1175–1184, 2017

    work page 2017

  8. [5]

    Ease: Robust facial expression recognition via emotion ambiguity-sensitive cooperative networks,

    L. Wang, G. Jia, N. Jiang, H. Wu, and J. Yang, “Ease: Robust facial expression recognition via emotion ambiguity-sensitive cooperative networks,” inProc. 30th ACM Int. Conf. Multimedia, 2022, pp. 218– 227

    work page 2022

Show all 52 references
  1. [6]

    Deep facial expression recognition: A survey,

    S. Li and W. Deng, “Deep facial expression recognition: A survey,” IEEE Trans. Affective Comput., vol. 13, no. 3, pp. 1195–1215, 2020

    2020

  2. [7]

    D 2S: Dynamic distribution supervision for multi-label facial expression recognition,

    L. Wang, X. Zhang, N. Jiang, H. Wu, and J. Yang, “D 2S: Dynamic distribution supervision for multi-label facial expression recognition,” inProc. IEEE Int. Conf. Multimedia Expo, 2022, pp. 1–6

    2022

  3. [8]

    Survey on emotional body gesture recognition,

    F. Noroozi, C. A. Corneanu, D. Kami ´nska, T. Sapi´nski, S. Escalera, and G. Anbarjafari, “Survey on emotional body gesture recognition,”IEEE Trans. Affective Comput., vol. 12, no. 2, pp. 505–523, 2018

    2018

  4. [9]

    Self supervised adversarial domain adaptation for cross-corpus and cross- language speech emotion recognition,

    S. Latif, R. Rana, S. Khalifa, R. Jurdak, and B. W. Schuller, “Self supervised adversarial domain adaptation for cross-corpus and cross- language speech emotion recognition,”IEEE Trans. Affective Comput., 2022

    2022

  5. [10]

    M2fnet: Multi-modal fusion network for emotion recognition in conversation,

    V . Chudasama, P. Kar, A. Gudmalwar, N. Shah, P. Wasnik, and N. Onoe, “M2fnet: Multi-modal fusion network for emotion recognition in conversation,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 4652–4661

    2022

  6. [11]

    Facial expression recognition with visual transformers and attentional selective fusion,

    F. Ma, B. Sun, and S. Li, “Facial expression recognition with visual transformers and attentional selective fusion,”IEEE Trans. Affective Comput., 2021

    2021

  7. [13]

    Facial expression recognition with identity and emotion joint learning,

    M. Li, H. Xu, X. Huang, Z. Song, X. Liu, and X. Li, “Facial expression recognition with identity and emotion joint learning,”IEEE Trans. Affective Comput., vol. 12, no. 2, pp. 544–550, 2018

    2018

  8. [14]

    Eeg- based emotion recognition for multi channel fast empirical mode decomposition using vgg-16,

    M. A. Asghar, M. J. Khan, Y . Amin, A. Akramet al., “Eeg- based emotion recognition for multi channel fast empirical mode decomposition using vgg-16,” inProc. Int. Conf. Eng. Emerging Technol., 2020, pp. 1–7

    2020

  9. [15]

    Quo vadis, action recognition? a new model and the kinetics dataset,

    J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 6299–6308

    2017

  10. [16]

    Audio-visual emotion recognition in video clips,

    F. Noroozi, M. Marjanovic, A. Njegus, S. Escalera, and G. Anbarjafari, “Audio-visual emotion recognition in video clips,”IEEE Trans. Affective Comput., vol. 10, no. 1, pp. 60–75, 2017

    2017

  11. [17]

    Attention driven fusion for multi-modal emotion recognition,

    D. Priyasad, T. Fernando, S. Denman, S. Sridharan, and C. Fookes, “Attention driven fusion for multi-modal emotion recognition,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process., 2020, pp. 3227–3231

    2020

  12. [18]

    Facial emotion recognition: State of the art performance on fer2013,

    Y . Khaireddin and Z. Chen, “Facial emotion recognition: State of the art performance on fer2013,”arXiv:2105.03588, 2021

    2021 arXiv

  13. [19]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inProc. IEEE/CVF Int. Conf. Comput. Vis., 2021, pp. 10 012–10 022

    2021

  14. [20]

    Two- level attention with two-stage multi-task learning for facial emotion recognition,

    W. Xiaohua, P. Muzi, P. Lijuan, H. Min, J. Chunhua, and R. Fuji, “Two- level attention with two-stage multi-task learning for facial emotion recognition,”J. Visual Commun. Image Represent., vol. 62, pp. 217– 225, 2019

    2019

  15. [21]

    Context-aware emotion recognition networks,

    J. Lee, S. Kim, S. Kim, J. Park, and K. Sohn, “Context-aware emotion recognition networks,” inProc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 10 143–10 152

    2019

  16. [22]

    An attention model for group-level emotion recognition,

    A. Gupta, D. Agrawal, H. Chauhan, J. Dolz, and M. Pedersoli, “An attention model for group-level emotion recognition,” inProc. 20th ACM Int. Conf. Multimodal Interact., 2018, pp. 611–615

    2018

  17. [23]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inProc. Int. Conf. Mach. Learn., 2017, pp. 1321–1330

    2017

  18. [24]

    Calibrating deep neural networks using focal loss,

    J. Mukhoti, V . Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania, “Calibrating deep neural networks using focal loss,” inProc. Int. Conf. Neural Inf. Process. Syst., 2020, pp. 15 288–15 299

    2020

  19. [25]

    Don’t just blame over- parametrization for over-confidence: Theoretical analysis of calibration in binary classification,

    Y . Bai, S. Mei, H. Wang, and C. Xiong, “Don’t just blame over- parametrization for over-confidence: Theoretical analysis of calibration in binary classification,” inProc. Int. Conf. Mach. Learn., 2021, pp. 566–576

    2021

  20. [26]

    Be confident! towards trustworthy graph neural networks via confidence calibration,

    X. Wang, H. Liu, C. Shi, and C. Yang, “Be confident! towards trustworthy graph neural networks via confidence calibration,” inProc. Int. Conf. Neural Inf. Process. Syst., 2021, pp. 23 768–23 779

    2021

  21. [27]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recog., 2022, pp. 16 000–16 009

    2022

  22. [28]

    Mage: Masked generative encoder to unify representation learning and image synthesis,

    T. Li, H. Chang, S. Mishra, H. Zhang, D. Katabi, and D. Krishnan, “Mage: Masked generative encoder to unify representation learning and image synthesis,” inProc. IEEE/CVF Con. Comput. Vis. Pattern Recog., 2023, pp. 2142–2152

    2023

  23. [29]

    Video swin transformer,

    Z. Liu, J. Ning, Y . Cao, Y . Wei, Z. Zhang, S. Lin, and H. Hu, “Video swin transformer,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recog., 2022, pp. 3202–3211

    2022

  24. [30]

    CNN architectures for large-scale audio classification,

    S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seyboldet al., “CNN architectures for large-scale audio classification,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process., 2017, pp. 131–135

    2017

  25. [31]

    The relationship between precision-recall and roc curves,

    J. Davis and M. Goadrich, “The relationship between precision-recall and roc curves,” inProc. Int. Conf. Mach. Learn., 2006, pp. 233–240

    2006

  26. [32]

    IEMOCAP: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,”Lang. Resour . Eval., vol. 42, pp. 335– 359, 2008

    2008

  27. [33]

    Deep learning-based late fusion of multimodal information for emotion classification of music video,

    Y . R. Pandeya and J. Lee, “Deep learning-based late fusion of multimodal information for emotion classification of music video,” Multimedia Tools Appl., vol. 80, pp. 2887–2905, 2021

    2021

  28. [34]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,” inProc. Int. Conf. Neural Inf. Process. Syst., 2019, pp. 1–12

    2019

  29. [35]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” inProc. 3rd Int. Conf. Learn. Represent., 2015, pp. 1–15

    2015

  30. [36]

    Context-dependent sentiment analysis in user-generated videos,

    S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, and L.- P. Morency, “Context-dependent sentiment analysis in user-generated videos,” inProc. 55th Annu. Meeting Assoc. Comput. Ling., 2017, pp. 873–883

    2017

  31. [37]

    DialogueGCN: A graph convolutional neural network for emotion recognition in conversation,

    D. Ghosal, N. Majumder, S. Poria, N. Chhaya, and A. Gelbukh, “DialogueGCN: A graph convolutional neural network for emotion recognition in conversation,” inProc. Conf. Empirical Methods Nat. Lang. Process., 2019, pp. 154–164

    2019

  32. [38]

    Directed acyclic graph network for conversational emotion recognition,

    W. Shen, S. Wu, Y . Yang, and X. Quan, “Directed acyclic graph network for conversational emotion recognition,” inProc. 59th Annu. Meeting Assoc. Comput. Ling., 2021, pp. 1551–1560

    2021

  33. [39]

    Multi-task learning for multi-modal emotion recognition and sentiment analysis,

    M. S. Akhtar, D. S. Chauhan, D. Ghosal, S. Poria, A. Ekbal, and P. Bhattacharyya, “Multi-task learning for multi-modal emotion recognition and sentiment analysis,” inProc. Conf. N. Am. Chapter Assoc. Comput. Ling., 2019, pp. 370–379

    2019

  34. [40]

    COGMEN: COntextualized GNN based multimodal emotion recognition,

    A. Joshi, A. Bhat, A. Jain, A. V . Singh, and A. Modi, “COGMEN: COntextualized GNN based multimodal emotion recognition,” inProc. Conf. N. Am. Chapter Assoc. Comput. Ling., 2022, pp. 4418–4164

    2022

  35. [41]

    MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation,

    J. Hu, Y . Liu, J. Zhao, and Q. Jin, “MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation,” pp. 5666–5675, 2021

    2021

  36. [42]

    Opensmile: the munich versatile and fast open-source audio feature extractor,

    F. Eyben, M. W ¨ollmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” inProc. 18th ACM Int. Conf. Multimedia, 2010, pp. 1459–1462

    2010

  37. [43]

    Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends,

    B. W. Schuller, “Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends,”Commun. ACM, vol. 61, no. 5, pp. 90–99, 2018

    2018

  38. [44]

    Speech emotion recognition based on attention weight correction using word-level confidence measure

    J. Santoso, T. Yamada, S. Makino, K. Ishizuka, and T. Hiramura, “Speech emotion recognition based on attention weight correction using word-level confidence measure.” inInterspeech, 2021, pp. 1947–1951

    2021

  39. [45]

    Confidence measures for speech Junxiao Xue et al.:A Trustworthy Method for Multimodal Emotion Recognition ...15 emotion recognition: A start,

    J. Deng, W. Han, and B. Schuller, “Confidence measures for speech Junxiao Xue et al.:A Trustworthy Method for Multimodal Emotion Recognition ...15 emotion recognition: A start,” inProc. Speech Commun. 10. ITG Symp., 2012, pp. 1–4

    2012

  40. [46]

    Confidence measures in speech emotion recognition based on semi-supervised learning,

    J. Deng and B. Schuller, “Confidence measures in speech emotion recognition based on semi-supervised learning,” inProc. Annu. Conf. Int. Speech Commun. Assoc., 2012

    2012

  41. [47]

    Confidence estimation for speech emotion recognition based on the relationship between emotion categories and primitives,

    Y . Li, C. Papayiannis, V . Rozgic, E. Shriberg, and C. Wang, “Confidence estimation for speech emotion recognition based on the relationship between emotion categories and primitives,” inProc. Int. Conf. Acoust. Speech Signal Process.IEEE, 2022, pp. 7352–7356

    2022

  42. [48]

    Meld: A multimodal multi-party dataset for emotion recognition in conversations,

    S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “Meld: A multimodal multi-party dataset for emotion recognition in conversations,”arXiv preprint arXiv:1810.02508, 2018

    2018 arXiv

  43. [49]

    Affective video content analysis: Decade review and new perspectives,

    J. Xue, J. Wang, X. Liu, Q. Zhang, and X. Wu, “Affective video content analysis: Decade review and new perspectives,”Big Data Mining and Analytics, vol. 8, no. 1, pp. 118–144, 2024

    2024

  44. [51]

    Gm2rc: Graph-based multitask modality refinement and complement for multimodal sentiment analysis,

    J. Shi, Y . Chen, S. Zhou, and M. Li, “Gm2rc: Graph-based multitask modality refinement and complement for multimodal sentiment analysis,” in2024 7th International Symposium on Autonomous Systems (ISAS). IEEE, 2024, pp. 1–6

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.