pith. machine review for the scientific record. sign in

arxiv: 2605.08847 · v1 · submitted 2026-05-09 · 💻 cs.CL

Recognition: no theorem link

EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding

Derek F. Wong, Jingxi Liang, Pengze Guo, Qifeng Wang, Zhiwen Xie

Authors on Pith no claims yet

Pith reviewed 2026-05-12 02:24 UTC · model grok-4.3

classification 💻 cs.CL
keywords emotional understandingmultimodal benchmarkstreaming monologuefine-grained labelingemotion recognitionempathy modelsbilingual datasetMLLM fine-tuning
0
0 comments X

The pith

EmoS benchmark supplies continuous emotional labels through dual-layer annotations on filtered static slices and streaming monologues.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing emotional datasets often suffer from low ecological validity, unclear signals, and unreliable fine-grained labels that hinder training of empathetic AI systems. The paper introduces EmoS as a bilingual collection that pairs strictly filtered static slices with dynamic streaming monologues to overcome these gaps. Its dual-layer human annotation pipeline creates trusted ground truth that tracks how emotions evolve over time. Fine-tuning multimodal large language models on EmoS produces clear performance gains compared with zero-shot use, which supports building more effective emotion recognition and empathy tools for real-world demands.

Core claim

EmoS resolves limitations of ecological validity and noise in prior datasets by combining strictly filtered static slices with a dynamic Streaming Monologue subset in bilingual form. A rigorous dual-layer human annotation pipeline supplies trusted ground truth that captures continuous emotional evolution. Fine-tuning multimodal large language models on EmoS yields significant gains over zero-shot baselines and lays the foundation for training and evaluating future emotion recognition models and empathy models.

What carries the argument

Dual-layer human annotation pipeline applied to the Streaming Monologue subset and filtered static slices, which generates reliable continuous emotional labels.

If this is right

  • Multimodal large language models fine-tuned on EmoS achieve significant gains in fine-grained emotional understanding over zero-shot baselines.
  • The benchmark provides a foundation for training and evaluating future emotion recognition models.
  • Public release of the dataset and code supports development of empathy models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The annotation approach could extend to live video streams to track emotional shifts during real-time interactions.
  • Improved emotional tracking from this data might enhance AI support systems in high-stress settings such as counseling or customer service.
  • Bilingual coverage opens a path to test whether models trained this way generalize emotional patterns across languages.

Load-bearing premise

The dual-layer human annotation pipeline produces reliable low-noise fine-grained continuous emotional labels that are ecologically valid and superior to existing datasets.

What would settle it

An experiment in which multimodal models fine-tuned on EmoS show no performance advantage over zero-shot models on an independent emotional understanding test would indicate the benchmark does not deliver the claimed improvements.

Figures

Figures reproduced from arXiv: 2605.08847 by Derek F. Wong, Jingxi Liang, Pengze Guo, Qifeng Wang, Zhiwen Xie.

Figure 1
Figure 1. Figure 1: Illustration of limitations in current short [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The basic information and processing procedures of our dataset [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The performance of four annotator styles [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Precision vs. Recall per emotion label. The [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
read the original abstract

In the context of today's high-pressure, aging society, the demand for large-scale emotional models capable of providing empathetic support is more critical than ever. However, existing benchmarks fail to simultaneously achieve ecological validity, signal clarity, and reliable fine-grained labeling. We introduce EmoS, a high-fidelity bilingual benchmark designed to resolve the limitations of ecological validity and noise in existing datasets by combining strictly filtered static slices with a dynamic Streaming Monologue subset. Supported by a rigorous dual-layer human annotation pipeline, EmoS provides trusted ground truth that captures continuous emotional evolution. Empirical results show that fine-tuning MLLMs (multimodal large language models) on EmoS yields significant gains over zero-shot baselines, laying the foundation for the training and evaluation of future emotion recognition models and empathy models. The dataset and code are publicly available at https://github.com/NLP2CT/EmoS.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces EmoS, a high-fidelity bilingual multimodal benchmark for fine-grained streaming emotional understanding. It combines strictly filtered static slices with a dynamic streaming monologue subset, supported by a dual-layer human annotation pipeline to produce continuous emotional labels. The authors claim that fine-tuning MLLMs on EmoS yields significant gains over zero-shot baselines and position the dataset as a foundation for future emotion recognition and empathy models, with public release of data and code.

Significance. If the annotation pipeline delivers reliable low-noise continuous labels and the reported fine-tuning gains prove robust and attributable to dataset quality, EmoS could fill an important gap by improving ecological validity and signal clarity over existing emotion benchmarks. The streaming component and public availability would support reproducible progress in multimodal empathetic AI.

major comments (2)
  1. [Annotation Pipeline (Section 3)] The central claim of 'trusted ground truth' and superiority to existing datasets rests on the dual-layer human annotation pipeline. The manuscript provides no quantitative measures of reliability (e.g., inter-annotator agreement, correlation coefficients, or noise estimates for the continuous labels), leaving the weakest assumption unaddressed and undermining the high-fidelity assertion.
  2. [Empirical Results (Section 4)] The abstract and results claim 'significant gains' from fine-tuning MLLMs on EmoS, yet supply no specific metrics, statistical tests, baseline details, ablation studies, or error analysis. Without these, it is impossible to verify whether improvements stem from dataset quality rather than experimental artifacts or post-hoc filtering.
minor comments (2)
  1. [Abstract] The abstract would be strengthened by including one or two key quantitative results (e.g., accuracy deltas or correlation scores) to substantiate the 'significant gains' claim.
  2. [Throughout] Notation for continuous emotional dimensions (e.g., valence/arousal scales) should be defined explicitly on first use and kept consistent across figures and tables.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight important areas for strengthening our claims regarding annotation reliability and empirical validation. We address each major comment below and will incorporate the necessary revisions to improve the manuscript.

read point-by-point responses

  1. Referee: [Annotation Pipeline (Section 3)] The central claim of 'trusted ground truth' and superiority to existing datasets rests on the dual-layer human annotation pipeline. The manuscript provides no quantitative measures of reliability (e.g., inter-annotator agreement, correlation coefficients, or noise estimates for the continuous labels), leaving the weakest assumption unaddressed and undermining the high-fidelity assertion.

    Authors: We agree that quantitative reliability metrics are required to substantiate the high-fidelity claims. The manuscript describes the dual-layer pipeline (independent annotation followed by reconciliation) but does not report agreement statistics. In the revised version, we will add inter-annotator agreement measures such as Krippendorff's alpha for continuous labels, Pearson/Spearman correlations between annotators, and noise estimates derived from discrepancy rates resolved in the second layer. These additions will provide empirical support for label quality and enable direct comparison with prior emotion datasets. revision: yes

  2. Referee: [Empirical Results (Section 4)] The abstract and results claim 'significant gains' from fine-tuning MLLMs on EmoS, yet supply no specific metrics, statistical tests, baseline details, ablation studies, or error analysis. Without these, it is impossible to verify whether improvements stem from dataset quality rather than experimental artifacts or post-hoc filtering.

    Authors: We acknowledge that the current results section lacks the detailed quantitative evidence needed to fully support the claims. In the revision, we will expand Section 4 with specific metrics (e.g., MAE or correlation for continuous emotion prediction, accuracy/F1 for discrete categories), statistical significance tests (e.g., paired t-tests or Wilcoxon signed-rank with p-values), full baseline details (zero-shot MLLMs plus comparisons to other datasets), ablation studies isolating the streaming monologue subset and static slices, and error analysis highlighting failure modes. This will allow readers to assess whether gains are attributable to EmoS quality. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper is a benchmark construction and empirical evaluation work. It introduces EmoS via filtered static slices plus streaming monologues, supported by a dual-layer human annotation pipeline for continuous labels, then reports fine-tuning gains on MLLMs versus zero-shot baselines. No equations, parameter fits, or derivations are present that could reduce outputs to inputs by construction. No self-citation load-bearing steps, uniqueness theorems, or ansatzes appear in the provided abstract or structure. The central claims rest on dataset design and standard comparative experiments, which are self-contained and externally falsifiable without circular reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

This is an empirical benchmark paper with no mathematical derivations. The central claims rest on the assumption that the new annotation process yields higher-quality labels than prior work.

axioms (1)
  • domain assumption Dual-layer human annotation can produce reliable continuous fine-grained emotional labels with low noise and high ecological validity.
    Invoked to justify the ground truth quality of EmoS over existing datasets.

pith-pipeline@v0.9.0 · 5463 in / 1156 out tokens · 37924 ms · 2026-05-12T02:24:42.903954+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 64 canonical work pages · 4 internal anchors

  1. [1]

    Proceedings of the 28th

    DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild , author =. Proceedings of the 28th. 2020 , organization =

    work page 2020

  2. [2]

    Proceedings of the 30th

    MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild , author =. Proceedings of the 30th. 2022 , organization =

    work page 2022

  3. [3]

    arXiv , eprint =

    Yemin Shi and Yu Shu and Siwei Dong and Guangyi Liu and Jaward Sesay and Jingwen Li and Zhiting Hu , title =. arXiv , eprint =

    work page

  4. [4]

    2008 , publisher =

    Busso, Carlos and Bulut, Murtaza and Lee, Chi-Chun and Kazemzadeh, Abe and Mower, Emily and Kim, Samuel and Chang, Jeannette N and Lee, Sungbok and Narayanan, Shrikanth S , journal =. 2008 , publisher =

    work page 2008

  5. [5]

    Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) , pages =

    The Distress Analysis Interview Corpus of human and computer interviews , author =. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) , pages =. 2014 , address =

    work page 2014

  6. [6]

    2019 , doi =

    Poria, Soujanya and Hazarika, Devamanyu and Majumder, Navonil and Naik, Gautam and Cambria, Erik and Mihalcea, Rada , booktitle =. 2019 , doi =

    work page 2019

  7. [7]

    2019 , doi =

    Ghosal, Deepanway and Majumder, Navonil and Poria, Soujanya and Chhaya, Niyati and Gelbukh, Alexander , booktitle =. 2019 , doi =

    work page 2019

  8. [8]

    IEEE Intelligent Systems , volume =

    Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages , author =. IEEE Intelligent Systems , volume =. 2016 , doi =

    work page 2016

  9. [9]

    Multimodal Language Analysis in the Wild:

    Zadeh, AmirAli Bagher and Liang, Paul Pu and Poria, Soujanya and Cambria, Erik and Morency, Louis-Philippe , booktitle =. Multimodal Language Analysis in the Wild:. 2018 , doi =

    work page 2018

  10. [10]

    2020 , doi =

    Yu, Wenmeng and Xu, Hua and Meng, Fanyang and Zhu, Yilin and Ma, Yixiao and Wu, Jiele and Zou, Jiyun and Yang, Kaicheng , booktitle =. 2020 , doi =

    work page 2020

  11. [11]

    Findings of EMNLP , pages =

    Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis , author =. Findings of EMNLP , pages =. 2022 , doi =

    work page 2022

  12. [12]

    Kollias, Dimitrios and Zafeiriou, Stefanos , journal =

    work page

  13. [15]

    Make Acoustic and Visual Cues Matter:

    Liu, Yihe and Yuan, Ziqi and Mao, Huisheng and Liang, Zhiyun and Yang, Wanqiuyue and Qiu, Yuanzhe and Cheng, Tie and Li, Xiaoteng and Xu, Hua and Gao, Kai , booktitle =. Make Acoustic and Visual Cues Matter:. 2022 , doi =

    work page 2022

  14. [17]

    Cognition & Emotion , volume =

    An Argument for Basic Emotions , author =. Cognition & Emotion , volume =. 1992 , doi =

    work page 1992

  15. [18]

    Journal of the Royal Statistical Society

    Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm , author =. Journal of the Royal Statistical Society. Series C (Applied Statistics) , volume =. 1979 , publisher =

    work page 1979

  16. [19]

    McKeown, Gary and Valstar, Michel and Cowie, Roddy and Pantic, Maja and Schr. The. IEEE Transactions on Affective Computing , volume =. 2012 , publisher =

    work page 2012

  17. [20]

    Introducing the

    Ringeval, Fabien and Sonderegger, Andreas and Sauer, J. Introducing the. 2013 10th. 2013 , address =

    work page 2013

  18. [21]

    , journal =

    Mollahosseini, Amir and Hasani, Behzad and Mahoor, Mohammad H. , journal =. 2019 , publisher =

    work page 2019

  19. [22]

    Barros, Pablo and Churamani, Nikhil and Lakomkin, Egor and Sutherland, Andrew and Wermter, Stefan , booktitle =. The. 2018 , publisher =

    work page 2018

  20. [23]

    2016 , publisher =

    The Emotional Arcs of Stories are Dominated by Six Basic Shapes , author =. 2016 , publisher =

    work page 2016

  21. [25]

    2023 , eprint =

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions , author =. 2023 , eprint =

    work page 2023

  22. [26]

    2023 , eprint =

    Gemini: A Family of Highly Capable Multimodal Models , author =. 2023 , eprint =

    work page 2023

  23. [27]

    2025 , eprint =

    Qwen2.5-Omni Technical Report , author =. 2025 , eprint =

    work page 2025

  24. [28]

    2025 , eprint =

    Qwen3-Omni Technical Report , author =. 2025 , eprint =

    work page 2025

  25. [29]

    Journal of Machine Learning Research , volume =

    Learning from Crowds , author =. Journal of Machine Learning Research , volume =. 2010 , url =

    work page 2010

  26. [30]

    Advances in Neural Information Processing Systems , year =

    Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , author =. Advances in Neural Information Processing Systems , year =

    work page

  27. [31]

    Psychological Bulletin , volume =

    Measuring Nominal Scale Agreement among Many Raters , author =. Psychological Bulletin , volume =. 1971 , doi =

    work page 1971

  28. [32]

    Dhall, Abhinav and Goecke, Roland and Joshi, Jyoti and Wagner, Michael and Gedeon, Tom , booktitle =. The. 2013 , doi =

    work page 2013

  29. [33]

    Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov

    Multimodal Transformer for Unaligned Multimodal Language Sequences , author =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year =. doi:10.18653/v1/P19-1656 , url =

    work page doi:10.18653/v1/p19-1656

  30. [35]

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions , author =. 2024 , doi =. 2311.05232 , archivePrefix =

    work page internal anchor Pith review arXiv 2024

  31. [36]

    Barros, Pablo and Churamani, Nikhil and Lakomkin, Egor and Sutherland, Andrew and Wermter, Stefan , booktitle =. The. 2018 , publisher =. doi:10.1109/IJCNN.2018.8489099 , url =

    work page doi:10.1109/ijcnn.2018.8489099 2018

  32. [37]

    EPJ Data Science , year =

    The emotional arcs of stories are dominated by six basic shapes , author =. EPJ Data Science , year =. doi:10.1140/epjds/s13688-016-0093-1 , url =

    work page doi:10.1140/epjds/s13688-016-0093-1

  33. [38]

    Journal of Machine Learning Research , volume =

    Learning From Crowds , author =. Journal of Machine Learning Research , volume =. 2010 , url =

    work page 2010

  34. [39]

    2011 , url =

    Computing Krippendorff's Alpha-Reliability , author =. 2011 , url =

    work page 2011

  35. [40]

    2025 , url =

    Gemini 3 Pro Model Card , author =. 2025 , url =

    work page 2025

  36. [42]

    Ana Aguilera, Diego Mellado, and Felipe Rojas. 2023. https://doi.org/10.3390/s23115184 An assessment of in-the-wild datasets for multimodal emotion recognition . Sensors, 23(11):5184

    work page doi:10.3390/s23115184 2023

  37. [43]

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. https://doi.org/10.1007/s10579-008-9076-6 IEMOCAP : Interactive emotional dyadic motion capture database . Language Resources and Evaluation, 42(4):335--359

    work page doi:10.1007/s10579-008-9076-6 2008

  38. [44]

    Hauptmann

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander G. Hauptmann. 2024. https://arxiv.org/abs/2406.11161 Emotion-LLaMA : Multimodal emotion recognition and reasoning with instruction tuning . In Advances in Neural Information Processing Systems

    work page arXiv 2024

  39. [45]

    A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20--28. Original Dawid--Skene model for aggregating multiple annotator labels via EM

    work page 1979

  40. [46]

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. https://doi.org/10.18653/v1/2020.acl-main.372 Goemotions: A dataset of fine-grained emotions . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040--4054, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2020.acl-main.372 2020

  41. [47]

    Abhinav Dhall, Roland Goecke, Jyoti Joshi, Michael Wagner, and Tom Gedeon. 2013. https://doi.org/10.1145/2522848.2531739 The E motion R ecognition in the W ild challenge 2013 . In Proceedings of the 15th ACM International Conference on Multimodal Interaction, pages 509--515

    work page doi:10.1145/2522848.2531739 2013

  42. [48]

    Paul Ekman. 1992. https://doi.org/10.1080/02699939208411068 An argument for basic emotions . Cognition & Emotion, 6(3/4):169--200. Classic argument for discrete basic emotions (e.g., anger, joy, fear, sadness, disgust, surprise)

    work page doi:10.1080/02699939208411068 1992

  43. [49]

    Joseph L. Fleiss. 1971. https://doi.org/10.1037/h0031619 Measuring nominal scale agreement among many raters . Psychological Bulletin, 76(5):378--382

    work page doi:10.1037/h0031619 1971

  44. [50]

    Gemini Team . 2023. https://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal models

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  45. [51]

    Google . 2025. https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf Gemini 3 pro model card . Model card listed on Google's official Model Cards site

    work page 2025

  46. [52]

    Jonathan Gratch, Ron Artstein, Gale Lucas, Giota Stratou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, David Traum, Skip Rizzo, and Louis-Philippe Morency. 2014. https://aclanthology.org/L14-1421/ The distress analysis interview corpus of human and computer interviews . In Proceedings of the Ninth International...

    work page 2014

  47. [53]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2024. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    work page doi:10.1145/3703155 2024

  48. [54]

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.123 Towards mitigating hallucination in large language models via self-reflection . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827--1843, Singapore. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.findings-emnlp.123 2023

  49. [55]

    Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. 2020. https://doi.org/10.1145/3394171.3413620 Dfew: A large-scale database for recognizing dynamic facial expressions in the wild . In Proceedings of the 28th ACM International Conference on Multimedia , pages 2881--2889, New York, NY, USA. ACM

    work page doi:10.1145/3394171.3413620 2020

  50. [56]

    Klaus Krippendorff. 2011. https://repository.upenn.edu/asc_papers/43/ Computing krippendorff's alpha-reliability . Technical report, University of Pennsylvania

    work page 2011

  51. [57]

    Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, Jiangyan Yi, and Jianhua Tao. 2025. https://arxiv.org/abs/2501.16566 Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models . In Proceedings of the 42nd International Conference on Mach...

    work page arXiv 2025

  52. [58]

    Yihe Liu, Ziqi Yuan, Huisheng Mao, Zhiyun Liang, Wanqiuyue Yang, Yuanzhe Qiu, Tie Cheng, Xiaoteng Li, Hua Xu, and Kai Gao. 2022 a . https://doi.org/10.1145/3536221.3556630 Make acoustic and visual cues matter: CH-SIMS v2.0 dataset and AV-Mixup consistent module . In Proceedings of the 24th International Conference on Multimodal Interaction

    work page doi:10.1145/3536221.3556630 2022

  53. [59]

    Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. 2022 b . https://doi.org/10.1145/3503161.3548190 Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild . In Proceedings of the 30th ACM International Conference on Multimedia , New York, NY, USA. ACM

    work page doi:10.1145/3503161.3548190 2022

  54. [60]

    Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Marc Schr \"o der. 2012. https://doi.org/10.1109/T-AFFC.2011.20 The SEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent . IEEE Transactions on Affective Computing, 3(1):5--17

    work page doi:10.1109/t-affc.2011.20 2012

  55. [61]

    Amir Mollahosseini, Behzad Hasani, and Mohammad H. Mahoor. 2019. https://doi.org/10.1109/TAFFC.2017.2740923 AffectNet : A database for facial expression, valence, and arousal computing in the wild . IEEE Transactions on Affective Computing, 10(1):18--31

    work page doi:10.1109/taffc.2017.2740923 2019

  56. [62]

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. https://doi.org/10.18653/v1/P19-1050 MELD : A multimodal multi-party dataset for emotion recognition in conversations . In Proceedings of ACL, pages 527--536

    work page doi:10.18653/v1/p19-1050 2019

  57. [63]

    Raykar, Shipeng Yu, Linda H

    Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. https://jmlr.org/papers/v11/raykar10a.html Learning from crowds . Journal of Machine Learning Research, 11:1297--1322

    work page 2010

  58. [64]

    Sauer, and Denis Lalanne

    Fabien Ringeval, Andreas Sonderegger, J \"u rgen S. Sauer, and Denis Lalanne. 2013. https://doi.org/10.1109/FG.2013.6553805 Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions . In 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition ( FG ) , pages 1--8, Shanghai, China. IEEE

    work page doi:10.1109/fg.2013.6553805 2013

  59. [65]

    Yemin Shi, Yu Shu, Siwei Dong, Guangyi Liu, Jaward Sesay, Jingwen Li, and Zhiting Hu. 2025. https://arxiv.org/abs/2505.02707 Voila: Voice-language foundation models for real-time autonomous interaction and voice role-play . arXiv

    work page arXiv 2025

  60. [66]

    Movellan, and Paul L

    Jacob Whitehill, Ting-Fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. https://papers.nips.cc/paper_files/paper/2009/hash/0c6415c27e9e6c31f9f6c0f6a0c3d1b0-Abstract.html Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In Advances in Neural Information Processing Systems

    work page 2009

  61. [67]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025 a . https://arxiv.org/abs/2503.20215 Qwen2.5-omni technical report . Preprint, arXiv:2503.20215

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  62. [68]

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025 b . https://arxiv.org/abs/2509.17765 Qwen3-omni technical report . Preprint, arXiv:2509.17765

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  63. [69]

    Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. https://doi.org/10.1109/MIS.2016.94 Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages . IEEE Intelligent Systems, 31(6):82--88

    work page doi:10.1109/mis.2016.94 2016

  64. [70]

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. https://doi.org/10.18653/v1/P18-1208 Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph . In Proceedings of ACL, pages 2236--2246

    work page doi:10.18653/v1/p18-1208 2018