Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EmotionTalk supplies a 23.6-hour, actor-recorded Chinese dialogue corpus in which every utterance carries emotion, sentiment, and speech-caption labels.

desk verdict The dataset is a real contribution if it ships, but the annotation formula in §3.2 is wrong as written and the train/test split never mentions speaker separation, so the benchmark numbers are uninterpretable until both are fixed. read the letter →

arxiv 2505.23018 v3 pith:BULSTZZ7 submitted 2025-05-29 cs.MM

Haoqin Sun , Xuechen Wang , Jinghua Zhao , Shiwan Zhao , Jiaming Zhou , Hui Wang , Jiabei He , Aobo Kong
show 4 more authors
Xi Yang Yequan Wang Yonghua Lin Yong Qin
This is my paper · ORCID
classification cs.MM
keywords multimodalemotionrecognitionChinesedatasetdyadicdialogueannotationspeechcaptioningsentimentanalysisaffectivecomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EmotionTalk is a new Chinese-language resource for studying how emotion travels through speech, video, and text together. The paper's claim is that it is the first large-scale, recorded, interactive Chinese multimodal emotion dataset: 19 professional actors performed 744 dyadic dialogues, yielding 23.6 hours of audio and 19,250 utterances, each annotated with a seven-way emotion category, a five-level sentiment label aggregated to a continuous -2 to 2 score, and four speech-caption dimensions (speaker, speaking style, emotion, overall). It matters because existing Mandarin emotion datasets are mostly scraped from television or lack dialogue context, while recorded interactive corpora are almost all English, so EmotionTalk aims to close that gap for cross-cultural affective computing. The paper supports the claim by benchmarking unimodal models, multimodal fusion models, conversational models, and emotion captioners on the new data.

What carries the argument

The central object is a dyadic-dialogue corpus recorded by professional actors under a structured annotation pipeline. The load-bearing mechanism is the annotation design: per-modality annotators see only one modality at a time (text, then audio, then silent video, then full audiovisual), each vote carries a confidence score, and the final category is the argmax of confidence-weighted votes, with expert review for low-confidence cases. The continuous sentiment label is the average of five annotators' five-level scores, and each caption is expanded into five stylistically varied versions using a large language model. This machinery is what allows the dataset to ground both recognition-style benchmarks and generation-style captioning benchmarks.

What would settle it

Inspect the released file names and the train/validation/test partition for actor IDs: if any actor appears in more than one partition, rerun the best unimodal and multimodal configurations on an actor-exclusive split. A large accuracy drop would show that the reported numbers are inflated by speaker familiarity; little change would support the claimed generalization.

Watch

Extended reading notes

Core claim

The paper's core discovery is the dataset itself together with the annotation protocol that lets the same utterances serve many tasks at once. Each utterance exists in three synchronized modalities, and each modality was annotated in isolation by five annotators before a multimodal label was aggregated with confidence weighting, so the data can support unimodal studies, fusion studies, and missing-modality simulations. The paper reports inter-annotator agreement values it reads as good, and it shows that utterance-level fusion methods generally outperform frame-level ones on this corpus, while the captioning experiments demonstrate that the caption annotations are usable for automated generation. The resulting resource is positioned as the first interactive Chinese multimodal emotion dataset with both fine-grained labels and emotional speaking-style captions.

Load-bearing premise

The reported benchmark numbers only measure what the paper claims they measure if no actor appears in more than one of the train, validation, and test partitions, and the paper never states that this is the case.

Editorial extensions

If this is right

  • Each modality has its own ground-truth labels, so unimodal emotion recognition can be trained and evaluated separately, and missing-modality robustness can be tested by withholding one or two tracks at test time.
  • Multimodal fusion methods can be compared under a common Chinese benchmark; the paper's own results indicate that utterance-level fusion, notably LMF and attention-based fusion, outperforms frame-level fusion on this data.
  • The caption tracks give speech emotion captioning an open, reproducible Chinese benchmark with speaker, style, emotion, and overall descriptions, addressing the lack of published emotion-caption datasets.
  • Because the data are organized into 744 dialogues with speaker and timing metadata, conversational emotion recognition and emotion prediction in conversation can be studied in a language other than English.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the per-modality labels allow quantifying which modality is most reliable for each emotion category, so researchers could use EmotionTalk to decide when a single microphone or camera suffices in deployment.
  • Beyond the paper: the five LLM-expanded caption variants per utterance make the dataset usable for preference-based caption generation, where systems are trained or evaluated by ranking outputs against the human-preferred version.
  • Beyond the paper: the speaker and speaking-style captions are shaped like the prompt texts used in expressive speech synthesis, so EmotionTalk could plausibly be repurposed for style-controlled Mandarin text-to-speech, a direction the paper itself does not pursue.
  • Beyond the paper: the heavy skew toward negative sentiment in the continuous labels suggests the corpus could double as a stress or conflict detection benchmark, though the paper does not frame it that way.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces EmotionTalk, a Mandarin dyadic conversational dataset recorded by 19 professional actors, comprising 19,250 utterances / 23.6 hours of audio along with video and text. Each utterance is annotated with 7-class discrete emotion labels, 5-level sentiment intensity labels, and four dimensions of speech captions (speaker, style, emotion, overall), with five annotators per item. The authors report Fleiss kappa values between 0.66 and 0.79 across modalities, describe a stratified train/validation/test split in Appendix A.3, and benchmark a wide range of unimodal encoders, multimodal fusion methods, and captioning decoders. The paper's central claim is that EmotionTalk is the first large-scale, recorded, interactive Chinese multimodal emotion dataset with fine-grained and caption-level annotations, and that the benchmark experiments demonstrate its usefulness.

Significance. If the dataset and its labels are as described, EmotionTalk fills a real gap: it is substantially larger than CH-SIMS (19,250 vs. 2,281 utterances), covers dialogue context, and is, to my knowledge, the first Mandarin multimodal resource with structured speech-caption annotations, making it valuable for emotion recognition, missing-modality research, and speech captioning. The paper is also commendable for including a datasheet, an ethics statement, a planned public release, and extensive comparisons across many pre-trained encoders. The core contribution is empirical, not theoretical, so there is no circularity concern in the technical sense. However, the value of the resource depends on the correctness of the label aggregation pipeline and on whether the benchmark results estimate generalization to unseen speakers; both points need to be resolved before the contribution can be accepted.

major comments (3)
  1. [§3.2, Eq. (1)] The weighted-confidence aggregation formula x_k = (1/N_k) Σ I(y_i=k)·c_i is not a valid aggregation rule as written. If four annotators choose 'neutral' with confidence 0.1 and one chooses 'happy' with confidence 0.9, the formula gives x_neutral = 0.1 and x_happy = 0.9, so argmax selects 'happy' even though 4/5 annotators chose neutral; it also divides by zero for any category with N_k = 0. Because Table 6 identifies emotion_result as the final aggregated label computed by this method, this pathology could affect the label distribution in Table 7 and every accuracy reported in Tables 2–4 and 12–13. Please either replace the denominator with the total number of annotators (or use a sum rather than an average), or specify and quantify the expert-review override, including how 'low confidence' and 'inconsistent' are defined and what fraction of utterances was overridden.
  2. [Appendix A.3] The split is described as stratified so that category distributions are similar across train, validation, and test, but the paper does not state that utterances of the same actor are confined to a single partition. With only 19 actors, an utterance-level random split can place the same speaker in both training and test sets, allowing models to exploit speaker identity and inflating the reported accuracies. Please state explicitly whether the split is speaker-disjoint, and if it is not, re-run the benchmarks with a speaker-independent split or provide per-speaker results.
  3. [§3.2 / §4.1] The annotation procedure says annotators label each modality separately in a fixed order, but the paper does not specify whether the final emotion_result label and the Table 7 distribution refer to the text, audio, video, or multimodal-integration annotation, or to some combination. This ambiguity makes it difficult to interpret Table 2's distinction between 'unimodal labels' and 'multimodal labels' and should be clarified, because the released dataset is expected to include a single emotion_result per utterance yet the experiments rely on modality-specific annotations.
minor comments (6)
  1. [§2.2.1] The sentence describing Fan et al. is duplicated: 'Design an attention aggregation network and a auxiliary uni-modal classifier to align shared emotional information across modalities' appears twice in the same paragraph.
  2. [§4.2 vs. Table 3] The text states that 'LMF achieves the highest score in the Multimodal(Four) setting (83.04%) and also yields the best average performance (75.53%)', but Table 3 reports LMF at 81.31% and 75.21%, with the highest Four-class result being MulT's 82.17%; these numbers should be reconciled.
  3. [§4.3] The claim that 'Qwen-2 outperforms other models across all four tasks' is too strong given that GPT-2 obtains the highest ROUGEL (0.430) and CLAPScore (0.899) on the Speaker task in Table 5; please qualify the claim.
  4. [Appendix B.2 vs. Appendix D] There is an optimizer inconsistency: Appendix B.2 says all models are trained with AdamW, whereas Appendix D says all models are trained with Adam; please use one consistent description.
  5. [Tables 2 and 3] Several formatting typos need fixing: Table 3 contains 'Uttrance-level' and runs-together values such as '80.3869.3174.85', and Table 2 has similar missing spaces such as '73.6966.15'.
  6. [Appendix F vs. Abstract and §G.1] The release status is inconsistent: Appendix F says 'The EmotionTalk dataset will be released soon', while Section G.1 says it 'has been publicly released' and the abstract promises open-source availability; please state the actual current availability and provide a working link or a concrete release date.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EmotionTalk is a measurement resource, and its benchmark experiments are standard dataset validation rather than a derivation that reduces to its own inputs.

full rationale

The paper constructs a dataset and reports benchmark accuracies on it; there is no claimed derivation of a phenomenon from fitted parameters. The annotation aggregation formula in Section 3.2, x_k = (1/N_k) sum I(y_i=k) c_i with y = argmax_k x_k, defines how individual annotator labels and confidences are combined into a final label; it is a labeling convention, not a prediction derived from the dataset, and no experimental result is claimed to follow from it by construction. The self-citations in Sections 2 and 4 (e.g., prior work by the authors on speech emotion recognition and multimodal methods) are contextual or methodological and are not load-bearing for the central dataset claim. The use of EmotionTalk to evaluate models on EmotionTalk is the normal and expected validation procedure for a dataset paper and does not make the results circular because the models are independently pretrained and the reported metrics quantify model performance rather than restating the annotation inputs. Appendix A.3 describes a stratified split, and Appendix H candidly notes limitations regarding dataset scale, the small number of actors, and uneven geographical distribution; these are scale and generalization concerns, not circularity. No step in the paper reduces, by its own equations or by a self-citation chain, to its inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central contribution is a dataset, so it relies on domain assumptions about annotation validity, acted-data naturalness, and split fairness rather than free parameters or invented entities.

assumptions (5)
  • domain assumption The 7 discrete emotion categories (happy, surprise, sad, disgust, anger, fear, neutral) and the 5-point sentiment scale are appropriate for annotating Chinese emotional expressions.
    The annotation scheme in Section 3.2 is based on basic emotion theory, but the paper provides no evidence that these categories are culturally appropriate for Chinese speech.
  • domain assumption Dialogues performed by 19 professional actors following situational scripts capture natural, authentic emotional expressions representative of real-world Chinese conversations.
    Section 3.1 states actors are encouraged to express authentic emotions, but acted data may not match spontaneous interaction; no validation against real conversations is provided.
  • domain assumption The train/validation/test split avoids speaker leakage, i.e., utterances from the same actor do not appear in more than one partition.
    Appendix A.3 describes a stratified split by category but never mentions speaker identity; with 19 actors, this is critical for valid benchmarks.
  • domain assumption The LLM-expanded speaking-style captions are semantically consistent with the original human annotations.
    Section 3.2 says an LLM expands integrated descriptions into five versions, but no human evaluation of these expansions is reported.
  • ad hoc to paper The weighted-confidence aggregation formula x_k = (1/N_k) sum I(y_i=k)*c_i is numerically correct for producing final emotion labels.
    The formula in Section 3.2 divides by the number of annotators per category N_k, which can make a category chosen by a single high-confidence annotator outrank a majority choice; the intended behavior is unclear.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations." pith.science (2026). https://pith.science/paper/BULSTZZ7

@misc{pith2026250523018,
  author       = {Pith},
  title        = {Pith review of: EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BULSTZZ7}},
  note         = {Machine review of arXiv:2505.23018}
}
read the original abstract

In recent years, emotion recognition plays a critical role in applications such as human-computer interaction, mental health monitoring, and sentiment analysis. While datasets for emotion analysis in languages such as English have proliferated, there remains a pressing need for high-quality, comprehensive datasets tailored to the unique linguistic, cultural, and multimodal characteristics of Chinese. In this work, we propose \textbf{EmotionTalk}, an interactive Chinese multimodal emotion dataset with rich annotations. This dataset provides multimodal information from 19 actors participating in dyadic conversational settings, incorporating acoustic, visual, and textual modalities. It includes 23.6 hours of speech (19,250 utterances), annotations for 7 utterance-level emotion categories (happy, surprise, sad, disgust, anger, fear, and neutral), 5-dimensional sentiment labels (negative, weakly negative, neutral, weakly positive, and positive) and 4-dimensional speech captions (speaker, speaking style, emotion and overall). The dataset is well-suited for research on unimodal and multimodal emotion recognition, missing modality challenges, and speech captioning tasks. To our knowledge, it represents the first high-quality and versatile Chinese dialogue multimodal emotion dataset, which is a valuable contribution to research on cross-cultural emotion analysis and recognition. Additionally, we conduct experiments on EmotionTalk to demonstrate the effectiveness and quality of the dataset. It will be open-source and freely available for all academic purposes. The dataset and codes will be made available at: https://github.com/NKU-HLT/EmotionTalk.

Figures

Figures reproduced from arXiv: 2505.23018 by the authors.

Figure 1
Figure 1. Data distribution across different modalities. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Snapshots of audio and video samples in the EmotionTalk dataset. All files are named [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Overview of the annotation platform interface. [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens

    cs.SD 2026-07 conditional novelty 5.0 of 10

    Downstream ASR/SER performance is largely insensitive to the neural-audio-codec training language but strongly depends on the SSL pre-training language in codec-based self-supervised speech learning.

Reference graph

Works this paper leans on

68 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in neural information processing systems, 33:12449–12460, 2020

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in neural information processing systems, 33:12449–12460, 2020

    2020

  2. [2]

    Data2vec: A general framework for self-supervised learning in speech, vision and language

    Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International conference on machine learning, pages 1298–1312. PMLR, 2022

    2022

  3. [3]

    Iemocap: Interactive emotional dyadic motion capture database.Language resources and evaluation, 42:335–359, 2008

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database.Language resources and evaluation, 42:335–359, 2008

    work page 2008

  4. [4]

    Msp-improv: An acted corpus of dyadic interactions to study emotion perception.IEEE Transactions on Affective Computing, 8(1):67–80, 2016

    Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. Msp-improv: An acted corpus of dyadic interactions to study emotion perception.IEEE Transactions on Affective Computing, 8(1):67–80, 2016

    work page 2016

  5. [5]

    Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014

    2014

  6. [6]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre- training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

    2022

  7. [7]

    Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality.See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality.See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023

    work page 2023

  8. [8]

    Lert: A linguistically-motivated pre-trained language model.arXiv preprint arXiv:2211.05344, 2022

    Yiming Cui, Wanxiang Che, Shijin Wang, and Ting Liu. Lert: A linguistically-motivated pre-trained language model.arXiv preprint arXiv:2211.05344, 2022

    arXiv 2022

Show all 68 references
  1. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...

    2019

  2. [10]

    Glm: General language model pretraining with autoregressive blank infilling.arXiv preprint arXiv:2103.10360, 2021

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling.arXiv preprint arXiv:2103.10360, 2021

    2021 arXiv

  3. [11]

    Atta-net: attention aggregation network for audio-visual emotion recognition

    Ruijia Fan, Hong Liu, Yidi Li, Peini Guo, Guoquan Wang, and Ti Wang. Atta-net: attention aggregation network for audio-visual emotion recognition. InICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8030–8034. IEEE, 2024

    2024

  4. [12]

    Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171, 2024

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171, 2024. 10

    2024

  5. [13]

    Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation

    Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the...

    2019

  6. [14]

    Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis.arXiv preprint arXiv:2109.00412, 2021

    Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis.arXiv preprint arXiv:2109.00412, 2021

    2021 arXiv

  7. [15]

    Conversational memory network for emotion recognition in dyadic dialogue videos

    Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. Conversational memory network for emotion recognition in dyadic dialogue videos. InProceedings of the conference. Association for Computational Linguistics. North Americ...

    2018

  8. [16]

    Misa: Modality-invariant and-specific representations for multimodal sentiment analysis

    Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. InProceedings of the 28th ACM international conference on multimedia, pages 1122–1131, 2020

    2020

  9. [17]

    Deberta: Decoding-enhanced bert with disentangled attention.arXiv preprint arXiv:2006.03654, 2020

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention.arXiv preprint arXiv:2006.03654, 2020

    2006 arXiv

  10. [18]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM Trans

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29: 3451–3460, October...

    2021

  11. [19]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3...

    2021

  12. [20]

    Mm-dfn: Multimodal dynamic fusion network for emotion recognition in conversations

    Dou Hu, Xiaolong Hou, Lingwei Wei, Lianxin Jiang, and Yang Mo. Mm-dfn: Multimodal dynamic fusion network for emotion recognition in conversations. InICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7037–7041, 2022. doi...

    2022

  13. [21]

    Libritts-p: A corpus with speaking style and speaker identity prompts for text- to-speech and style captioning, 2024

    Masaya Kawamura, Ryuichi Yamamoto, Yuma Shirahata, Takuya Hasumi, and Kentaro Tachibana. Libritts-p: A corpus with speaking style and speaker identity prompts for text- to-speech and style captioning, 2024. URLhttps://arxiv.org/abs/2406.07969

    2024 arXiv

  14. [22]

    Gpt2: Empirical slant delay model for radio space geodetic techniques.Geophysical research letters, 40(6):1069–1073, 2013

    Klemens Lagler, Michael Schindelegger, Johannes Böhm, Hana Krásná, and Tobias Nilsson. Gpt2: Empirical slant delay model for radio space geodetic techniques.Geophysical research letters, 40(6):1069–1073, 2013

    2013

  15. [23]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.arXiv preprint arXiv:1910.13461, 2019

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.arXiv preprint arXiv:1910.13461, 2019

    1910 arXiv

  16. [24]

    Merbench: A unified evaluation benchmark for multimodal emotion recognition.arXiv preprint arXiv:2401.03429, 2024

    Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, and Jianhua Tao. Merbench: A unified evaluation benchmark for multimodal emotion recognition.arXiv preprint arXiv:2401.03429, 2024

    2024 arXiv

  17. [25]

    AlignCap: Aligning speech emotion captioning to human preferences

    Ziqi Liang, Haoxiang Shi, and Hanhui Chen. AlignCap: Aligning speech emotion captioning to human preferences. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3837–3846,...

    2024 doi

  18. [26]

    Aligncap: Aligning speech emotion captioning to human preferences.arXiv preprint arXiv:2410.19134, 2024

    Ziqi Liang, Haoxiang Shi, and Hanhui Chen. Aligncap: Aligning speech emotion captioning to human preferences.arXiv preprint arXiv:2410.19134, 2024

    2024 arXiv

  19. [27]

    Emotion and intent joint understanding in multimodal conversation: A benchmarking dataset.arXiv preprint arXiv:2407.02751, 2024

    Rui Liu, Haolin Zuo, Zheng Lian, Xiaofen Xing, Björn W Schuller, and Haizhou Li. Emotion and intent joint understanding in multimodal conversation: A benchmarking dataset.arXiv preprint arXiv:2407.02751, 2024

    2024 arXiv

  20. [28]

    Discriminative feature representation based on cascaded attention network with adversarial joint loss for speech emotion recognition

    Yang Liu, Haoqin Sun, Wenbo Guan, Yuqi Xia, and Zhen Zhao. Discriminative feature representation based on cascaded attention network with adversarial joint loss for speech emotion recognition. InInterspeech 2022, pages 4750–4754, 2022. doi: 10.21437/Interspeech. 2022-11480

    2022 doi

  21. [29]

    Multi-level knowledge distillation for speech emotion recognition in noisy conditions

    Yang Liu, Haoqin Sun, Geng Chen, Qingyue Wang, Zhen Zhao, Xugang Lu, and Longbiao Wang. Multi-level knowledge distillation for speech emotion recognition in noisy conditions. InProc. Interspeech 2023, pages 1893–1897, 2023

    2023

  22. [30]

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  23. [31]

    Efficient low-rank multimodal fusion with modality-specific factors

    Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064, 2018

    2018 arXiv

  24. [32]

    Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english.PloS one, 13(5):e0196391, 2018

    2018

  25. [33]

    A transformer-based model with self-distillation for multimodal emotion recognition in conversations.IEEE Trans- actions on Multimedia, 26:776–788, 2024

    Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, and Bo Xu. A transformer-based model with self-distillation for multimodal emotion recognition in conversations.IEEE Trans- actions on Multimedia, 26:776–788, 2024. doi: 10.1109/TMM.2023.3271019

    2024

  26. [34]

    Dialoguernn: An attentive rnn for emotion detection in conversations

    Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6818–6825, 2019

    2019

  27. [35]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  28. [36]

    Found in translation: Learning robust joint representations by cyclic translations between modalities

    Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. Found in translation: Learning robust joint representations by cyclic translations between modalities. InProceedings of the AAAI conference on artificial intelligence, volume 33, pages 6892–6...

    2019

  29. [37]

    Meld: A multimodal multi-party dataset for emotion recognition in conver- sations

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conver- sations. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pag...

    2019

  30. [38]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  31. [39]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  32. [40]

    Sentence-bert: Sentence embeddings using siamese bert- networks.arXiv preprint arXiv:1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks.arXiv preprint arXiv:1908.10084, 2019. 12

    1908 arXiv

  33. [41]

    Audio-visual emotion forecasting: Characterizing and predicting future emotion using deep learning

    Sadat Shahriar and Yelin Kim. Audio-visual emotion forecasting: Characterizing and predicting future emotion using deep learning. In2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pages 1–7. IEEE, 2019

    2019

  34. [42]

    Dimensional emotion prediction based on interactive context in conversation

    Xiaohan Shi, Sixia Li, and Jianwu Dang. Dimensional emotion prediction based on interactive context in conversation. InINTERSPEECH, pages 4193–4197, 2020

    2020

  35. [43]

    Emotion awareness in multi-utterance turn for improving emotion prediction in multi-speaker conversation

    Xiaohan Shi, Xingfeng Li, and Tomoki Toda. Emotion awareness in multi-utterance turn for improving emotion prediction in multi-speaker conversation. InProc. Interspeech, pages 765–769, 2023

    2023

  36. [44]

    Iterative prototype refinement for ambiguous speech emotion recognition

    Haoqin Sun, Shiwan Zhao, Xiangyu Kong, Xuechen Wang, Hui Wang, Jiaming Zhou, and Yong Qin. Iterative prototype refinement for ambiguous speech emotion recognition. InProc. Interspeech 2024, pages 3200–3204, 2024

    2024

  37. [45]

    Fine- grained disentangled representation learning for multimodal emotion recognition

    Haoqin Sun, Shiwan Zhao, Xuechen Wang, Wenjia Zeng, Yong Chen, and Yong Qin. Fine- grained disentangled representation learning for multimodal emotion recognition. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11051...

    2024

  38. [46]

    Enhancing emotion recognition in incom- plete data: A novel cross-modal alignment, reconstruction, and refinement framework

    Haoqin Sun, Shiwan Zhao, Shaokai Li, Xiangyu Kong, Xuechen Wang, Jiaming Zhou, Aobo Kong, Yong Chen, Wenjia Zeng, and Yong Qin. Enhancing emotion recognition in incom- plete data: A novel cross-modal alignment, reconstruction, and refinement framework. In ICASSP 2025-2025 IEEE...

    2025

  39. [47]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Advances in neural information processing systems, 35:10078–10093, 2022

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Advances in neural information processing systems, 35:10078–10093, 2022

    2022

  40. [48]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023 arXiv

  41. [49]

    Learning factorized multimodal representations.arXiv preprint arXiv:1806.06176, 2018

    Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Rus- lan Salakhutdinov. Learning factorized multimodal representations.arXiv preprint arXiv:1806.06176, 2018

    2018 arXiv

  42. [50]

    Multimodal transformer for unaligned multimodal language sequences

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. InProceedings of the conference. Association for computational linguistics. Meeting, volume 20...

    2019

  43. [51]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  44. [52]

    Mmgcn: Multi-modal graph convolution network for personalized recommendation of micro- video

    Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. Mmgcn: Multi-modal graph convolution network for personalized recommendation of micro- video. InProceedings of the 27th ACM international conference on multimedia, pages 1437– 1445, 2019

    2019

  45. [53]

    Bloom: A 176b-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022

    BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022

    2022 arXiv

  46. [54]

    Secap: Speech emotion captioning with large language model

    Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shi-Xiong Zhang, Guangzhi Li, Yi Luo, and Rongzhi Gu. Secap: Speech emotion captioning with large language model. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19323–19331, 2024. 13

    2024

  47. [55]

    Baichuan 2: Open large-scale language models.arXiv preprint arXiv:2309.10305, 2023

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models.arXiv preprint arXiv:2309.10305, 2023

    2023 arXiv

  48. [56]

    Qwen2 technical report, 2024.URL https://arxiv

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report, 2024.URL https://arxiv. org/abs/2407.10671

    2024 arXiv

  49. [57]

    Disentangled representation learning for multimodal emotion recognition

    Dingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du, and Lihua Zhang. Disentangled representation learning for multimodal emotion recognition. InProceedings of the 30th ACM International Conference on Multimedia, pages 1642–1651, 2022

    2022

  50. [58]

    Self-adaptive context and modal-interaction modeling for multimodal emotion recognition

    Haozhe Yang, Xianqiang Gao, Jianlong Wu, Tian Gan, Ning Ding, Feijun Jiang, and Liqiang Nie. Self-adaptive context and modal-interaction modeling for multimodal emotion recognition. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Findings of the Association for...

    2023 doi

  51. [59]

    CLGSI: A multimodal sentiment analysis framework based on contrastive learning guided by sentiment intensity

    Yang Yang, Xunde Dong, and Yupeng Qiang. CLGSI: A multimodal sentiment analysis framework based on contrastive learning guided by sentiment intensity. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Findings of the Association for Computa- tional Linguistics: NAACL 202...

    2024 doi

  52. [60]

    An interaction-aware attention network for speech emotion recognition in spoken dialogs

    Sung-Lin Yeh, Yun-Shao Lin, and Chi-Chun Lee. An interaction-aware attention network for speech emotion recognition in spoken dialogs. InICASSP 2019-2019 IEEE International conference on acoustics, speech and signal processing (ICASSP), pages 6685–6689. IEEE, 2019

    2019

  53. [61]

    Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality

    Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. InProceedings of the 58th annual meeting of the association for computational ling...

    2020

  54. [62]

    Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos.arXiv preprint arXiv:1606.06259, 2016

    Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos.arXiv preprint arXiv:1606.06259, 2016

    2016 arXiv

  55. [63]

    Tensor fusion network for multimodal sentiment analysis.arXiv preprint arXiv:1707.07250, 2017

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis.arXiv preprint arXiv:1707.07250, 2017

    2017 arXiv

  56. [64]

    Memory fusion network for multi-view sequential learning

    Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. Memory fusion network for multi-view sequential learning. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  57. [65]

    Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. InProceedings of the 56th Annual Meeting of the Association for Computational Ling...

    2018

  58. [66]

    Tag-assisted multimodal sentiment analysis under uncertain missing modalities

    Jiandian Zeng, Tianyi Liu, and Jiantao Zhou. Tag-assisted multimodal sentiment analysis under uncertain missing modalities. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1545–1554, 2022

    2022

  59. [67]

    Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities

    Jiandian Zeng, Jiantao Zhou, and Tianyi Liu. Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2924–2934, 2022

    2022

  60. [68]

    <group_No>_<session_No>_<Speaker_id>_<Utt_No>.json

    Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, and Haizhou Li. M3ed: Multi-modal multi-scene multi-label emotional dialogue database.arXiv preprint arXiv:2205.10237, 2022. 14 A Datasheets for datasets A.1 Dataset Snapshots The dataset comprises 744...

    2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.