Pith. sign in

REVIEW 5 major objections 6 minor 39 references

MELT: Towards Automated Multimodal Emotion Data Annotation by Leveraging LLM Embedded Knowledge

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GPT-4o can annotate emotional speech from transcripts alone, and its labels beat human labels in preference tests and improve downstream speech emotion recognition.

desk verdict Solid empirical contribution with a real caveat: the paper never checks whether GPT-4o memorized MELD's public labels, which weakens its central claim of annotation from embedded knowledge. read the letter →

arxiv 2505.24493 v1 pith:2OU2NZRB submitted 2025-05-30 cs.AI cs.SDeess.AS

Xin Jing , Jiadong Wang , Iosif Tsangko , Andreas Triantafyllopoulos , Björn W. Schuller This is my paper
classification cs.AIcs.SDeess.AS
keywords speechemotionrecognitionLLM-basedannotationGPT-4omultimodaldatasetMELTpromptengineeringself-supervisedlearningaffectivecomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a large language model can annotate emotional speech data without hearing a single audio file, by drawing on the cultural knowledge it absorbed during pretraining. The authors take the MELD corpus, 13,708 utterances from the TV show Friends, filter it to 8,821 utterances, and have GPT-4o re-annotate each line from a text prompt that names the speaker, episode, and dialogue and asks for the emotion and voice qualities in a fixed JSON format. In a blind comparison, human raters preferred GPT-4o's labels to MELD's original human labels, and speech emotion recognition models fine-tuned on the new labels generally transferred better to four held-out emotion datasets. If these results hold, LLM embedded knowledge becomes a viable substitute for costly, inconsistent human annotation in affective computing.

What carries the argument

The load-bearing mechanism is GPT-4o's embedded knowledge of popular culture, activated by a purpose-built prompting framework. Each prompt supplies the speaker, season/episode, utterance text, and a seven-emotion taxonomy, then asks the model to reason through how the voice would sound — emotion, loudness, pitch, rhythm speed, emotional impact — before emitting a structured JSON answer; Chain-of-Thought prompting, cross-validation questions, and pre-filled output structure keep the annotations grounded and consistent. The textual context is the only multimodal input the model receives: the audio and video of Friends are never shown, so the acoustic and situational commonsense must come from what GPT-4o learned at scale.

What would settle it

Take a freshly recorded emotional speech dataset that has never been published or discussed online, run the paper's exact prompt on its transcripts, and have human raters compare the GPT-4o labels against new human labels. If the model's agreement, preference scores, and downstream SER gains fall to near chance, the reported advantage comes from memorized knowledge of MELD rather than a general ability to annotate emotional speech from text.

Watch

Extended reading notes

Core claim

The central claim is that MELT is a multimodal emotion dataset fully annotated by GPT-4o, and that its labels are both closer to human preference and more useful for training speech emotion recognition systems than the human majority-vote labels of MELD. The paper reports that 46.43% of training labels and 47.52% of test labels changed relative to MELD, that aggregate MOS ratings from 20 blind raters favored MELT (with the largest gaps for anger and surprise), and that fine-tuning four self-supervised backbones on MELT produced higher UAR, accuracy, and F1 than training on MELD in most configurations across IEMOCAP, TESS, RAVDESS, and CREMA-D. The authors frame this as the first evidence that GPT-class models can act as annotators for multimodal emotion data, using only text plus the knowledge the model has internalized about a well-known television series.

Load-bearing premise

The method assumes GPT-4o's emotion judgments come from its general cultural knowledge and are not answers memorized from MELD, the public dataset whose episode-and-utterance prompts likely appear in the model's training data.

Editorial extensions

If this is right

  • Emotion annotation for widely known media can be produced from transcripts alone for roughly $10, replacing multi-annotator campaigns.
  • SER models fine-tuned on LLM labels should generalize to unseen emotion corpora at least as well as, and often better than, models trained on human majority-vote labels.
  • The same structured prompting template can be applied to other scripted dialogue datasets to obtain consistent, context-aware emotion labels without additional human labor.
  • LLM re-annotation yields a more balanced emotion distribution than the original human labels, which can reduce majority-class bias in downstream training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: Because the prompt reveals the exact show, season, episode, and line, and MELD is a public dataset, some of the agreement and preference scores could be inflated by GPT-4o having memorized MELD's published labels; the paper does not test this.
  • Editorial extension: The method is likely to transfer only to content that is densely present in web-scale training data; for private conversations, low-resource languages, or novel recordings the embedded-knowledge advantage may shrink or disappear.
  • Editorial extension: Adding a brief audio-derived description to the prompt, rather than only transcripts, could let the pipeline annotate never-seen content while keeping LLM reasoning, a testable variant the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes MELT, a re-annotation of a filtered subset of the MELD dataset (8,821 utterances from 42 Friends characters) using GPT-4o with text-only prompts that include character, episode, and utterance context. The authors report that MELT labels differ from MELD on about half the utterances, are preferred by human raters in a MOS experiment, and improve fine-tuned SSL-based SER performance on IEMOCAP, TESS, RAVDESS, and CREMA-D relative to training on MELD. They also report that GPT-4o's perceived pitch/loudness descriptions correlate with eGeMAPS-derived categories above chance. The central claim is that this demonstrates that LLM embedded cultural knowledge can automatically annotate multimodal emotion data without human labor.

Significance. The work is potentially significant: if the annotations are genuinely produced from general knowledge rather than memorized benchmark labels, the approach offers a very low-cost, scalable alternative to crowdsourced emotion annotation and provides a public dataset (MELT) with audio-attribute pseudo-labels. Strengths include the public release of the dataset, a clearly described prompting framework, four SSL backbones, four external SER corpora, and a human preference study. The cross-corpus design partially grounds the evaluation in independent benchmarks, which is a positive feature. However, the validity of the central claim depends on ruling out memorization of MELD, and the empirical comparisons lack statistical support. These issues are addressable but currently open.

major comments (5)
  1. [Sec. 2.3 / Sec. 3] The paper does not address the possibility that GPT-4o has memorized MELD's published emotion labels. The prompt in Sec. 2.3 reveals the exact speaker, season, episode, and utterance text; MELD was released in 2019 and its label files are widely available, well before the model's October 2023 knowledge cutoff stated in Sec. 2.2. If GPT-4o reproduces memorized MELD labels, then the 46.43%/47.52% label-change rates in Sec. 3, the MOS preferences in Fig. 3, and the SER gains in Table 3 do not establish annotation from embedded cultural knowledge and do not support scalability to unannotated data. This is an identifiability problem in the experimental design. A concrete remedy is to run control annotations in which episode/character identifiers are removed or paraphrased, and on utterances from a different sitcom not in MELD, and to compare agreement with MELD across conditions; the authors could also directly prompt GPT-4o to reproduce MELD labels for a sample and quantify overlap. Without such a check, the headline claim is not established.
  2. [Sec. 4.2 / Table 3] The objective experiments are reported without variance or significance testing. Each cell in Table 3 appears to come from a single training run, and the claimed 'consistently improved' performance is contradicted by several cells, e.g., wav2vec 2.0 Aud on TESS has F1 0.1926 (MELT) vs 0.2063 (MELD), and WavLM Base+ on TESS has F1 0.2372 vs 0.2405. Without repeated seeds, standard deviations, or paired tests, the cross-corpus advantage of MELT cannot be distinguished from noise. Please report means and standard deviations over at least 3-5 seeds and appropriate significance tests (e.g., paired t-test or Wilcoxon test) on the per-utterance or per-run scores.
  3. [Sec. 4.1 / Fig. 3] The MOS experiment is described only as 20 participants choosing between MELT and MELD annotations on video clips. The paper reports aggregate preference percentages per emotion but omits the number of items rated, the number of raters per item, inter-rater agreement, and any statistical test. Since the subjective preference is a headline result supporting the claim that MELT 'aligns more closely with human preferences,' the experiment needs per-item details and a significance test (e.g., Wilcoxon signed-rank test over items or raters).
  4. [Sec. 2.2] Annotations are sampled at temperature 1.0, yet the paper claims a prompt framework ensuring 'stability and reproducibility' in Sec. 2.3. No repeated annotation runs, sampling strategy, or consistency metrics are reported. If the labels are not stable across samples, the released dataset and downstream fine-tuning results are not reproducible. Please report the number of samples per utterance, the aggregation rule (if any), and agreement across repeated API calls.
  5. [Sec. 4.2.2] The filtering rule for training data is underspecified for partially overlapping test sets. The paper says 'only emotion categories present in the test set are retained,' but it does not state how MELT/MELD's seven labels are mapped or filtered for IEMOCAP (4 classes) and CREMA-D (6 classes), nor how the classification head is set up when the training and test label sets differ. Please specify the exact label filtering/mapping per dataset, including any mapping of 'joy' to 'happy' or 'pleasant surprise' to 'surprise,' and confirm that the same procedure is applied to both MELT and MELD training conditions.
minor comments (6)
  1. [Sec. 2.1 / Table 1] The total number of MELD utterances is given as 13,708 in Sec. 2.1, but Table 1 lists 9,989 + 2,610 = 12,599 for the train and test splits; please clarify that the 13,708 includes a development split not used in this work.
  2. [Abstract / Index Terms] The abstract contains 'consistence performance improvement,' which should be 'consistent performance improvement,' and the index terms contain 'affective compution,' which should be 'affective computing.'
  3. [Fig. 1] The caption refers to a 'confusion matrix and inter-label transition matrix,' but both panels appear to be transition matrices from MELD to MELT; please use consistent terminology.
  4. [Sec. 2.3] The individual prompt components (context, Chain-of-Thought, cross-validation, and prefilling) are not ablated, so the contribution of each to annotation quality is unverified; a small ablation study would strengthen the prompt-design contribution.
  5. [Sec. 5.2 / Table 4] The pitch and loudness predictions are evaluated against binned eGeMAPS attributes with bin boundaries defined by distribution percentiles, but it is not stated whether the percentiles are estimated on the training split, test split, or both; please clarify to avoid possible leakage.
  6. [Sec. 1] The cost claim of '≤$10' is not supported by a calculation; please provide the number of API calls, input/output token counts, and the pricing used to obtain the estimate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MELT labels are generated by GPT-4o from text prompts and validated on independent corpora; the memorization concern is a data-contamination risk, not a construction-level circularity.

full rationale

The claimed derivation chain is: GPT-4o annotates Friends utterances from text-only prompts (Section 2.3), producing MELT labels (Section 3); these labels are then evaluated by MOS preference tests (Section 4.1) and by fine-tuning SSL backbones for SER on four independent datasets (Section 4.2.2). No step defines its output in terms of its input by construction. The label-change rates in Section 3 (46.43% and 47.52%) are descriptive statistics, not fitted parameters, and the SER improvements are measured on IEMOCAP, TESS, RAVDESS, and CREMA-D, none of which are derived from MELT or from the GPT-4o annotation process. The paper's self-citations (e.g., ParaCLAP for the audio-characteristic query pipeline) are methodological and not load-bearing for the central claim. The skeptical concern that GPT-4o might have memorized MELD's public labels, especially because the prompt includes the exact episode and utterance, is a legitimate external-validity threat, but the paper provides no evidence that the agreement is explained by memorization rather than by text semantics and general emotion knowledge. Without demonstrating that the model reproduces MELD labels beyond what ordinary linguistic and emotional reasoning predicts, this remains a contamination risk rather than a circularity established by the paper's own construction. Therefore, under the requirement to exhibit a specific reduction, no circular step is found, and the score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

Central claim rests on GPT-4o's unsupervised text-only annotation quality, validated by subjective preference and SER benchmarks. The counted free parameters are hand-chosen settings that directly shape the dataset and its evaluation; standard SSL training hyperparameters are omitted.

free parameters (4)
  • Minimum utterance duration filter = 1 second
    Hand-chosen exclusion threshold for short utterances (Section 2.1); changes dataset composition and label counts.
  • Speaker inclusion criterion = 42 speakers
    Characters without clear names/gender identifiers are excluded; the resulting speaker set is a modeling choice, not a principled rule.
  • GPT-4o sampling temperature = 1.0
    Chosen for annotation; introduces stochasticity and affects reproducibility.
  • Pitch and loudness bin boundaries = bottom 30%, middle 40%, top 30%
    Arbitrary binning for the pseudo-caption evaluation in Section 5.2.
assumptions (4)
  • domain assumption GPT-4o's text-only embedded knowledge of Friends is sufficient to produce valid audio emotion annotations.
    The entire pipeline rests on this; introduced in Sections 1 and 2.3 without independent verification.
  • domain assumption MELD majority-vote human labels are a meaningful baseline for annotation quality.
    Used as the comparator in MOS and SER training; no adjudication of whether MELD labels are themselves correct.
  • domain assumption Downstream cross-corpus SER performance is a valid proxy for annotation label quality.
    Table 3 interprets fine-tuning results on IEMOCAP, TESS, RAVDESS, and CREMA-D as evidence that MELT labels are better; this assumes transferability.
  • ad hoc to paper GPT-4o has not memorized MELD's published labels.
    The paper never states or verifies this; without it, the MELT-vs-MELD comparison is partly circular.
invented entities (1)
  • MELT dataset independent evidence
    purpose: New emotion annotation resource for multimodal SER research derived from MELD via GPT-4o.
    Not a theoretical postulate; a released artifact with a public GitHub link, but its quality is exactly what the paper must establish.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MELT: Towards Automated Multimodal Emotion Data Annotation by Leveraging LLM Embedded Knowledge." pith.science (2026). https://pith.science/paper/2OU2NZRB

@misc{pith2026250524493,
  author       = {Pith},
  title        = {Pith review of: MELT: Towards Automated Multimodal Emotion Data Annotation by Leveraging LLM Embedded Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2OU2NZRB}},
  note         = {Machine review of arXiv:2505.24493}
}
read the original abstract

Although speech emotion recognition (SER) has advanced significantly with deep learning, annotation remains a major hurdle. Human annotation is not only costly but also subject to inconsistencies annotators often have different preferences and may lack the necessary contextual knowledge, which can lead to varied and inaccurate labels. Meanwhile, Large Language Models (LLMs) have emerged as a scalable alternative for annotating text data. However, the potential of LLMs to perform emotional speech data annotation without human supervision has yet to be thoroughly investigated. To address these problems, we apply GPT-4o to annotate a multimodal dataset collected from the sitcom Friends, using only textual cues as inputs. By crafting structured text prompts, our methodology capitalizes on the knowledge GPT-4o has accumulated during its training, showcasing that it can generate accurate and contextually relevant annotations without direct access to multimodal inputs. Therefore, we propose MELT, a multimodal emotion dataset fully annotated by GPT-4o. We demonstrate the effectiveness of MELT by fine-tuning four self-supervised learning (SSL) backbones and assessing speech emotion recognition performance across emotion datasets. Additionally, our subjective experiments\' results demonstrate a consistence performance improvement on SER.

Figures

Figures reproduced from arXiv: 2505.24493 by the authors.

Figure 1
Figure 1. Confusion matrix and inter-label transition matrix of the training and test dataset Audio wav SSL backbone “Emo Prediction” [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. MOS results on every emotion categories. 5. Results and Analysis 5.1. Performance The overall MOS result is shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 22 canonical work pages

  1. [1]

    MELT: Towards Automated Multimodal Emotion Data Annotation by Leveraging LLM Embedded Knowledge

    Introduction Recognizing human emotion and responding accordingly is a cornerstone of human-computer interaction [1]. The progress made by contemporary deep-learning-based emotion recog- nition heavily depends on the availability of well-annotated datasets [2]. However, ensuring accurate and consistent anno- tation requires multiple annotators and validat...

    work page Pith review arXiv 2025

  2. [2]

    1st Customer

    Methodology 2.1. Data Preparation Multimodal EmotionLines Dataset (MELD) [16], built from the TV series “Friends”, comprises 1,433 dialogues and 13,708 ut- terances. Each utterance is annotated with one of seven cate- gories (Joy, Sadness, Fear, Anger, Surprise, Disgust, Neutral) based on a majority vote among three annotators. MELT is de- rived from MELD...

    work page 2024

  3. [3]

    Emo Prediction

    The MELT Corpus Following Section 2.1, MELT retains 42 unique speakers across both the training and test sets, ensuring consistency and fair evaluation. As summarized in Table 1, approximately 70% of the original utterances in MELD have been preserved. Table 1:Summary statistics of MELT and MELD for training and test sets. #Utt, #Spk, #Avg. Sec. represent...

    work page

  4. [4]

    Subjective Experiment To assess the emotion annotation quality, we invited 20 par- ticipants, comprising 11 males and 9 females to conduct a Mean Opinion Score (MOS) experiment

    Experiments 4.1. Subjective Experiment To assess the emotion annotation quality, we invited 20 par- ticipants, comprising 11 males and 9 females to conduct a Mean Opinion Score (MOS) experiment. Participants were in- structed to watch the video clips and were presented with two options—MELT and MELD annotations, without knowing their sources —from which t...

    work page arXiv

  5. [5]

    Performance The overall MOS result is shown in Fig

    Results and Analysis 5.1. Performance The overall MOS result is shown in Fig. 3, participants over- all demonstrated a preference for MELT annotations. The high agreement (>70%) for ‘anger’ and ‘surprise’ indicates that GPT-4o effectively integrates internet-sourced knowledge to capture the diversity in expressions. However, an opposite trend is observed ...

    work page

  6. [6]

    To achieve this, we developed a prompting strategy incorporating cross-validation and CoT rea- soning to ensure consistent and accurate annotations

    Conclusion In this work, we introduce MELT , a multimodal emotion dataset fully annotated by GPT-4o using a context-aware au- tomatic annotation method. To achieve this, we developed a prompting strategy incorporating cross-validation and CoT rea- soning to ensure consistent and accurate annotations. The MOS and classification results highlights that MELT...

    work page

  7. [7]

    Schuller is also with the Munich Data Science Insti- tute and the Konrad Zuse School of Excellence in Reliable AI, both in Munich, Germany

    Acknowledgment Bj¨orn W. Schuller is also with the Munich Data Science Insti- tute and the Konrad Zuse School of Excellence in Reliable AI, both in Munich, Germany. We are grateful to the China Schol- arship Council (CSC), Grant # 202006290013 to support this work. We would also like to thank Prof. Xinzhou Xu and ev- eryone involved in the MOS experiments...

    work page

  8. [8]

    Be- yond deep learning: Charting the next frontiers of affective com- puting,

    A. Triantafyllopoulos, L. Christ, A. Gebhard, X. Jing, A. Kathan, M. Milling, I. Tsangko, S. Amiriparian, and B. W. Schuller, “Be- yond deep learning: Charting the next frontiers of affective com- puting,”Intelligent Computing, vol. 3, p. 0089, 2024

    work page 2024

Show all 39 references
  1. [9]

    En- hancing emotional text-to-speech controllability with natural lan- guage guidance through contrastive learning and diffusion mod- els,

    X. Jing, K. Zhou, A. Triantafyllopoulos, and B. W. Schuller, “En- hancing emotional text-to-speech controllability with natural lan- guage guidance through contrastive learning and diffusion mod- els,” inProceedings of International Conference on Acoustics, Speech and Signal P...

    2025

  2. [10]

    Emotion recognition in context,

    R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza, “Emotion recognition in context,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1667–1675

    2017

  3. [11]

    Contextual emotion recognition using large vision language models,

    Y . Etesam, ¨O. N. Yalc ¸ın, C. Zhang, and A. Lim, “Contextual emotion recognition using large vision language models,”arXiv preprint arXiv:2405.08992, 2024

    2024 arXiv

  4. [12]

    The human in emotion recognition on social media: Attitudes, outcomes, risks,

    N. Andalibi and J. Buss, “The human in emotion recognition on social media: Attitudes, outcomes, risks,” inProceedings of the 2020 CHI Conference on Human Factors in Computing Systems, New York, NY , USA, 2020, p. 1–16

    2020

  5. [13]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  6. [14]

    Language models are few-shot learners,

    T. B. Brown, “Language models are few-shot learners,”arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  7. [15]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radfordet al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  8. [16]

    Large language models for data annotation: A survey,

    Z. Tan, D. Li, S. Wang, A. Beigi, B. Jiang, A. Bhattacharjee, M. Karami, J. Li, L. Cheng, and H. Liu, “Large language models for data annotation: A survey,”arXiv preprint arXiv:2402.13446, 2024

    2024 arXiv

  9. [17]

    Chatgpt vs. human annotators: A comprehen- sive analysis of chatgpt for text annotation,

    M. Aldeen, J. Luo, A. Lian, V . Zheng, A. Hong, P. Yetukuri, and L. Cheng, “Chatgpt vs. human annotators: A comprehen- sive analysis of chatgpt for text annotation,” inProceedings of International Conference on Machine Learning and Applications (ICMLA), Florida, USA, 2023, pp. 602–609

    2023

  10. [18]

    Chatgpt outperforms crowd workers for text-annotation tasks,

    F. Gilardi, M. Alizadeh, and M. Kubli, “Chatgpt outperforms crowd workers for text-annotation tasks,”Proceedings of the Na- tional Academy of Sciences, vol. 120, no. 30, 2023

    2023

  11. [19]

    Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multimodal research,

    X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumb- ley, Y . Zou, and W. Wang, “Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multimodal research,”IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 32, ...

    2024

  12. [20]

    Pengi: An audio language model for audio tasks,

    S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,”Advances in Neural Infor- mation Processing Systems, vol. 36, pp. 18 090–18 108, 2023

    2023

  13. [21]

    Secap: Speech emotion captioning with large language model,

    Y . Xu, H. Chen, J. Yu, Q. Huang, Z. Wu, S.-X. Zhang, G. Li, Y . Luo, and R. Gu, “Secap: Speech emotion captioning with large language model,” inProceedings of the AAAI Conference on Ar- tificial Intelligence, vol. 38, 2024, pp. 19 323–19 331

    2024

  14. [22]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  15. [23]

    Meld: A multimodal multi-party dataset for emo- tion recognition in conversations,

    S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “Meld: A multimodal multi-party dataset for emo- tion recognition in conversations,” inProceedings of Annual Meeting of the Association for Computational Linguistics, Flo- rence, Italy, 2019, pp. 527–536

    2019

  16. [24]

    On the time course of vocal emotion recognition,

    M. D. Pell and S. A. Kotz, “On the time course of vocal emotion recognition,”PlOS one, vol. 6, no. 11, p. e27256, 2011

    2011

  17. [25]

    Applying tdnn architectures for an- alyzing duration dependencies on speech emotion recognition

    P. Kumawat and A. Routray, “Applying tdnn architectures for an- alyzing duration dependencies on speech emotion recognition.” in Interspeech, 2021, pp. 3410–3414

    2021

  18. [26]

    A wide evaluation of chatgpt on affective computing tasks,

    M. M. Amin, R. Mao, E. Cambria, and B. W. Schuller, “A wide evaluation of chatgpt on affective computing tasks,”IEEE Trans- actions on Affective Computing, vol. 15, no. 4, pp. 2204–2212, 2024

    2024

  19. [27]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  20. [28]

    Dawn of the trans- former era in speech emotion recognition: closing the valence gap,

    J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, “Dawn of the trans- former era in speech emotion recognition: closing the valence gap,”IEEE Transactions on Pattern Analysis and Machine Intel- ligence, vol. 45, no. 9, pp. ...

    2023

  21. [29]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  22. [30]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  23. [31]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  24. [32]

    Toronto emotional speech set (tess)

    M. K. Pichora-Fuller and K. Dupuis, “Toronto emotional speech set (tess).” Borealis, 2020. [Online]. Available: https: //doi.org/10.5683/SP2/E8H2MF

    2020 doi

  25. [33]

    The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,

    S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,”PloS one, vol. 13, no. 5, 2018

    2018

  26. [34]

    Crema-d: Crowd-sourced emotional multimodal actors dataset,

    H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “Crema-d: Crowd-sourced emotional multimodal actors dataset,”IEEE Transactions on Affective Computing, vol. 5, no. 4, pp. 377–390, 2014

    2014

  27. [35]

    Emo-superb: An in-depth look at speech emotion recognition,

    H. Wu, H.-C. Chou, K.-W. Chang, L. Goncalves, J. Du, J.-S. R. Jang, C.-C. Lee, and H.-Y . Lee, “Emo-superb: An in-depth look at speech emotion recognition,”arXiv preprint arXiv:2402.13018, 2024

    2024 arXiv

  28. [36]

    Paraclap– towards a general language-audio model for computational par- alinguistic tasks,

    X. Jing, A. Triantafyllopoulos, and B. Schuller, “Paraclap– towards a general language-audio model for computational par- alinguistic tasks,” inProceedings of International Speech Com- munication Association (INTERSPEECH), Kos Island, Greece, September 2024

    2024

  29. [37]

    The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,

    F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. Andr ´e, C. Busso, L. Y . Devillers, J. Epps, P. Laukka, S. S. Narayanan et al., “The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,”IEEE transactions on affective computi...

    2015

  30. [38]

    openSMILE: the Munich versatile and fast open-source audio feature extractor,

    F. Eyben, M. W ¨ollmer, and B. Schuller, “openSMILE: the Munich versatile and fast open-source audio feature extractor,” inProc. the International Conference on Multimedia, 2010, pp. 1459–1462

    2010

  31. [39]

    How do we describe other people from voices and faces?

    N. Lavan, “How do we describe other people from voices and faces?”Cognition, vol. 230, p. 105253, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.