Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LLM-labeled transcripts, combined with staged audio-text fusion, set new state-of-the-art scores for emotion recognition in conversations on MELD and CMU-MOSI.

desk verdict Useful recipe with a clear missing experiment: the LLM text pretraining is never ablated in the final fused model, so the headline attribution is not yet proven. read the letter →

arxiv 2501.11468 v1 pith:KYM24DLW submitted 2025-01-20 eess.AS cs.SD

Soumya Dutta , Sriram Ganapathy This is my paper · ORCID
classification eess.AScs.SD
keywords multimodalemotionrecognitionconversationalLLMpseudo-labelingself-supervisedpretraininghierarchicaltrainingco-attentionfusionspeech-textembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that emotion recognition in conversations can be bootstrapped without human emotion labels on the text side: speech transcripts from a large unlabeled corpus are transcribed by an ASR system, labeled as positive, neutral, or negative by an LLM, and used to fine-tune RoBERTa. These text embeddings are then combined with frozen CARE speech embeddings in a three-stage hierarchical trainer that first learns per-utterance emotions, then conversational context with Bi-GRUs and self-attention, and finally cross-attention fusion between modalities. The resulting model, MERITS-L, reports weighted F1 scores of 86.48 on IEMOCAP, 66.02 on MELD, and 86.81 on CMU-MOSI, beating prior published models on the latter two without using speaker identity. The sympathetic reader would care because the recipe suggests abundant unlabeled audio, an off-the-shelf LLM, and a staged fusion scheme can substitute for expensive human emotion annotation.

What carries the argument

The load-bearing machinery is a three-stage pipeline. Stage I fine-tunes RoBERTa-large on LLM pseudo-labels over Whisper-transcribed MSP-PODCAST utterances, and trains lightweight classifiers on frozen CARE embeddings. Stage II passes each modality's utterance embeddings through a Bi-GRU with self-attention to capture conversational context. Stage III fuses the resulting sequences with a co-attention network of cross-attention and self-attention blocks, letting each modality query the other at every utterance. The key identity is the staged freezing: each stage's outputs become the next stage's inputs, which the paper argues prevents overfitting on small conversation datasets.

What would settle it

Train the full three-stage MERITS-L on MELD and CMU-MOSI with the identical setup except replace the LLM-pseudo-labeled RoBERTa-FT with the original pretrained RoBERTa-large (no Stage I fine-tuning). If the weighted F1 scores do not drop materially from 66.02 and 86.81, the LLM-guided pretraining is not responsible for the state-of-the-art results.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that an LLM's coarse sentiment labels on noisy ASR transcripts are a viable supervisory signal for pretraining a text emotion encoder, and that a hierarchical fusion of this encoder's outputs with speech embeddings—rather than end-to-end multimodal training—yields the best conversational emotion recognition performance. The paper presents Stage I text-only gains of 8-22% relative over plain RoBERTa with GPT-3.5 Turbo labels, and shows that merging the two modalities after separate conversational modeling adds further gains of up to 16% relative on IEMOCAP, with smaller gains where text already dominates. It also reports that the method is not the best on IEMOCAP, where a prior semi-supervised model scores higher.

Load-bearing premise

The paper's central claim depends on the assumption that the LLM-generated pseudo-labels are what produce the reported gains in the final fused model, yet no ablation removes or replaces those labels in the full pipeline—the only supporting evidence is in the text-only Stage I.

Editorial extensions

If this is right

  • LLM-generated pseudo-labels on ASR transcripts can serve as a no-human-annotation pretraining signal for text emotion recognition, cutting the need for labeled emotion data.
  • Hierarchical training (utterance to conversation to multimodal) beats end-to-end training on all three datasets, suggesting staged freezing is a viable regularization for small conversational datasets.
  • The co-attention fusion yields the largest gains when the two modalities start at comparable accuracy, so the benefit of fusion is tied to balancing modality strengths.
  • The choice of LLM matters: GPT-3.5 Turbo's pseudo-labels outperform smaller open models, and downstream performance tracks how well LLM labels agree with gold valence annotations.
  • Because the method does not use speaker identity, its results are not directly comparable to speaker-aware systems; on MELD, speaker-aware methods report higher scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A control experiment that pretrains RoBERTa with the gold valence-arousal-dominance labels from MSP-PODCAST instead of LLM pseudo-labels is missing in the paper; if gold labels match or beat LLM labels in the full MERITS-L pipeline, the LLM's role reduces to approximating existing annotations rather than adding new information.
  • The paper's comparison table excludes speaker-aware methods; a fair comparison under the same no-speaker constraint would reveal whether the claimed MELD result is limited by the exclusion rather than by model design.
  • The correlation between LLM-gold label overlap and downstream F1 suggests a cheap selection strategy: keep only pretraining transcripts where LLM confidence is high, which should raise the quality of the silver labels.
  • Extending the fusion to dynamically weight modalities by their running confidence could address the paper's observation that gains shrink when one modality strongly dominates, as on MELD and CMU-MOSI.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MERITS-L, a hierarchical multimodal (speech and text) model for emotion recognition in conversations. The central novelty is an LLM-guided text pretraining stage: speech from the unlabeled MSP-PODCAST corpus is transcribed with Whisper-large-v3, pseudo-labels are generated by GPT-3.5 Turbo, and a RoBERTa-large encoder is fine-tuned on these silver labels. This text encoder, together with frozen CARE speech embeddings, is then used in a three-stage pipeline: Stage I trains utterance-level uni-modal classifiers, Stage II adds Bi-GRU conversational context with self-attention, and Stage III fuses the two modalities with a co-attention network. The model is evaluated on IEMOCAP, MELD, and CMU-MOSI, reporting weighted F1 scores of 86.48, 66.02, and 86.81 respectively, and claiming state-of-the-art results on MELD and CMU-MOSI. The paper also compares several LLM annotators for pseudo-label quality and analyzes the contribution of hierarchical training.

Significance. If the central claim were fully supported, the work would make a useful contribution: it demonstrates a low-cost way to transfer emotion knowledge from unlabeled speech into a text encoder via LLM pseudo-labels, and it shows a sensible staged training recipe for multimodal conversational emotion recognition. The use of a large unlabeled emotional speech corpus, the oracle VAD overlap analysis, and the comparison of three LLM annotators are commendable elements that go beyond a simple benchmark report. However, the significance is currently limited because the paper does not isolate the effect of the proposed LLM-guided pretraining in the final fused system, and the reported performance gains are not accompanied by variance estimates or significance tests.

major comments (4)
  1. [Sec. III-C, Sec. IV-D, Fig. 3] The central contribution of the paper is the LLM-guided text pretraining, but its impact is never evaluated in the final MERITS-L system. The only evidence for this component is Fig. 3, which reports Stage I text-only results; RoBERTa-FT is then a fixed input to Stages II and III, and no ablation replaces it with a baseline text encoder (e.g., vanilla RoBERTa or RoBERTa fine-tuned only on downstream labels) in the full hierarchical model. Consequently, the final numbers in Table I cannot be attributed to the LLM pretraining as opposed to the CARE embeddings, the hierarchical training, or the co-attention fusion. Please add an ablation that varies the text encoder within the full MERITS-L pipeline, and ideally also a control using the gold VAD labels from MSP-PODCAST to create the pseudo-labels.
  2. [Table I, Sec. IV-F] All reported results are single runs with no error bars, no multiple random seeds, and no significance tests. Several improvements are small in absolute terms (e.g., MELD Stage II to Stage III goes from 65.24 to 66.02, and CMU-MOSI from 86.02 to 86.81). Given that the SOTA claim on MELD and CMU-MOSI rests on margins of this size, the lack of variance information makes the claim fragile. Please report results over at least three random seeds as mean +/- std, and where possible run a significance test.
  3. [Table II, Sec. IV-F] The comparison in Table II is too narrow to support the state-of-the-art claim. Only five baselines are listed, mostly from 2019-2022, and the modality usage is not consistently marked: UniMSE is explicitly flagged as tri-modal, but SMIN and EmoCaps are not, even though their published systems may also use visual features. In addition, the exclusion of TelME and EACL because they use speaker identity is not matched by a systematic inclusion of speaker-agnostic recent baselines. Please expand the baseline table to recent methods (with modality and speaker-identity usage clearly marked) or soften the SOTA claim accordingly.
  4. [Fig. 3, Sec. IV-D] The baseline labeled 'pre-trained RoBERTa without any supervised fine-tuning' is not described with sufficient precision. It is unclear whether this baseline performs zero-shot prompting, uses a linear probe on frozen RoBERTa features, or fine-tunes a classifier head on downstream labels. Since this figure is the sole evidence for the benefit of LLM-guided pretraining, the evaluation protocol for the baseline must match the RoBERTa-FT pipeline (same downstream classification head and training schedule) except for the pretraining labels. Please specify this protocol explicitly.
minor comments (5)
  1. [Sec. IV-A] The IEMOCAP description says 'four-way classification task' but then lists five categories ('angry', 'happy', 'sad', 'neutral' and 'excited') before noting that happy and excited are merged. Please correct this wording to list the four final classes and clarify the mapping.
  2. [Sec. IV-B] The sentence 'we have not used any additional labeled datasets in pre-training as the pre-training framework for speech and text are purely based on self-supervised learning principles from raw data' is misleading: the text encoder is trained with LLM-generated pseudo-labels, which is a form of weak supervision, not self-supervision. Please rephrase to reflect that no human-annotated emotion labels are used.
  3. [Fig. 2, Sec. III-C] The co-attention network is described only verbally and by Figure 2. Adding the exact equations for the cross-attention and self-attention blocks would substantially improve reproducibility, since the paper does not otherwise specify the fusion mechanism in formal terms.
  4. [Sec. III-B2] There is a typo in 'Whipser' (should be 'Whisper') in the implementation details paragraph.
  5. [General] No code, model checkpoints, or data splits are provided. If available, a public release of the MERITS-L code and the pseudo-labeled MSP-PODCAST annotation files would strengthen the reproducibility of the reported results.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; the central claim rests on held-out ERC labels, though the contribution attribution is under-ablated.

full rationale

The derivation chain is not circular. RoBERTa-FT is trained on LLM pseudo-labels over Whisper transcripts of MSP-PODCAST (Sec. III-B2), and is then evaluated on IEMOCAP, MELD, and CMU-MOSI using gold test labels. The LLM pseudo-labels are weak external supervision over a different corpus; they are never taken from the ERC test sets, so the transfer evaluation is externally grounded. The CARE speech embeddings [35] are a self-cited component, but the paper uses them as frozen, independent features whose quality was assessed outside this paper (SUPERB), and the final Table I numbers are not derived from CARE's labels. The oracle experiment in Sec. IV-D uses MSP-PODCAST valence labels only to measure LLM overlap, not to train or define the downstream prediction. The hierarchical stages (I, II, III) are sequential training procedures, not definitional equivalences. The main weakness is empirical: there is no ablation removing LLM-guided text pretraining from the final MERITS-L system, so the reported gains cannot be cleanly attributed to the paper's central novelty. That is a missing control / attribution risk, not circularity: nothing in the paper defines the target result in terms of its inputs or forces the benchmark numbers by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; MERITS-L is an architecture assembled from existing components. The main uncharged assumptions concern the quality of ASR transcripts, the usefulness of LLM pseudo-labels, and the transferability of three-way sentiment pretraining to fine-grained ERC labels.

free parameters (3)
  • Training hyperparameters = lr=1e-4, batch_size=32, epochs=10 (text pretrain), 50 (MERITS-L stages)
    Chosen by hand and no sensitivity analysis is reported (Sec. IV-B).
  • Valence thresholds for oracle labels = positive: (5,7], negative: [1,3), neutral: otherwise
    A hand-chosen mapping used only for oracle evaluation of LLM labels, not for model training (Sec. IV-D).
  • LLM pseudo-label class count = 3 (positive, negative, neutral)
    Design choice with no ablation of finer-grained label sets (Sec. III-B2).
assumptions (4)
  • domain assumption Whisper-large-v3 ASR transcripts preserve enough emotional content for LLM labeling
    The text pretraining pipeline depends on transcripts being informative despite the known higher word error rate on emotional speech (Sec. IV-A, referencing [48]).
  • domain assumption GPT-3.5 Turbo pseudo-labels are a useful supervision signal for training RoBERTa for ERC
    The claimed benefit of pretraining; agreement with valence-derived oracle labels is only 52.98% (Sec. IV-D), and no control with gold VAD labels is run.
  • domain assumption CARE embeddings are appropriate frozen speech features
    CARE is taken as a fixed feature extractor from the authors' prior work [35] with no comparison against other speech encoders in this paper.
  • domain assumption MSP-PODCAST is a suitable unsupervised pretraining corpus with no speaker or label leakage into IEMOCAP, MELD, or CMU-MOSI
    No overlap analysis is reported; the claim of unsupervised pretraining assumes the corpora are disjoint.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations." pith.science (2026). https://pith.science/paper/KYM24DLW

@misc{pith2026250111468,
  author       = {Pith},
  title        = {Pith review of: LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KYM24DLW}},
  note         = {Machine review of arXiv:2501.11468}
}
read the original abstract

Emotion recognition in conversations (ERC) is challenging due to the multimodal nature of the emotion expression. In this paper, we propose to pretrain a text-based recognition model from unsupervised speech transcripts with LLM guidance. These transcriptions are obtained from a raw speech dataset with a pre-trained ASR system. A text LLM model is queried to provide pseudo-labels for these transcripts, and these pseudo-labeled transcripts are subsequently used for learning an utterance level text-based emotion recognition model. We use the utterance level text embeddings for emotion recognition in conversations along with speech embeddings obtained from a recently proposed pre-trained model. A hierarchical way of training the speech-text model is proposed, keeping in mind the conversational nature of the dataset. We perform experiments on three established datasets, namely, IEMOCAP, MELD, and CMU- MOSI, where we illustrate that the proposed model improves over other benchmarks and achieves state-of-the-art results on two out of these three datasets.

Figures

Figures reproduced from arXiv: 2501.11468 by the authors.

Figure 1
Figure 1. Block diagram of the proposed model. The pre-training stage is shown in the grey box at the top. An ASR system is used to generate the transcripts [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The co-attention network used in the proposed model. It consists of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The performance of the RoBERTa-large models on the different [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Using Rank-N-Contrast loss on valence-arousal rankings instead of symmetric cross-entropy improves ordinal consistency and cross-modal alignment for emotional speech and text.

  2. Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A structured review of multimodal emotion recognition in conversations, covering datasets, feature processing, methods, and open challenges, with emphasis on recent LLM-based approaches.

  3. ABHINAYA -- A System for Speech Emotion Recognition In Naturalistic Conditions Challenge

    cs.SD 2025-05 conditional novelty 4.0 of 10

    Abhinaya, an ensemble of fine-tuned SSL, SLLM, and LLM models with majority voting, achieved state-of-the-art macro-F1 (44.02%) on the Interspeech 2025 naturalistic speech emotion recognition test set.

Reference graph

Works this paper leans on

57 extracted references · 46 canonical work pages · cited by 3 Pith papers

  1. [1]

    Affective multimodal human-computer interaction,

    M. Pantic et al., “Affective multimodal human-computer interaction,” in ACM international conference on Multimedia , 2005, pp. 669–676

    work page 2005

  2. [2]

    Emotion detection and analysis on social media,

    B. Gaind et al., “Emotion detection and analysis on social media,” arXiv preprint arXiv:1901.08458, 2019

    arXiv 1901

  3. [3]

    Acoustic and lexical sentiment analysis for customer service calls,

    B. Li et al., “Acoustic and lexical sentiment analysis for customer service calls,” in ICASSP. IEEE, 2019, pp. 5876–5880

    work page 2019

  4. [4]

    EmoKey: An emotion-aware smartphone keyboard for mental health monitoring,

    S. Ghosh et al., “EmoKey: An emotion-aware smartphone keyboard for mental health monitoring,” in COMSNETS. IEEE, 2019, pp. 496–499

    work page 2019

  5. [5]

    K-EmoCon, a multimodal sensor dataset for contin- uous emotion recognition in naturalistic conversations,

    C. Y . Park et al., “K-EmoCon, a multimodal sensor dataset for contin- uous emotion recognition in naturalistic conversations,” Scientific Data, vol. 7, no. 1, p. 293, 2020

    work page 2020

  6. [6]

    Emotion recognition in conversation: Research chal- lenges, datasets, and recent advances,

    S. Poria et al. , “Emotion recognition in conversation: Research chal- lenges, datasets, and recent advances,” IEEE Access, vol. 7, pp. 100 943– 100 953, 2019

    work page 2019

  7. [7]

    Emotion recognition using facial expressions,

    P. Tarnowski et al. , “Emotion recognition using facial expressions,” Procedia Computer Science , vol. 108, pp. 1175–1184, 2017

    work page 2017

  8. [8]

    K. R. Scherer et al. , Vocal expression of emotion. Oxford University Press, 2003

    work page 2003

Show all 57 references
  1. [9]

    Individuality in communicative bodily behaviours,

    C. Navarretta, “Individuality in communicative bodily behaviours,” in Cognitive Behavioural Systems . Springer, 2012, pp. 417–423

    2012

  2. [10]

    Physiological signals and their use in augment- ing emotion recognition for human–machine interaction,

    R. B. Knapp et al. , “Physiological signals and their use in augment- ing emotion recognition for human–machine interaction,” in Emotion- oriented systems. Springer, 2011, pp. 133–159

    2011

  3. [11]

    A review of affective computing: From unimodal analysis to multimodal fusion,

    S. Poria et al., “A review of affective computing: From unimodal analysis to multimodal fusion,” Information Fusion, vol. 37, pp. 98–125, 2017

    2017

  4. [12]

    Some aspects of fundamental frequency and envelope amplitude as related to the emotional content of speech,

    P. Lieberman and S. B. Michaels, “Some aspects of fundamental frequency and envelope amplitude as related to the emotional content of speech,” The Journal of the Acoustical Society of America , vol. 34, no. 7, pp. 922–927, 1962

    1962

  5. [13]

    Emotion in speech: Recognition and application to call centers,

    V . Petrushin, “Emotion in speech: Recognition and application to call centers,” in Proceedings of artificial neural networks in engineering, vol. 710, 1999, p. 22

    1999

  6. [14]

    Speech emotion recognition using spectrogram & phoneme embedding

    P. Yenigalla et al. , “Speech emotion recognition using spectrogram & phoneme embedding.” in Interspeech, vol. 2018, 2018, pp. 3688–3692

    2018

  7. [15]

    Effective attention mechanism in dynamic models for speech emotion recognition,

    P.-W. Hsiao and C.-P. Chen, “Effective attention mechanism in dynamic models for speech emotion recognition,” in ICASSP. IEEE, 2018, pp. 2526–2530

    2018

  8. [16]

    Multimodal transformer with learnable frontend and self attention for emotion recognition,

    S. Dutta and S. Ganapathy, “Multimodal transformer with learnable frontend and self attention for emotion recognition,” in ICASSP. IEEE, 2022, pp. 6917–6921

    2022

  9. [17]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski et al. , “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in Neural Information Processing Systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  10. [18]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu et al. , “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing , vol. 29, pp. 3451–3460, 2021

    2021

  11. [19]

    WavLM: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen et al. , “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  12. [20]

    SALMONN: Towards generic hearing abilities for large language models,

    C. Tang et al., “SALMONN: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289 , 2023

    2023 arXiv

  13. [21]

    WavLLM: Towards robust and adaptive speech large language model,

    S. Hu et al. , “WavLLM: Towards robust and adaptive speech large language model,” arXiv preprint arXiv:2404.00656 , 2024

    2024 arXiv

  14. [22]

    Sentiwordnet: A publicly available lexical resource for opinion mining,

    F. Sebastiani and A. Esuli, “Sentiwordnet: A publicly available lexical resource for opinion mining,” in Proceedings of the 5th international conference on language resources and evaluation. European Language Resources Association (ELRA) Genoa, Italy, 2006, pp. 417–422

    2006

  15. [23]

    Lexicon-based methods for sentiment analysis,

    M. Taboada et al. , “Lexicon-based methods for sentiment analysis,” Computational linguistics, vol. 37, no. 2, pp. 267–307, 2011

    2011

  16. [24]

    Convolutional neural networks for sentence classification,

    Y . Kim, “Convolutional neural networks for sentence classification,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Oct. 2014, pp. 1746–1751. [Online]. Available: https: //aclantho...

    2014

  17. [25]

    Opinion mining with deep recurrent neural networks,

    O. Irsoy and C. Cardie, “Opinion mining with deep recurrent neural networks,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014, pp. 720–728

    2014

  18. [26]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in NAACL-HLT, 2019, pp. 4171–4186

    2019

  19. [27]

    Aspect-based sentiment analysis using bert,

    M. Hoang et al. , “Aspect-based sentiment analysis using bert,” in Proceedings of the 22nd nordic conference on computational linguistics, 2019, pp. 187–196

    2019

  20. [28]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu et al., “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  21. [29]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  22. [30]

    Sentiment analysis in the era of large language models: A reality check,

    W. Zhang et al., “Sentiment analysis in the era of large language models: A reality check,” arXiv preprint arXiv:2305.15005 , 2023

    2023 arXiv

  23. [31]

    HCAM–Hierarchical Cross Atten- tion Model for Multi-modal Emotion Recognition,

    S. Dutta and S. Ganapathy, “HCAM–Hierarchical Cross Atten- tion Model for Multi-modal Emotion Recognition,” arXiv preprint arXiv:2304.06910, 2023

    2023 arXiv

  24. [32]

    DialogueGCN: A Graph Convolutional Neural Net- work for Emotion Recognition in Conversation,

    D. Ghosal et al. , “DialogueGCN: A Graph Convolutional Neural Net- work for Emotion Recognition in Conversation,” in EMNLP-IJCNLP, 2019, pp. 154–164

    2019

  25. [33]

    EmoCaps: Emotion capsule based model for conversational emotion recognition,

    Z. Li et al., “EmoCaps: Emotion capsule based model for conversational emotion recognition,” in Findings of the Association for Computational Linguistics: ACL 2022 , May 2022, pp. 1610–1618

    2022

  26. [34]

    Robust speech recognition via large-scale weak supervision,

    A. Radford et al. , “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  27. [35]

    Leveraging content and acoustic rep- resentations for efficient speech emotion recognition,

    S. Dutta and S. Ganapathy, “Leveraging content and acoustic rep- resentations for efficient speech emotion recognition,” arXiv preprint arXiv:2409.05566, 2024

    2024 arXiv

  28. [36]

    Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks,

    J. Lu et al., “Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks,” Advances in neural information processing systems, vol. 32, 2019

    2019

  29. [37]

    IEMOCAP: Interactive emotional dyadic motion capture database,

    C. Busso et al. , “IEMOCAP: Interactive emotional dyadic motion capture database,” LREC, vol. 42, no. 4, pp. 335–359, 2008

    2008

  30. [38]

    MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations,

    S. Poria et al., “MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations,” in ACL, 2019, pp. 527–536

    2019

  31. [39]

    MOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,

    A. Zadeh et al. , “MOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,” arXiv preprint arXiv:1606.06259, 2016

    2016 arXiv

  32. [40]

    Is ChatGPT a good sentiment analyzer? A preliminary study,

    Z. Wang et al., “Is ChatGPT a good sentiment analyzer? A preliminary study,” arXiv preprint arXiv:2304.04339 , 2023

    2023 arXiv

  33. [41]

    Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert,

    Q. Zhong et al., “Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert,” arXiv preprint arXiv:2302.10198 , 2023

    2023 arXiv

  34. [42]

    Emotionflow: Capture the dialogue level emotion transitions,

    X. Song et al. , “Emotionflow: Capture the dialogue level emotion transitions,” in ICASSP. IEEE, 2022, pp. 8542–8546

    2022

  35. [43]

    Supervised adversarial contrastive learning for emotion recognition in conversations,

    D. Hu et al. , “Supervised adversarial contrastive learning for emotion recognition in conversations,” in ACL (Volume 1: Long Papers) , 2023, pp. 10 835–10 852

    2023

  36. [44]

    Hierarchical dialogue understanding with special tokens and turn-level attention,

    X. Liu et al., “Hierarchical dialogue understanding with special tokens and turn-level attention,” arXiv preprint arXiv:2305.00262 , 2023

    2023 arXiv

  37. [45]

    Emotion-anchored contrastive learning framework for emotion recognition in conversation,

    F. Yu et al. , “Emotion-anchored contrastive learning framework for emotion recognition in conversation,” in Findings of the Association for Computational Linguistics: NAACL 2024 , 2024, pp. 4521–4534

    2024

  38. [46]

    Multi-task self-supervised learning for robust speech recognition,

    M. Ravanelli et al. , “Multi-task self-supervised learning for robust speech recognition,” in ICASSP. IEEE, 2020, pp. 6989–6993

    2020

  39. [47]

    SUPERB: Speech Processing Universal PERfor- mance Benchmark,

    S. wen Yang et al. , “SUPERB: Speech Processing Universal PERfor- mance Benchmark,” in Interspeech 2021, 2021, pp. 1194–1198

    2021

  40. [48]

    ASR and emotional speech: A word-level investigation of the mutual impact of speech and emotion recognition,

    Y . Li et al., “ASR and emotional speech: A word-level investigation of the mutual impact of speech and emotion recognition,” arXiv preprint arXiv:2305.16065, 2023

    2023 arXiv

  41. [49]

    Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,

    R. Lotfian and C. Busso, “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,” IEEE Transactions on Affective Computing , vol. 10, no. 4, pp. 471–483, 2017

    2017

  42. [50]

    Context-dependent sentiment analysis in user-generated videos,

    S. Poria et al., “Context-dependent sentiment analysis in user-generated videos,” in ACL (volume 1: Long papers) , 2017, pp. 873–883

    2017

  43. [51]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019

    2019

  44. [52]

    Locally confined modality fusion network with a global perspective for multimodal human affective computing,

    S. Mai et al. , “Locally confined modality fusion network with a global perspective for multimodal human affective computing,” IEEE Transactions on Multimedia , vol. 22, no. 1, pp. 122–137, 2019

    2019

  45. [53]

    M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues,

    T. Mittal et al. , “M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues,” in AAAI, vol. 34, 2020, pp. 1359– 1367

    2020

  46. [54]

    DialogueTRM: Exploring multi-modal emotional dy- namics in a conversation,

    Y . Mao et al. , “DialogueTRM: Exploring multi-modal emotional dy- namics in a conversation,” in EMNLP, 2021, pp. 2694–2704

    2021

  47. [55]

    SMIN: Semi-supervised Multi-modal Interaction Net- work for Conversational Emotion Recognition,

    Z. Lian et al. , “SMIN: Semi-supervised Multi-modal Interaction Net- work for Conversational Emotion Recognition,” IEEE Transactions on Affective Computing, 2022

    2022

  48. [56]

    UniMSE: Towards unified multimodal sentiment analysis and emotion recognition,

    G. Hu et al., “UniMSE: Towards unified multimodal sentiment analysis and emotion recognition,” arXiv preprint arXiv:2211.11256 , 2022

    2022 arXiv

  49. [57]

    TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation,

    T. Yun et al. , “TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation,” in NAACL (Volume 1: Long Papers), 2024, pp. 82–95

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.