Pith. sign in

REVIEW 5 major objections 5 minor 42 references

CLEP-DG: Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Text-prompt tuning of CLAP yields emotion recognition that generalizes across acoustic domains.

desk verdict A plausible new recipe for CLAP-based SER, but the key cross-modal classifier claim is untested and the prompt length is tuned on held-out data. read the letter →

arxiv 2507.04048 v1 pith:EWICAWTP submitted 2025-07-05 cs.SD eess.AS

Jiacheng Shi , Yanfu Zhang , Ye Gao This is my paper
classification cs.SDeess.AS
keywords speechemotionrecognitioncontrastivelanguage-audiopretrainingdomaingeneralizationsoftprompttuningcross-modaltransferabilityacousticcontextaugmentationArcFacelossCLAP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a contrastively aligned language–audio model can generalize to unseen acoustic conditions for speech emotion recognition without collecting audio from those conditions. Its recipe is to fine-tune CLAP on large emotional speech corpora, then learn small text prompts that describe acoustic contexts such as studio, talk show, or podcast, and finally train the emotion classifier on text embeddings alone while applying it to audio embeddings during inference. Across five benchmarks, it reports that this combination beats prior CLAP-based systems both in supervised settings and on three datasets unseen during training. A sympathetic reader would care because the approach converts an audio-labeling problem into a text-writing problem, and it relies on an alignment property, cross-modal transferability, that if real has broader uses.

What carries the argument

The load-bearing object is the shared CLAP embedding space, in which audio and text are brought together by contrastive pretraining and then by emotion-speech fine-tuning. On top of that space, Acoustic Context Prompt Tuning learns soundscape-conditioned prompts $V=[v_1,\dots,v_8,\text{CLASS}]$ and optimizes them with a softmax classification loss plus a ranking loss, so each emotion's text embedding also encodes acoustic context. The final decision rule is a linear classifier trained with ArcFace on text-derived embeddings and applied to $\ell^2$-normalized audio embeddings at inference. The mechanism works only if cosine similarity in that joint space transfers a text-only boundary to audio.

What would settle it

Train the identical linear classifier on audio embeddings with the same ArcFace loss and epochs, and compare with the text-trained version on the five benchmarks: if the text-trained classifier is not competitive on unseen domains, cross-modal transferability is not carrying the result. Alternatively, replace the audio side of the shared embedding space with a random projection while keeping the text side intact; a large accuracy drop would confirm the audio-text alignment is essential, while a small drop would reveal the gains come from the prompts alone.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that textual descriptions of acoustic contexts are a sufficient proxy for acoustic domain diversity inside a contrastive audio–text model. Fine-tuning CLAP on roughly 96,000 emotional utterances yields CLEP, an emotion-aware encoder; Acoustic Context Prompt Tuning optimizes eight learnable tokens per soundscape so that class-wise prompt embeddings separate emotions under varied recording conditions; and a linear classifier with ArcFace margin, trained on those text embeddings, is then reused on audio embeddings. The reported consequence is state-of-the-art weighted accuracy on IEMOCAP and MELD and 2.22 to 3.61 percentage-point gains over the strongest CLAP baseline on unseen RAVDESS, TESS, and SAVEE. The contribution is a new application of cross-modal transferability: the decision boundary is learned in text space and executed in audio space.

Load-bearing premise

The entire pipeline leans on the assumption that CLAP's text and audio embeddings are aligned closely enough for a classifier trained only on text to separate audio emotion embeddings correctly; the paper relies on prior citations for this and does not validate it on emotional speech.

Editorial extensions

If this is right

  • Fine-tuning the audio encoder while freezing the text encoder preserves broad language understanding and contributes the largest single gain in the ablation, from 23.89% to 72.91% weighted accuracy on IEMOCAP.
  • Adding Acoustic Context Prompt Tuning on top of fine-tuning improves average weighted accuracy by another 2.65 percentage points, showing that text-only augmentation can substitute for labeled audio in unseen domains.
  • An ArcFace-trained cross-modal classifier beats a softmax-trained one by about 1.05 points on average, so the angular margin matters for the audio-side decision boundary.
  • The system needs no audio labels from the target domain, only knowledge of the target emotion classes and text descriptions of acoustic contexts.
  • Longer prompt sequences help up to 8 tokens and then plateau, giving a practical recipe for prompt length in this setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors justify cross-modal transferability only by citing prior work; a direct test would be to train the same classifier on audio embeddings and compare, since the paper never reports that condition.
  • If the transferability assumption holds, the same text-to-audio trick could be applied to other paralinguistic judgments such as speaker traits, cognitive load, or health cues, because the bottleneck becomes prompt diversity rather than audio collection.
  • Because inference uses only the audio encoder and the linear head, the text encoder and prompt optimizer can be discarded at deployment, an efficiency consequence the paper does not state.
  • The 12 soundscapes are chosen to overlap with pretraining conditions, so an untested extension is whether prompts for rare or synthetic environments such as extreme reverberation would generalize or saturate; that experiment would isolate whether prompt diversity, not prompt count, drives the domain-generalization gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CLEP-DG, a framework built on LAION CLAP for speech emotion recognition. It fine-tunes CLAP's audio encoder on four emotion-labeled datasets (IEMOCAP, MELD, MEAD, CMU-MOSEI), introduces Acoustic Context Prompt Tuning (ACPT) with learnable soundscape-conditioned prompts for the frozen text encoder, and then trains a linear classifier on text-derived embeddings under ArcFace loss before applying it to audio embeddings at inference. Experiments on IEMOCAP and MELD report supervised gains over CompA-CLAP, and DG gains on RAVDESS, TESS, and SAVEE. The authors claim state-of-the-art performance in both supervised and domain generalization settings.

Significance. If the central claims are correct, the paper makes a useful contribution: it shows that prompt-based text augmentation can improve CLAP-based SER and that a text-trained classifier can transfer to audio embeddings for emotion classification. The idea of using textual soundscape descriptions as a cheap proxy for acoustic domain diversity is attractive, and the paper ships a concrete recipe with ablations. However, the current evidence is incomplete: the cross-modal classifier transfer is not validated against a same-feature control, the prompt length is selected using the held-out DG datasets, and the baseline set is narrow. These issues affect the strength of the state-of-the-art claim rather than the basic soundness of the method.

major comments (5)
  1. [3.5] The cross-modal transferability assumption is load-bearing but unvalidated. The paper trains a linear classifier solely on text-derived embeddings produced by the CLEP text encoder and then applies it to audio embeddings at inference. No control experiment trains the same linear classifier on audio embeddings from the CLEP audio encoder. Without that control, the reported gains cannot be attributed to the text-trained decision boundary rather than to fine-tuning plus ACPT. The citations [10-13] are from vision-language domain generalization and are not verified here for emotional speech; in particular, [10] is a co-author paper and should be supplemented with direct alignment statistics or an ablation.
  2. [Figure 2 and Table 2] The prompt length Np is selected using the held-out DG datasets RAVDESS, TESS, and SAVEE, and the same datasets are then used to report DG improvements in Table 2. Since the figure shows accuracy improvement versus the fine-tuned baseline for each token count, choosing Np=8 based on those curves leaks test-set information into the method selection. The DG gains of 2.22-3.61 percentage points are therefore not fully blind. Np should be selected on a validation split from the source domains, or the DG results should be reported for a fixed Np chosen without touching the target datasets.
  3. [Section 4.4, Table 3] The text states that 'ACPT further improves CLEP-DG's accuracy by 28.5%' when combined with fine-tuning. This is inconsistent with Table 3: comparing the 'Fine-tune + ACPT' row with the 'Fine-tune only' row gives +2.27 on IEMOCAP, +1.84 on MELD, +3.73 on RAVDESS, +2.61 on TESS, +3.81 on SAVEE, and +2.65 on average, all in percentage points. The 28.5% figure appears to be a typo or a miscomputed relative change; it should be corrected and the computation stated explicitly.
  4. [Table 2 and Section 4.1] The state-of-the-art claim is not fully supported because the comparison omits the CLAP-based competitors named in the paper (ParaCLAP, GEmo-CLAP, HuBERT-CLAP) and any non-CLAP SER state of the art. The paper states that direct comparisons are infeasible because code is not released, but it still claims 'state-of-the-art performance' in the abstract and conclusion. Either the claim should be scoped to the implemented baselines, or the paper should report published numbers for the omitted methods where available. In addition, no standard deviations are reported despite the statement that all results are averaged over three random seeds; without variance information, the small DG gaps (e.g., 2.22 pp on TESS) may not be significant.
  5. [Section 4.2] The number and identity of the 12 soundscapes is not specified. The paper says 'We select 12 soundscapes aligned with conditions shared by CLAP's pretraining and our fine-tuning data' but never lists them. Since ACPT is a central contribution and the soundscape vocabulary is a free design choice, the full list (e.g., 'studio', 'podcast', 'talk show') must be provided for reproducibility and to assess whether the choice biases the DG results.
minor comments (5)
  1. [Throughout] There are repeated formatting artifacts: 'RA VDESS', 'SA VEE', 'W A', 'LACP Tloss', and 'T able' appear as broken tokens in several places. These should be cleaned before submission.
  2. [References] Wav2CLIP appears as both [17] and [26], and AudioCLIP appears as both [18] and [27]; these duplicate entries should be merged or disambiguated.
  3. [Section 4.4, Table 3] The ACPT-only ablation is applied to the original CLAP model, not to the fine-tuned CLEP model. An additional row showing ACPT applied on top of CLEP without fine-tuning would isolate the contribution of ACPT more cleanly.
  4. [Figure 2] The figure shows improvement relative to the fine-tuned baseline but no error bars or per-seed spread. Given that all other results are claimed to be averaged over three random seeds, the prompt-length comparison should include variance information.
  5. [Section 4.3] The sentence 'All reported results are averaged over three runs with different random seeds' appears in the text but Table 2 does not report the individual runs or standard deviations; a reproducibility statement with code and seed values would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central results are measured against external audio benchmarks and no derivation step reduces to its own inputs by construction.

full rationale

The paper's derivation chain is not circular. CLEP is obtained by fine-tuning CLAP on emotional speech datasets (Sec. 3.3) using a standard contrastive objective, and the reported gains are evaluated on RAVDESS, TESS, and SAVEE, which are excluded from fine-tuning (Sec. 4.1). ACPT (Sec. 3.4) trains learnable prompt vectors on text descriptions with classification and ranking losses; it does not fit any parameter to the target DG accuracy. The cross-modal classifier (Sec. 3.5) is trained on text-derived embeddings and applied to audio embeddings, but this is an empirical assumption about CLAP's alignment rather than a quantity fitted to the evaluation results; the paper also cites external works [11-13] for the phenomenon, so the presence of a co-author citation [10] is not the sole load-bearing support. The prompt-length selection reported in Fig. 2 is a test-set-leakage / hyperparameter-selection concern, not a construction-level circularity: the reported accuracies are still genuine classifier outputs rather than re-expressions of the selected hyperparameter. No equation is defined in terms of the quantity it purports to predict, and no fitted parameter is renamed as a prediction. Hence the paper is self-contained against external benchmarks and should not be flagged for circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The model introduces no new physical entities or new modalities. The load-bearing inputs are hand-chosen soundscapes, a test-tuned prompt length, and the cross-modal transferability assumption. These are standard machine learning choices, but the paper does not justify their scope or report all values.

free parameters (4)
  • soundscape vocabulary (12 contexts) = not enumerated in text (examples: studio, talk show, podcast in Fig. 1)
    Hand-chosen acoustic contexts used to create text prompts; the set is asserted to match CLAP pretraining and fine-tuning data (Sec. 3.4, 4.2) and is never justified quantitatively.
  • prompt length N_p = 8
    Selected by sweeping 2, 4, 8, 16, 32 on all five datasets including held-out DG sets (Fig. 2), so the value is tuned with test-set influence.
  • temperature scaling omega in Eq. (1) = not reported
    Scales cosine similarities in the prompt-tuning classification loss; no value is given in the text.
  • ArcFace margin = not reported
    Additive angular margin in classifier training (Sec. 3.5); the exact margin value is never stated.
assumptions (4)
  • domain assumption Cross-modal transferability: a linear classifier trained on CLAP text embeddings will transfer to CLAP audio embeddings.
    Central premise of Sec. 3.5; supported only by citations [10-13], not by an in-paper validation.
  • domain assumption Hand-picked 12 soundscapes cover the unseen test acoustic conditions.
    The prompts are chosen to overlap with pretraining and fine-tuning conditions (Sec. 4.2); no evidence is given that this set spans the held-out domains.
  • domain assumption Emotion label sets from IEMOCAP (4 classes), MELD (7), MEAD (8), and CMU-MOSEI (7) can be jointly fine-tuned without explicit label mapping.
    Fine-tuning mixes 96,147 utterances with different class taxonomies (Tab. 1); the paper never specifies label unification, yet contrastive training pairs each audio with a single 'This is [EMOTION] sound' text.
  • domain assumption CLEP's embedding space preserves emotion cues after fine-tuning only the audio encoder with a frozen text encoder.
    Fine-tuning design in Sec. 3.3 assumes the frozen text encoder can match emotion text to tuned audio representations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLEP-DG: Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning." pith.science (2026). https://pith.science/paper/EWICAWTP

@misc{pith2026250704048,
  author       = {Pith},
  title        = {Pith review of: CLEP-DG: Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EWICAWTP}},
  note         = {Machine review of arXiv:2507.04048}
}
read the original abstract

Speech Emotion Recognition (SER) is fundamental to affective computing and human-computer interaction, yet existing models struggle to generalize across diverse acoustic conditions. While Contrastive Language-Audio Pretraining (CLAP) provides strong multimodal alignment, it lacks dedicated mechanisms for capturing emotional cues, making it suboptimal for SER. To address this, we propose CLEP-DG, a framework that enhances CLAP's robustness in emotion recognition. First, we fine-tune CLAP to obtain CLEP, adapting it on large-scale emotional speech datasets to better encode emotion-relevant features. Then, we introduce Acoustic Context Prompt Tuning (ACPT), a text-driven augmentation strategy that optimizes learnable prompt vectors to model diverse acoustic environments without additional labeled audio. Finally, leveraging cross-modal transferability, we train a classifier on text-derived embeddings and apply it to the audio encoder during inference, mitigating domain shifts between textual supervision and audio-based emotion recognition. Experiments across five benchmark datasets show that CLEP-DG outperforms prior CLAP-based approaches, achieving state-of-the-art performance in both supervised and domain generalization settings.

Figures

Figures reproduced from arXiv: 2507.04048 by the authors.

Figure 1
Figure 1. Illustration of the soft-prompt tuning, training a clas￾sifier and inference stage of our method. features hinders their ability to capture fine-grained emotional expressions. CLAP (Laion-AI) [4] and CLAP (Microsoft) [3] leverage large-scale datasets for multimodal learning but re￾main primarily optimized for speech content rather than affec￾tive understanding. CompA-CLAP [19] introduces composi￾tional prompts to im… view at source ↗
Figure 2
Figure 2. Performance improvement with different numbers of prompt tokens compared to the fine-tuned baseline. lights the effectiveness of text-driven augmentation in improv￾ing generalization across varied recording conditions. The length of the learnable prompt [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 31 canonical work pages

  1. [10]

    ParaCLAP -- Towards a general language-audio model for computational paralinguistic tasks

    X. Jing, A. Triantafyllopoulos, and B. Schuller, “Paraclap– towards a general language-audio model for computational par- alinguistic tasks,” arXiv preprint arXiv:2406.07203, 2024

    work page Pith review arXiv 2024

  2. [1]

    this is a sound of

    Introduction Speech Emotion Recognition (SER) classifies emotional states from speech and acoustic expressions, serving as a corner- stone for affective computing [1] and human-computer inter- action [2]. Recent advances in Contrastive Language-Audio Pretraining (CLAP) [3, 4] have demonstrated strong general- ization by aligning text and audio in a shared...

    work page

  3. [2]

    Large-scale pretrained mod- els like Whisper [14], WavLM [15], and HuBERT [16] en- hance SER robustness by leveraging extensive speech data

    Related Work Domain Generalization in SER. Large-scale pretrained mod- els like Whisper [14], WavLM [15], and HuBERT [16] en- hance SER robustness by leveraging extensive speech data. While effective, they rely on supervised learning and lack ex- plicit modeling of acoustic variability, limiting adaptability to unseen domains. Audio-Language Pretraining i...

    arXiv 2025

  4. [3]

    Happy” or “Sad

    Proposed Method 3.1. Preliminary The zero-shot classification in CLAP is performed by comput- ing the similarity between an audio embedding and text-based class embeddings derived from a prompt template. Given an au- dio input xi, its clip-level embedding f (xi) ∈ RD is extracted using the CLAP audio encoder f, where D represents the joint embedding space...

    work page

  5. [4]

    excited” and “happy

    Experiments 4.1. Datasets and Model Architecture. Datasets. To adapt CLAP for emotion recognition, we fine-tune it on multiple large-scale emotional speech datasets, as shown in Tab. 1. Specifically, we utilize IEMOCAP [22], MELD [23], MEAD [24], and CMU-MOSEI [25], covering a diverse range of speakers, recording environments, and conversational con- text...

    work page

  6. [5]

    Integrating Acoustic Context Prompt Tuning (ACPT), CLEP-DG improves general- ization across diverse acoustic environments without additional labeled speech data

    Conclusion We present CLEP-DG, a CLAP-based framework that enhances text-to-emotion audio alignment for SER. Integrating Acoustic Context Prompt Tuning (ACPT), CLEP-DG improves general- ization across diverse acoustic environments without additional labeled speech data. Leveraging cross-modal transferability, it bridges textual supervision and audio-based...

    work page

  7. [6]

    Affective computing: A review,

    J. Tao and T. Tan, “Affective computing: A review,” in Interna- tional Conference on Affective computing and intelligent interac- tion. Springer, 2005, pp. 981–995

    work page 2005

  8. [7]

    Preece, Y

    J. Preece, Y . Rogers, H. Sharp, D. Benyon, S. Holland, and T. Carey,Human-computer interaction. Addison-Wesley Long- man Ltd., 1994

    work page 1994

Show all 42 references
  1. [8]

    Clap learning audio concepts from natural language supervision,

    B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  2. [9]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (I...

    2023

  3. [11]

    Gemo-clap: Gender-attribute-enhanced contrastive language- audio pretraining for speech emotion recognition,

    Y . Pan, Y . Hu, Y . Yang, J. Yao, W. Fei, L. Ma, and H. Lu, “Gemo-clap: Gender-attribute-enhanced contrastive language- audio pretraining for speech emotion recognition,” arXiv preprint arXiv:2306.07848, 2023

    2023 arXiv

  4. [12]

    Cross-modal features interaction-and-aggregation network with self-consistency train- ing for speech emotion recognition,

    Y . Hu, H. Yang, H. Huang, and L. He, “Cross-modal features interaction-and-aggregation network with self-consistency train- ing for speech emotion recognition,” in Proc. Interspeech 2024, 2024, pp. 2335–2339

    2024

  5. [13]

    Conditional prompt learning for vision-language models,

    K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Conditional prompt learning for vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2022

    2022

  6. [14]

    Learning to prompt for vision-language models,

    ——, “Learning to prompt for vision-language models,” Interna- tional Journal of Computer Vision (IJCV), 2022

    2022

  7. [15]

    Diagnosing and rectifying vision models using lan- guage,

    Y . Zhang, J. Z. HaoChen, S.-C. Huang, K.-C. Wang, J. Zou, and S. Yeung, “Diagnosing and rectifying vision models using lan- guage,” arXiv preprint arXiv:2302.04269, 2023

    2023 arXiv

  8. [16]

    Using language to ex- tend to unseen domains

    L. Dunlap, C. Mohri, D. Guillory, H. Zhang, T. Darrell, J. E. Gon- zalez, A. Raghunathan, and A. Rohrbach, “Using language to ex- tend to unseen domains.” International Conference on Learning Representations (ICLR), 2023

    2023

  9. [17]

    Promptstyler: Prompt-driven style generation for source-free domain generaliza- tion,

    J. Cho, G. Nam, S. Kim, H. Yang, and S. Kwak, “Promptstyler: Prompt-driven style generation for source-free domain generaliza- tion,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 702–15 712

    2023

  10. [18]

    Dpstyler: dynamic prompt- styler for source-free domain generalization,

    Y . Tang, Y . Wan, L. Qi, and X. Geng, “Dpstyler: dynamic prompt- styler for source-free domain generalization,” IEEE Transactions on Multimedia, 2025

    2025

  11. [19]

    Whisper: Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu et al. , “Whisper: Robust speech recognition via large-scale weak supervision,” inProceed- ings of the Advances in Neural Information Processing Systems (NeurIPS), 2022, pp. 9855–9867

    2022

  12. [20]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  13. [21]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  14. [22]

    Wav2clip: Learning robust audio representations via contrastive learning,

    S. R. Wu, A. Jansen, S. Pancoast, and W.-N. Hsu, “Wav2clip: Learning robust audio representations via contrastive learning,” Proceedings of INTERSPEECH, 2023

    2023

  15. [23]

    Audioclip: Ex- tending clip to audio for zero-shot learning,

    N. Gontier, A. Avila, Z. Wang, and K. Garcia, “Audioclip: Ex- tending clip to audio for zero-shot learning,” IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, vol. 31, pp. 1234–1245, 2023

    2023

  16. [24]

    Compa-clap: Composi- tional prompts for improving audio-text alignment,

    G. Chen, M. Dong, Y . Wu, and W. Wang, “Compa-clap: Composi- tional prompts for improving audio-text alignment,” Proceedings of ICASSP, 2024

    2024

  17. [25]

    Deep convo- lutional ranking for multilabel image annotation,

    Y . Gong, Y . Jia, T. Leung, A. Toshev, and S. Ioffe, “Deep convo- lutional ranking for multilabel image annotation,” arXiv preprint arXiv:1312.4894, 2013

    2013 arXiv

  18. [26]

    Arcface: Additive angular margin loss for deep face recognition,

    J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, 2019, pp. 4690–4699

    2019

  19. [27]

    IEMOCAP: In- teractive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. Narayanan, “IEMOCAP: In- teractive emotional dyadic motion capture database,” in Proceed- ings of the International Conference on Language Resources and Evaluation (LREC), 2008

    2008

  20. [28]

    MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations,

    S. Poria, D. Hazarika, N. Majumder, R. Mihalcea, A. Gelbukh, and E. Cambria, “MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations,” inProceedings of the As- sociation for Computational Linguistics (ACL), 2019

    2019

  21. [29]

    MEAD: A Large-Scale Audio-Visual Dataset for Affective Un- derstanding and Emotional Expression Analysis,

    H. Wang, Z. Wang, D. He, J. Wang, H. Xue, S. Han, and T. Mei, “MEAD: A Large-Scale Audio-Visual Dataset for Affective Un- derstanding and Emotional Expression Analysis,” IEEE Transac- tions on Multimedia, 2020

    2020

  22. [30]

    CMU-MOSEI: A Dataset for Multimodal Sentiment Analysis and Emotion Recognition,

    A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “CMU-MOSEI: A Dataset for Multimodal Sentiment Analysis and Emotion Recognition,” in Proceedings of the Association for Computational Linguistics (ACL), 2018

    2018

  23. [31]

    Wav2clip: Learning robust audio representations from clip,

    H.-H. Wu, P. Seetharaman, K. Kumar, and J. P. Bello, “Wav2clip: Learning robust audio representations from clip,” in ICASSP. IEEE, 2022

    2022

  24. [32]

    Audioclip: Extending clip to image, text and audio,

    A. Guzhov, F. Raue, J. Gall, and B. Edin, “Audioclip: Extending clip to image, text and audio,” arXiv preprint arXiv:2106.13043, 2021

    2021 arXiv

  25. [33]

    Compa: Addressing the gap in composi- tional reasoning in audio-language models,

    S. Ghosh, A. Seth, S. Kumar, U. Tyagi, C. K. Evuru, S. Ramaneswaran, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha, “Compa: Addressing the gap in composi- tional reasoning in audio-language models,” arXiv preprint arXiv:2310.08753, 2023

    arXiv

  26. [34]

    The Ryerson Audio-Visual Database of Emotional Speech and Song (RA VDESS),

    S. R. Livingstone and F. A. Russo, “The Ryerson Audio-Visual Database of Emotional Speech and Song (RA VDESS),” PLoS ONE, vol. 13, no. 5, 2018

    2018

  27. [35]

    Toronto emotional speech set (tess)-younger talker happy,

    K. Dupuis and M. K. Pichora-Fuller, “Toronto emotional speech set (tess)-younger talker happy,” 2010

    2010

  28. [36]

    SA VEE: Surrey Audio-Visual Expressed Emotion,

    P. Jackson and S. Haq, “SA VEE: Surrey Audio-Visual Expressed Emotion,” in Proceedings of the International Conference on Af- fective Computing and Intelligent Interaction (ACII), 2014

    2014

  29. [37]

    DST: Deformable Speech Transformer for Emotion Recognition,

    W. Chen, X. Xing, X. Xu, J. Pang, and L. Du, “DST: Deformable Speech Transformer for Emotion Recognition,” inProceedings of the IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), 2023

    2023

  30. [38]

    Tem- poral modeling matters: A novel temporal emotional modeling approach,

    J. Ye, X.-C. Wen, Y . Wei, Y . Xu, K. Liu, and H. Shan, “Tem- poral modeling matters: A novel temporal emotional modeling approach,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

    2023

  31. [39]

    The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,

    S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,” PloS one, vol. 13, no. 5, p. e0196391, 2018

    2018

  32. [40]

    Speaker-dependent audio- visual emotion recognition

    S. Haq, P. J. Jackson, and J. Edge, “Speaker-dependent audio- visual emotion recognition.” inAVSP, vol. 2009, 2009, pp. 53–58

    2009

  33. [41]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumb- ley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020

    2020

  34. [42]

    Roberta: A robustly optimized bert pretraining ap- proach,

    Y . Liu, “Roberta: A robustly optimized bert pretraining ap- proach,” arXiv preprint arXiv:1907.11692, vol. 364, 2019

    1907 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.