Pith. sign in

REVIEW 3 major objections 6 minor 34 references

ParaSpeechCLAP maps speech and rich text style captions into one embedding space, beating prior dual-encoders on retrieval, classification, and training-free TTS style guidance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 16:08 UTC pith:SHWJ7X7P

load-bearing objection Solid dual-encoder extension of CLAP/ParaCLAP to the full ParaSpeechCaps style taxonomy, with a clean specialized-vs-unified comparison, a useful multitask classification trick, and a practical best-of-N TTS reward use case. the 3 major comments →

arxiv 2603.28737 v2 pith:SHWJ7X7P submitted 2026-03-30 eess.AS cs.AIcs.CLcs.SD

ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining

Anuj Diwan , Eunsol Choi , David Harwath This is my paper
classification eess.AS cs.AIcs.CLcs.SD
keywords rich stylescontrastive learningdual-encoderspeech-text alignmentreward modelingstyle-prompted TTSspeech representationsattribute classification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing speech–text models only handle a narrow set of emotions, yet real speech also varies in pitch, texture, accent, clarity and other free-form styles. This paper builds dual-encoder models that place speech clips and natural-language style captions into a shared vector space covering both speaker-level (intrinsic) and utterance-level (situational) tags. Specialized models trained on each tag family are stronger on their own dimensions, while a single combined model is best when captions mix both kinds of attributes. An extra classification loss that re-uses the text encoder, plus class-balanced sampling, further helps the intrinsic model. The resulting embeddings support style caption retrieval, attribute classification, and best-of-N selection that improves style consistency of style-prompted text-to-speech without any extra training of the synthesizer.

Core claim

ParaSpeechCLAP dual-encoder models, trained with contrastive alignment on rich style captions, produce a joint speech–text embedding space that covers a far wider tag set than prior models. Specialized Intrinsic and Situational variants outperform on their respective evaluation sets; the Combined model is strongest on compositional captions that require both tag types. The Intrinsic model further benefits from a multitask classification loss and class-balanced sampling. Across style-caption retrieval, speech attribute classification, and inference-time best-of-N guidance for style-prompted TTS, these models outperform dual-encoder baselines on most metrics while leaving naturalness and intel

What carries the argument

A CLAP-style dual encoder (WavLM speech backbone + multilingual text backbone) trained with bidirectional InfoNCE; for the Intrinsic model this is joined by an inference-like classification loss that embeds class-label templates through the same text encoder so speech embeddings can be scored against tag embeddings without a separate head.

Load-bearing premise

The paper treats labels and human listening scores drawn from its own rich-style caption dataset as reliable ground truth for the full tag set, so measured gains are assumed to transfer beyond that data family.

What would settle it

Independent re-annotation or listening tests on speech outside the authors’ caption dataset that show no retrieval, classification, or TTS style-consistency gain of ParaSpeechCLAP over a strong dual-encoder baseline trained on the same data would falsify the central performance claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Style-prompted TTS systems can be steered toward more faithful output at inference time by scoring candidate samples against a style caption, without retraining the synthesizer.
  • When only intrinsic or only situational tags matter, a specialized dual-encoder is preferable; when prompts mix both, a single combined model is the better default.
  • Free-form natural-language style descriptions, not fixed emotion inventories, become usable queries for speech retrieval and multi-label classification.
  • Multitask contrastive-plus-classification training with class balancing improves coverage of rare speaker-level style tags.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Closing the remaining gap between the Combined model and the specialized models would remove the need to pick the right encoder at inference time.
  • Best-of-N guidance cost grows linearly with N; amortizing that cost via guided decoding or score-based sampling is a natural next engineering step.
  • The same shared space could be inverted for automatic style captioning or as a continuous style-consistency metric for TTS evaluation.
  • Gains rest heavily on one dataset family’s labels; multi-site re-annotation would be the cleanest test of broader transfer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces ParaSpeechCLAP, a family of dual-encoder contrastive models (Intrinsic, Situational, and Combined) that align speech with free-form rich style captions spanning 28 intrinsic and 23 situational tags from ParaSpeechCaps. Models use WavLM-Large and Granite Embedding backbones with InfoNCE; Intrinsic additionally uses a text-encoder-derived classification loss plus class-balanced sampling. The authors report gains over dual-encoder baselines (including ParaCLAP and a ParaSpeechCaps-finetuned ParaCLAP-PSC) on holdout retrieval and classification (Table 1), ablations isolating encoders/multitask/class-balancing (Table 3), and improved style consistency of a style-prompted TTS system via best-of-N selection without harming NMOS or WER (Table 2, human listening tests). Code and models are released.

Significance. If the reported gains hold, the work meaningfully expands dual-encoder speech-text models beyond the narrow emotion tag sets of prior systems (ParaCLAP, SSE) to a much richer intrinsic+situational taxonomy, and demonstrates a practical training-free use of such models as inference-time reward scorers for style-prompted TTS. Strengths include released code/models, a data-controlled baseline (ParaCLAP-PSC), component ablations for Intrinsic, and human evaluation with AB/BA order control. The specialized-vs-unified comparison is a useful empirical finding for the community. The main caveat is that nearly all evaluation lives inside the authors’ ParaSpeechCaps family, so significance for out-of-family generalization remains to be established; that is largely a field limitation given the absence of independent rich-style benchmarks covering this tag breadth.

major comments (3)
  1. §3.2 and Table 1: Nearly all retrieval/classification evaluation (and the 246-example TTS test set) is drawn from ParaSpeechCaps holdout/test partitions produced by the same group. ParaCLAP-PSC partially isolates the data contribution, but does not address label noise, caption-template artifacts, or transfer beyond this family. The paper correctly notes that no independent benchmark covers the full 28+23 taxonomy; still, at least one external or cross-corpus check (even on a reduced tag subset, e.g., standard emotion sets or VoxCeleb-style attributes) would strengthen the central claim that gains are not dataset-specific.
  2. §2.4 and §3.2 Classification Setup: Classification logits and evaluation both rely on a single fixed template (“A person is speaking in a {label} style”) plus Gemini-generated paraphrases for training. The authors acknowledge prompt sensitivity but leave analysis to future work. Because classification UAR/F1 is a primary reported application (Table 1), a short sensitivity study (alternate templates or paraphrase sets) is needed to show that the Intrinsic gains and the multitask loss benefit are not template-dependent.
  3. Table 2 / §3.2 TTS setup: Best-of-N (N=10) with ParaSpeechCLAP improves CMOS and tag recall without degrading NMOS/WER, which is a useful result. However, there is no comparison against alternative selection criteria (e.g., random selection among N, or an off-the-shelf emotion/attribute classifier as reward). Without that, it is hard to quantify how much of the gain is due to the learned joint space versus simply selecting among diverse TTS samples. A minimal control would make the reward-model claim more load-bearing.
minor comments (6)
  1. Table 1 caption and §4.2: Clarify that classification metrics are omitted for the Combined eval set because compositional captions do not map to single class labels (mentioned only in a footnote-like sentence). Make this explicit in the table caption.
  2. §2.2: State the projection-head output dimension (768) earlier and confirm whether speech mean-pooling and text CLS are L2-normalized before the cosine similarity used in InfoNCE (standard but not fully explicit).
  3. §3.1: Class-balanced sampling is applied only to Intrinsic; a one-sentence note on why the same procedure did not help Situational/Combined would aid reproducibility.
  4. Figure 1: The right-hand classification panel shows a single audio embedding dotted with class prompts; ensure the figure legend matches the multi-hot BCE formulation in §2.4.
  5. Typos / polish: Abstract and intro use both “ParaSpeechCLAP” and “PSCLAP” inconsistently in tables; standardize. “performs best-of-N” in the intro summary should be “perform best-of-N”.
  6. §5: The practical limitation that specialized models require choosing the right variant at inference is well noted; a short quantitative “oracle vs. Combined” comparison on mixed prompts would make the open challenge more concrete.

Circularity Check

0 steps flagged

No significant circularity: empirical dual-encoder training and evaluation with standard losses and held-out metrics; self-use of ParaSpeechCaps is dataset dependence, not a by-construction reduction.

full rationale

ParaSpeechCLAP is an empirical systems paper. It trains dual-encoder models with the standard bidirectional InfoNCE contrastive loss (and, for the Intrinsic variant only, an auxiliary BCE classification loss that re-uses the text encoder on templated class prompts) on the ParaSpeechCaps corpus, then measures retrieval (R@k, MedR), classification (UAR, macro-F1), and best-of-N TTS selection (human CMOS / tag recall / NMOS / WER) on held-out splits of the same corpus and against explicit controls (Random Projection, original ParaCLAP, ParaCLAP-PSC that isolates the data contribution, and VoxProfile-VQ). Ablations (Table 3) further isolate encoder choice, multitask loss, and class-balancing. None of these steps reduces a claimed prediction or first-principles result to its own inputs by definition or by an unverified self-citation chain: the losses are ordinary, the metrics are computed after training, and human listening tests supply an external check for the TTS application. Heavy reliance on the authors’ prior ParaSpeechCaps taxonomy and labels is a field limitation (no independent full-taxonomy benchmark exists) rather than internal circularity of the reported numbers. Score 0 is therefore the correct, proportionate finding.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Empirical dual-encoder paper: central claims rest on standard contrastive learning assumptions, pretrained encoder quality, reliability of ParaSpeechCaps labels (including auto-annotations), Gemini-generated class paraphrases for the intrinsic classification loss, and human MOS as style ground truth. Free parameters are ordinary training hyperparameters; no new physical entities.

free parameters (5)
  • InfoNCE temperature τ (learnable)
    Initialized to 0.07 and learned; scales the similarity matrix and affects alignment sharpness.
  • Learning rate / steps / batch size
    Fixed at 1e-5 Adam, 4500 steps, per-GPU batch 32 on 4 A40s; chosen by authors, not derived.
  • Speech clip length (10 s pad/truncate)
    Hand-chosen fixed length for all training examples.
  • Number of Gemini paraphrases per intrinsic tag (6)
    Ad hoc count of class prompts sampled each minibatch for L_classify.
  • Best-of-N candidates N=10
    Inference selection budget for TTS guidance experiments.
axioms (5)
  • domain assumption Bidirectional InfoNCE on speech-text pairs yields a useful joint style embedding space.
    Core training objective (§2.3); standard CLAP/CLIP assumption, not re-proved here.
  • domain assumption ParaSpeechCaps intrinsic/situational tags and captions are sufficiently accurate labels for training and holdout evaluation.
    All main tables use this dataset family (§3.1–3.2); includes manual and automatic annotations.
  • domain assumption WavLM-Large mean-pool and Granite CLS + projection heads are adequate backbones for style alignment.
    Encoder choice justified by SUPERB/MTEB rankings (§2.2); ablated vs older ParaCLAP encoders.
  • ad hoc to paper Cosine similarity between speech embedding and style-caption embedding is a valid reward for selecting more style-faithful TTS outputs.
    Load-bearing for the TTS guidance claim (§3.2 Inference-Time Guidance; Table 2).
  • ad hoc to paper Template 'A person is speaking in a {label} style' plus Gemini paraphrases adequately represent class semantics for classification loss and eval.
    Used for L_classify (§2.4) and classification setup; authors note prompt sensitivity left to future work.
invented entities (2)
  • ParaSpeechCLAP-Intrinsic / Situational / Combined model family independent evidence
    purpose: Shared speech-text embedding models specialized or unified over rich style tags.
    Primary artifact; evaluated empirically, not a postulated physical object.
  • Text-encoder class-embedding classification loss for intrinsic tags independent evidence
    purpose: Produce multi-label logits without a separate classification head while keeping the text encoder active.
    Methodological construct in §2.4; ablated in Table 3.

pith-pipeline@v1.1.0-grok45 · 14221 in / 3354 out tokens · 29894 ms · 2026-07-13T16:08:49.936417+00:00 · methodology

0 comments
read the original abstract

We introduce ParaSpeechCLAP, a family of dual-encoder models that map speech and text style captions into a shared embedding space, supporting rich intrinsic (speaker-level) and situational (utterance-level) descriptors, such as pitch, texture, and emotion, beyond the narrow set handled by existing models. We train separate Intrinsic and Situational models alongside a unified Combined model, finding that specialized models are stronger on individual style dimensions while the unified model excels on compositional evaluation. We further show that ParaSpeechCLAP-Intrinsic benefits from an additional classification loss and class-balanced training. We demonstrate performance on style caption retrieval, speech attribute classification, and usability as inference-time reward models for style-prompted TTS. ParaSpeechCLAP models outperform baselines on most metrics across all three applications. Our models and code are released at https://github.com/ajd12342/paraspeechclap .

Figures

Figures reproduced from arXiv: 2603.28737 by Anuj Diwan, David Harwath, Eunsol Choi.

Figure 1
Figure 1. Figure 1: An overview of the ParaSpeechCLAP dual-encoder training methodology. (Left) All ParaSpeechCLAP models are trained with an InfoNCE contrastive loss to align speech and text embeddings. (Right) For ParaSpeechCLAP-Intrinsic, this is supplemented with a classification loss, where audio embeddings are aligned with text embeddings generated from templates filled with class labels. model is similar to the Contras… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 19 linked inside Pith

  1. [1]

    Yet, real-world speech varies along many more dimensions: pitch, texture, clarity and be- yond [2]

    Introduction Existing speech-caption alignment models [1] handle only a narrow set of stylistic attributes. Yet, real-world speech varies along many more dimensions: pitch, texture, clarity and be- yond [2]. While emotion recognition has made significant progress [3–5], support for this broader set of speech styles, especially when specified as freeform n...

  2. [2]

    Our arXiv:2603.28737v1 [eess.AS] 30 Mar 2026 Audios Speech Encoder Text Style Prompts A fem ale speaker has a shrill nasal Am erican accent

    Methodology As depicted in Figure 1, ParaSpeechCLAP consists of speech and text encoders that project raw speech and text cap- tions into a common multimodal embedding space. Our arXiv:2603.28737v1 [eess.AS] 30 Mar 2026 Audios Speech Encoder Text Style Prompts A fem ale speaker has a shrill nasal Am erican accent. Text Encoder T1 T2 T3 A3A2A1 A1•T1 A2•T1 ...

    Pith/arXiv arXiv 2026

  3. [3]

    Experimental Setup 3.1. Training DatasetWe train our models on ParaSpeechCaps [2], a large- scale dataset that provides both manually and automatically an- notated style prompts for speech clips from Expresso, EARS and subsets of V oxCeleb and Emilia. We train on the intrinsic- tag and situational-tag subsets for ParaSpeechCLAP-Intrinsic and Situational r...

  4. [4]

    Baselines For TTS guidance, we compare inference with and without ap- plying ParaSpeechCLAP guidance

    Results and Discussion 4.1. Baselines For TTS guidance, we compare inference with and without ap- plying ParaSpeechCLAP guidance. For retrieval and classifica- tion: •Random Projection: Our ParaSpeechCLAP architecture with pretrained encoder weights and random projector weights. •ParaCLAP[1]: An existing speech-prompt model trained on MSP-Podcast that con...

  5. [5]

    Conclusion We introduced ParaSpeechCLAP, a dual-encoder model that creates a shared embedding space for speech and rich textual style descriptions. Through extensive experiments, we demon- strated ParaSpeechCLAP’s ability to handle a diverse range of intrinsic and situational attributes for retrieval and classifica- tion, pioneered its use as an inference...

  6. [6]

    Generative AI tools were only used for light editing, polishing and finding typos

    Generative AI Usage Disclosure The authors take full responsibility and are accountable for the contents of this paper. Generative AI tools were only used for light editing, polishing and finding typos

  7. [7]

    Paraclap – towards a general language-audio model for computational paralinguistic tasks,

    X. Jing, A. Triantafyllopoulos, and B. Schuller, “Paraclap – towards a general language-audio model for computational paralinguistic tasks,” 2024. [Online]. Available: https://arxiv.org/ abs/2406.07203

    Pith/arXiv arXiv 2024

  8. [8]

    Scaling rich style-prompted text-to-speech datasets,

    A. Diwan, Z. Zheng, D. Harwath, and E. Choi, “Scaling rich style-prompted text-to-speech datasets,” 2025. [Online]. Available: https://arxiv.org/abs/2503.04713

    arXiv 2025

  9. [9]

    emotion2vec: Self-supervised pre-training for speech emotion representation,

    Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen, “emotion2vec: Self-supervised pre-training for speech emotion representation,” 2023. [Online]. Available: https://arxiv.org/abs/2312.15185

    Pith/arXiv arXiv 2023

  10. [10]

    Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,

    C. yu Huang et. al., “Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,” 2025. [Online]. Available: https://arxiv.org/abs/2411.05361

    Pith/arXiv arXiv 2025

  11. [11]

    Secap: Speech emotion captioning with large language model,

    Y . Xu, H. Chen, J. Yu, Q. Huang, Z. Wu, S. Zhang, G. Li, Y . Luo, and R. Gu, “Secap: Speech emotion captioning with large language model,” 2023. [Online]. Available: https://arxiv.org/abs/2312.10381

    Pith/arXiv arXiv 2023

  12. [12]

    Prompttts: Con- trollable text-to-speech with text descriptions,

    Z. Guo, Y . Leng, Y . Wu, S. Zhao, and X. Tan, “Prompttts: Con- trollable text-to-speech with text descriptions,” 2022. [Online]. Available: https://arxiv.org/abs/2211.12171

    Pith/arXiv arXiv 2022

  13. [13]

    Parler-tts,

    Y . Lacombe, V . Srivastav, and S. Gandhi, “Parler-tts,” 2024. [Online]. Available: https://github.com/huggingface/parler-tts

    2024

  14. [14]

    Building naturalistic emotionally bal- anced speech corpus by retrieving emotional speech from existing podcast recordings,

    R. Lotfian and C. Busso, “Building naturalistic emotionally bal- anced speech corpus by retrieving emotional speech from existing podcast recordings,”IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, 2019

    2019

  15. [15]

    Stylecap: Automatic speaking-style captioning from speech based on speech and language self-supervised learning models,

    K. Yamauchi, Y . Ijima, and Y . Saito, “Stylecap: Automatic speaking-style captioning from speech based on speech and language self-supervised learning models,” 2023. [Online]. Available: https://arxiv.org/abs/2311.16509

    Pith/arXiv arXiv 2023

  16. [16]

    Factor-conditioned speaking-style captioning,

    A. Ando, T. Moriya, S. Horiguchi, and R. Masumura, “Factor-conditioned speaking-style captioning,” 2024. [Online]. Available: https://arxiv.org/abs/2406.18910

    Pith/arXiv arXiv 2024

  17. [17]

    Emonews: A spoken dialogue system for expressive news conversations,

    R. Matsuura, S. Bharadwaj, J. Liu, and D. K. Govindarajan, “Emonews: A spoken dialogue system for expressive news conversations,” 2025. [Online]. Available: https://arxiv.org/abs/ 2506.13894

    Pith/arXiv arXiv 2025

  18. [18]

    Ld-sds: Towards an expressive spoken dialogue system based on linked-data,

    A. Papangelis, P. Papadakos, M. Kotti, Y . Stylianou, Y . Tzitzikas, and D. Plexousakis, “Ld-sds: Towards an expressive spoken dialogue system based on linked-data,” 2017. [Online]. Available: https://arxiv.org/abs/1710.02973

    Pith/arXiv arXiv 2017

  19. [19]

    Clap: Learning audio concepts from natural language supervision,

    B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang, “Clap: Learning audio concepts from natural language supervision,”

  20. [20]

    Available: https://arxiv.org/abs/2206.04769

    [Online]. Available: https://arxiv.org/abs/2206.04769

    Pith/arXiv arXiv

  21. [21]

    Learning to summarize from human feedback,

    N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. V oss, A. Radford, D. Amodei, and P. Christiano, “Learning to summarize from human feedback,” 2022. [Online]. Available: https://arxiv.org/abs/2009.01325

    Pith/arXiv arXiv 2022

  22. [22]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  23. [23]

    Speechclip: Integrating speech with pre- trained vision and language model,

    Y .-J. Shih, H.-F. Wang, H.-J. Chang, L. Berry, H. yi Lee, and D. Harwath, “Speechclip: Integrating speech with pre- trained vision and language model,” 2022. [Online]. Available: https://arxiv.org/abs/2210.00705

    Pith/arXiv arXiv 2022

  24. [24]

    Sse: A speaking style extractor based on fine-grained contrastive learning between speech and descriptive text,

    Z. Zhang, Y . Wu, Z. Dong, W. Xiang, S. Shen, and B. W. Schuller, “Sse: A speaking style extractor based on fine-grained contrastive learning between speech and descriptive text,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  25. [25]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  26. [26]

    Superb: Speech processing universal performance benchmark,

    S. wen Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, “Superb: Speech processing universal performance benchmark,” 2021. [Online]. Available: https://arxiv.org/abs/2105.01051

    Pith/arXiv arXiv 2021

  27. [27]

    Granite embedding models,

    P. Awasthy, A. Trivedi, Y . Li, M. Bornea, D. Cox, A. Daniels, M. Franz, G. Goodhart, B. Iyer, V . Kumar, L. Lastras, S. McCarley, R. Murthy, V . P, S. Rosenthal, S. Roukos, J. Sen, S. Sharma, A. Sil, K. Soule, A. Sultan, and R. Florian, “Granite embedding models,” 2025. [Online]. Available: https://arxiv.org/abs/2502.20204

    Pith/arXiv arXiv 2025

  28. [28]

    Mteb: Massive text embedding benchmark,

    N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “Mteb: Massive text embedding benchmark,” 2023. [Online]. Available: https://arxiv.org/abs/2210.07316

    Pith/arXiv arXiv 2023

  29. [29]

    V ox-profile: A speech foundation model benchmark for characterizing diverse speaker and speech traits,

    T. Feng, J. Lee, A. Xu, Y . Lee, T. Lertpetchpun, X. Shi, H. Wang, T. Thebaud, L. Moro-Velazquez, D. Byrd, N. Dehak, and S. Narayanan, “V ox-profile: A speech foundation model benchmark for characterizing diverse speaker and speech traits,”

  30. [30]

    Available: https://arxiv.org/abs/2505.14648

    [Online]. Available: https://arxiv.org/abs/2505.14648

    Pith/arXiv arXiv

  31. [31]

    Representation learning with contrastive predictive coding,

    A. van den Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” 2019. [Online]. Available: https://arxiv.org/abs/1807.03748

    Pith/arXiv arXiv 2019

  32. [32]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    G. T. et. al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” 2025. [Online]. Available: https://arxiv.org/abs/2507.06261

    Pith/arXiv arXiv 2025

  33. [33]

    IEMOCAP: interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: interactive emotional dyadic motion capture database,”Language Resources and Evaluation, vol. 42, no. 4, pp. 335–359, Dec. 2008

    2008

  34. [34]

    The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,

    S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,”PloS one, vol. 13, no. 5, p. e0196391, 2018

    2018