Pith. sign in

REVIEW 4 major objections 5 minor 42 references

WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WhiSPA claims that aligning Whisper's audio embeddings to text-derived semantic and psychological embeddings yields a single speech encoder that can replace a two-stage speech-to-text-plus-language-model pipeline for psychological…

desk verdict Promising distillation approach, but the headline error reductions rest on partly circular evaluation and an unverified split protocol. read the letter →

arxiv 2501.16344 v4 pith:JOOR5MGB submitted 2025-01-15 eess.AS cs.AIcs.CLcs.SD

Rajath Rao , Adithya Ganesan , Oscar Kjell , Jonah Luby , Akshay Raghavan , Scott Feltman , Whitney Ringwald , Ryan L. Boyd
show 5 more authors
Benjamin Luft Camilo Ruggero Neville Ryant Roman Kotov H. Andrew Schwartz
This is my paper · ORCID
classification eess.AScs.AIcs.CLcs.SD
keywords speechrepresentationlearningcross-modalalignmentcontrastivesemanticembeddingspsychologicalassessmentWhisperstudent-teacherdistillationmentalhealthlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a speech-to-text model's internal language model can be repurposed so that a separate text language model is no longer needed for understanding the psychology of speech. The authors take Whisper's pooled decoder embeddings and align them, through a contrastive student-teacher objective, to SBERT sentence embeddings and to PsychEmb dimensions for emotion and personality. On more than 500,000 segments of clinical interview speech, the aligned model, WhiSPA, reports average error reductions of 73.4% on self-supervised affective dimensions and 83.8% on downstream psychological scales relative to existing speech encoders. Appending SBERT embeddings to WhiSPA's own representations adds almost nothing, which the authors read as evidence that the audio encoder now carries the semantic and psychological content that used to require a text model.

What carries the argument

The central mechanism is a student-teacher alignment objective: a frozen Whisper encoder-decoder is the student whose pooled decoder hidden state is projected, while SBERT (all-MiniLM-L12-v2) supplies the text teacher embeddings; in WhiSPA-394, ten standardized PsychEmb scalars are concatenated to the SBERT embedding and a learned 384-by-10 projection maps Whisper's 384-dimensional pooled output to 394 dimensions. The training loss is a noise contrastive estimation loss with temperature 0.1: each audio embedding is pulled toward its matching transcript embedding and pushed away from all other text embeddings in a batch of 900. This is what transfers the text language model's semantic and psychological geometry into audio in a self-supervised way.

What would settle it

Take audio segments with identical or near-identical transcripts but opposite affective delivery (for example, flat versus tearful, or sincere versus sarcastic) and compare WhiSPA-394's valence and arousal predictions within each transcript-matched cluster. If the predictions do not vary systematically with delivery, the claim that the audio model captures the psychological signal without a text language model is falsified for acoustic information.

Watch

Extended reading notes

Core claim

WhiSPA's central claim is that a speech encoder alone, without a downstream text language model, can carry semantic and psychological information sufficient for psychological assessment. The method aligns the mean-pooled last hidden state of Whisper's decoder to SBERT sentence embeddings, optionally enriched with ten PsychEmb dimensions (valence, arousal, Big Five traits, anger, anxiety, depression), using a noise contrastive estimation loss with Whisper as the student and the text embeddings as the teacher. On person-level ridge regression tasks over mental-health interview data, WhiSPA-394 reports average error reductions of 73.4% on ten self-supervised psychological dimensions and 83.8% on eleven downstream clinical scales relative to current speech encoders, and appending SBERT features to WhiSPA gives almost no further gain. The paper interprets this as evidence that the alignment distills the text model's knowledge into audio, closing the semantic-psychological gap between the two modalities.

Load-bearing premise

The load-bearing premise, acknowledged in the paper's limitations as a 'lexical bias', is that the whisper-large-v2 transcripts from which SBERT and PsychEmb teachers are computed already contain all the psychological content that matters, so any acoustic cue lost in transcription can never enter WhiSPA's representations.

Editorial extensions

If this is right

  • A single audio encoder can replace the Whisper-to-SBERT cascade for psychological prediction tasks on mental-health interview data, with no loss and sometimes a gain.
  • Noise contrastive estimation gives better-aligned audio representations than cosine similarity, so the choice of contrastive objective is part of the method's effectiveness.
  • The same distillation transfers to larger Whisper and SBERT configurations, with even larger error reductions at 1024-dimensional embeddings.
  • WhiSPA's representations make known PTSD-relevant language markers, such as first-person pronouns and negative-affect terms, more detectable than Whisper's, indicating that clinically meaningful linguistic content is encoded in audio.
  • Because adding SBERT features on top of WhiSPA yields only marginal gains, the paper concludes that the text teacher's information has largely been absorbed into the audio encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the same alignment works with a prosody-aware teacher, but its own arousal results and Section 7 limitation suggest a ceiling: any acoustic cue not in the transcripts is inaccessible to WhiSPA, so applications relying on tone, pauses, or voice quality would still need acoustic information.
  • A testable extension is cross-corpus evaluation: train on WTC and HiTOP interviews and evaluate on an emotion corpus with varied speaking styles to see whether the semantic alignment generalizes beyond clinical interviews.
  • The person-level averaging in the evaluation means the demonstrated gains are for aggregate clinical inference, not segment-level affect recognition; segment-level benchmarks where prosody matters more would be a harsher test of the no-text-LM claim.
  • The near-zero benefit of appending SBERT features is evidence of distillation, but it also implies WhiSPA is bounded by the teacher's lexical knowledge; a richer teacher would be needed to capture psychological content expressed non-lexically.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces WhiSPA, a method that aligns Whisper's decoder embeddings with SBERT text embeddings and PsychEmb psychological dimension embeddings using a contrastive (NCE) or cosine similarity loss. The authors claim that the resulting audio encoder surpasses standard speech encoders (Wav2Vec2-BERT, HuBERT, Whisper) on self-supervised psychological dimension prediction and downstream clinical scale prediction, with average error reductions of 73.4% and 83.8%, respectively, and that it obviates the need for a separate text language model. Training and evaluation use two mental-health interview datasets (WTC and HiTOP) comprising over 500k audio segments, with evaluation via 10-fold cross-validated ridge regression on person-level averaged embeddings.

Significance. If the results are valid, the paper would offer a practical and computationally efficient way to transfer semantic and psychological information from a text LM into an audio encoder, simplifying pipelines for psychological assessment. The training recipe is concrete and reproducible in principle, including hyperparameters, model variants, and a planned code release. However, the current evaluation protocol does not rule out participant leakage between WhiSPA training and ridge-regression test folds, and the self-supervised tasks in Table 2 are circular because they predict the same teacher values used as training targets. These issues must be resolved before the headline claims can be accepted.

major comments (4)
  1. [Section 3 and Section 5] The evaluation protocol does not establish participant-disjoint generalization. Section 3 states that after extracting PsychEmb features a single 80:10:10 (train/val/test) split is performed, while Section 5 reports 10-fold cross-validated ridge regression on person-level averaged embeddings. The paper never states whether the 80/10/10 split or the 10 folds are split by participant ID. If WhiSPA's contrastive training (Eq. 2) includes segments from the same participants who appear in the ridge test folds, the results in Tables 2, 3, and 5 may reflect memorization of speaker-specific cues rather than transfer of psychological content to new speakers. The authors should clarify the split scheme and, if it is not participant-disjoint, rerun the evaluation with a participant-disjoint protocol such as GroupKFold on speaker ID.
  2. [Table 2 and Eq. (2)] The self-supervised PsychEmb evaluation is circular. The NCE loss in Eq. (2) trains the audio embedding to match SBERT and PsychEmb teacher values, and Table 2 then regresses those same PsychEmb values on the aligned audio embeddings. High correlation is therefore a measure of fit to the teacher, not an independent demonstration that the encoder captures psychological content. Comparisons to W2V2B and HuBERT are uninformative for this objective because those models were never optimized against PsychEmb. The paper should reframe Table 2 as a sanity check or reconstruction measure, and place the weight of evidence on the downstream tasks in Tables 3 and 6.
  3. [Abstract and Section 4] The claim that 'it is not always necessary to run a subsequent text LM' is stronger than what the training setup can support. The teacher embeddings (SBERT and PsychEmb) are computed from whisper-large-v2 transcriptions (Section 4), so WhiSPA can only learn the lexical content that survives ASR; prosody, pauses, and voice quality are not available to the teacher. The limitation is acknowledged in Section 7 as 'lexical bias,' but the abstract and conclusion still present the no-text-LM result as a general finding. The authors should either restrict the claim to lexical-semantic content or provide an analysis showing that acoustic information is retained (e.g., on the human-annotated valence task with acoustic-only baselines).
  4. [Abstract and Section 5] The headline 'average error reduction' figures (73.4% for self-supervised and 83.8% for downstream) are not defined or reproducible from the tables. Section 5 reports an 83.38% reduction for the 384-to-394 comparison, but the 73.4% figure is not derived anywhere, and the baseline for each reduction is unspecified. The authors should state the exact formula, the baseline model, and the set of tasks over which each average is computed, and ensure the abstract numbers are consistent with the tables.
minor comments (5)
  1. [Table 4] The model naming is inconsistent: Table 2 uses 'WhiSPA-384r' while Table 4 uses 'WhiSPA-384' for the replacement variant; please unify the notation throughout.
  2. [Figure 5 caption] The caption states 'The distribution of WTC is shown in blue. The distribution of WTC is shown in red.' The second mention should be HiTOP.
  3. [Section 4.3] The phrase 'T anHactiivation' is a typo for 'TanH activation'.
  4. [Section 3 and Appendix A.1.1] The main text describes HiTOP as 'outpatients with psychiatric diagnoses,' while the appendix says the interviews were between 'World Trade Center responder participants and clinicians'; please clarify which population is which.
  5. [Appendix A.4] The HiTOP outcome description cites 'Roman and Meyer, 2024,' but this reference is not in the reference list and appears unrelated to the interview instrument; please correct or remove the citation.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: Table 2's self-supervised PsychEmb 'predictions' are the training objective by construction, while downstream clinical and human-annotation results remain external.

  1. fitted input called prediction [Sec. 4.3 (Eq. 2), Sec. 5 (Table 2)]
    "WhiSPA injects the PsychEmb dimensions into the SBERT embeddings under two settings: (1) with replacement ... (2) with projection: We concatenate the PsychEmb dimensions to the text embedding from SBERT. ... L_NCE = -log exp(sim(A_i,T_i)/τ) / Σ_{b∈B(i)} exp(sim(A_i,T_b)/τ). ... We evaluated the speech-based models' ability to capture the psychological dimensions of language by comparing our models' predictions to PsychEmb derived values at the segment level."

    For WhiSPA, the teacher vector T_i in Eq. 2 contains the PsychEmb dimensions (either replacing the first 10 SBERT dimensions or concatenated through a learned projection). The contrastive loss trains the audio embedding A_i to be similar to that exact teacher vector. Table 2 then evaluates the model by ridge-regressing those same PsychEmb values on the resulting audio embeddings. The evaluation target is the same signal used as the training target, so the reported self-supervised accuracy and the 73.4% average error reduction measure how well the student retained the teacher, not whether psychological content was predicted independently. The baselines (W2V2B, HuBERT, Whisper) were never given the PsychEmb teacher signal, so the comparison is favorable by construction.

full rationale

The central downstream claim is not circular: Tables 3, 5, and 6 predict external PCL/HiTOP clinical scales and 300 human-annotated valence/arousal segments, none of which were used as WhiSPA training targets. Those results give independent support for the claim that a text-LM pipeline may be unnecessary. The circularity is concentrated in Table 2, where the 'self-supervised prediction' of PsychEmb is a readout of the training teacher. A separate methodological risk, noted by the skeptic, is that the paper never states whether its 80:10:10 split or 10-fold ridge evaluation is participant-disjoint; if training and test folds share participants, the downstream numbers could be inflated by speaker-level memorization. That is a leakage/correctness concern rather than a construction-level circularity, since PCL/HiTOP labels were not part of the alignment loss. Score 6 reflects one set of headline predictions reducing to the training objective while the central claim retains independent external grounding.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the premise that text-derived teacher embeddings computed from ASR transcripts are sufficient targets for learning psychological audio representations, plus standard contrastive-learning assumptions. Hyperparameters such as temperature, batch size, and projection choices are tuned, and the self-supervised evaluation reuses the training targets.

free parameters (7)
  • Contrastive temperature tau = 0.1
    NCE temperature in Eq. 2; chosen by hand; affects contrastive sharpness.
  • Batch size = 900
    Authors state NCE efficacy depends strongly on batch size; 900 was chosen and distributed across 3 A6000 GPUs (Appendix A.2).
  • Learning rate = 1e-5
    Training hyperparameter reported in Appendix A.2.
  • Weight decay = 1e-2
    Training hyperparameter reported in Appendix A.2.
  • Number of epochs = 50
    Training hyperparameter reported in Appendix A.2.
  • WhiSPA-384r replaced dimensions = first 10 SBERT dimensions
    Authors considered correlation-based dimension selection but chose naive replacement to avoid statistical bias; an arbitrary modeling choice (Appendix A.2).
  • Projection matrix P for WhiSPA-394 = 384x10 learned projection
    Learned projection used to concatenate PsychEmb dimensions to SBERT-like space; fitted during training (Section 4.3).
assumptions (6)
  • standard math Cosine similarity and NCE contrastive loss provide valid gradients for distribution alignment
    Invoked in Section 4.1 as the training objective; no formal proof required by the paper.
  • domain assumption Text is derivable from speech, so audio can be mapped into text-derived semantic space
    Section 1: 'Still, since text is derivable from speech, speech should intrinsically be mappable to the same rich semantic features from the text.' This is the central premise.
  • domain assumption whisper-large-v2 transcriptions are accurate enough teacher inputs
    Section 4: 'audio segments are previously transcribed with whisper-large-v2... SBERT and PsychEmb representations were encoded using these transcriptions.'
  • domain assumption SBERT and PsychEmb capture semantic and psychological constructs needed for downstream tasks
    Sections 4.2 and 4.3 use them as teachers; validity rests on prior work, not revalidated here.
  • domain assumption Mean-pooling Whisper decoder hidden states and averaging segment embeddings preserves person-level psychological signal
    Sections 4 and 5; no analysis of information loss from pooling.
  • domain assumption 10-fold person-level ridge regression CV is a valid evaluation protocol for the frozen encoder
    Section 5; the paper does not clarify whether WhiSPA training split excludes CV test participants.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learning." pith.science (2026). https://pith.science/paper/JOOR5MGB

@misc{pith2026250116344,
  author       = {Pith},
  title        = {Pith review of: WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JOOR5MGB}},
  note         = {Machine review of arXiv:2501.16344}
}
read the original abstract

Current speech encoding pipelines often rely on an additional text-based LM to get robust representations of human communication, even though SotA speech-to-text models often have a LM within. This work proposes an approach to improve the LM within an audio model such that the subsequent text-LM is unnecessary. We introduce WhiSPA (Whisper with Semantic and Psychological Alignment), which leverages a novel audio training objective: contrastive loss with a language model embedding as a teacher. Using over 500k speech segments from mental health audio interviews, we evaluate the utility of aligning Whisper's latent space with semantic representations from a text autoencoder (SBERT) and lexically derived embeddings of basic psychological dimensions: emotion and personality. Over self-supervised affective tasks and downstream psychological tasks, WhiSPA surpasses current speech encoders, achieving an average error reduction of 73.4% and 83.8%, respectively. WhiSPA demonstrates that it is not always necessary to run a subsequent text LM on speech-to-text output in order to get a rich psychological representation of human communication.

Figures

Figures reproduced from arXiv: 2501.16344 by the authors.

Figure 1
Figure 1. Speech processing pipelines that are further [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Diagram of WhiSA and WhiSPA training procedure involving a student-teacher model paradigm. Whisper [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Bivariate KDE contour plot of PCA dimen [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: WhiSPA Bridges the Semantic/Psychological Representation Gap. WhiSPA consistently outperforms every baseline audio model and, in most cases, matches or exceeds the performance of the text-based language model teacher. based teacher model, SBERT. (2) The marginal return…
Figure 5
Figure 5. Figure 5: Standardized distributions of [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Distributions of psychological features stan [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Pearson r correlation heatmap of SBERT￾384’s mean embedding. This visual displays the corre￾lations of SBERT’s 384 dimensions with each of the 10 PsychEmb dimensions. ases when training, and so we naively replaced the first 10 dimensions. One should note that the set o…
Figure 8
Figure 8. Figure 8: Annotator’s affective circumplex visual grid [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 20 canonical work pages

  1. [1]

    Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. https://arxiv.org/abs/2006.11477 wav2vec 2.0: A framework for self-supervised learning of speech representations . Preprint, arXiv:2006.11477

    arXiv 2020

  2. [2]

    Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. 2022. https://arxiv.org/abs/2202.01374 mslam: Massively multilingual joint pre-training for speech and text . Preprint, arXiv:2202.01374

    arXiv 2022

  3. [3]

    E B Blanchard, J Jones-Alexander, T C Buckley, and C A Forneris. 1996. Psychometric properties of the PTSD checklist ( PCL ). Behav. Res. Ther., 34(8):669--673

    work page 1996

  4. [4]

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP : interactive emotional dyadic motion capture database. Lang. Resour. Eval., 42(4):335--359

    work page 2008

  5. [5]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. https://arxiv.org/abs/2002.05709 A simple framework for contrastive learning of visual representations . Preprint, arXiv:2002.05709

    arXiv 2020

  6. [6]

    Yanbei Chen, Yongqin Xian, A Koepke, Ying Shan, and Zeynep Akata. 2021. Distilling audio-visual knowledge by compositional contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7016--7025

    work page 2021

  7. [7]

    Yining Chen, Jianqiang Li, Changwei Song, Qing Zhao, Yongsheng Tong, and Guanghui Fu. 2024. https://arxiv.org/abs/2409.06164 Deep learning and large language models for audio and text analysis in predicting suicidal acts in chinese psychological support hotlines . Preprint, arXiv:2409.06164

    arXiv 2024

  8. [8]

    Yung-Sung Chuang, Chi-Liang Liu, Hung-Yi Lee, and Lin shan Lee. 2020. https://arxiv.org/abs/1910.11559 Speechbert: An audio-and-text jointly learned language model for end-to-end spoken question answering . Preprint, arXiv:1910.11559

    arXiv 2020

Show all 42 references
  1. [9]

    Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. https://arxiv.org/abs/2108.06209 W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training . Preprint, arXiv:2108.06209

    2021 arXiv

  2. [10]

    Clark and Michael F

    Herbert H. Clark and Michael F. Schober. 1992. Asking questions and influencing answers

    1992

  3. [11]

    Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li...

    2023 arXiv

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    2019 arXiv

  5. [13]

    Jingjing Dong, Jiayi Fu, Peng Zhou, Hao Li, and Xiaorui Wang. 2022. Improving spoken language understanding with cross-modal contrastive learning. In Interspeech, pages 2693--2697

    2022

  6. [14]

    Yumeng Fu. 2024. https://arxiv.org/abs/2403.07260 Ckerc : Joint large language models with commonsense knowledge for emotion recognition in conversation . Preprint, arXiv:2403.07260

    2024 arXiv

  7. [15]

    Yue Gu, Xinyu Li, Shuhong Chen, Jianyu Zhang, and Ivan Marsic. 2017. Speech intention classification with multimodal deep learning. Adv. Artif. Intell., 10233:260--271

    2017

  8. [16]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . Preprint, arXiv:1503.02531

    2015 arXiv

  9. [17]

    Dirk Hovy and Diyi Yang. 2021. https://doi.org/10.18653/v1/2021.naacl-main.49 The importance of modeling social factors of language: Theory and practice . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Huma...

    2021 doi

  10. [18]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. https://arxiv.org/abs/2106.07447 Hubert: Self-supervised speech representation learning by masked prediction of hidden units . Preprint, arXiv:2106.07447

    2021 arXiv

  11. [19]

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2021. https://arxiv.org/abs/2004.11362 Supervised contrastive learning . Preprint, arXiv:2004.11362

    2021 arXiv

  12. [20]

    Schwartz

    Oscar Kjell, Adithya V Ganesan, Ryan Boyd, Joshua Oltmanns, Alfredo Rivero, Scott Feltman, Melissa Carr, Benjamin Luft, Roman Kotov, and H. Schwartz. 2024. https://doi.org/10.31234/osf.io/xw24e Demonstrating high validity of a new ai-language assessment of ptsd: A sequential e...

    2024 doi

  13. [21]

    Roman Kotov, David C Cicero, Christopher C Conway, Colin G DeYoung, Alexandre Dombrovski, Nicholas R Eaton, Michael B First, Miriam K Forbes, Steven E Hyman, Katherine G Jonas, Robert F Krueger, Robert D Latzman, James J Li, Brady D Nelson, Darrel A Regier, Craig Rodriguez-Sei...

    2022

  14. [22]

    Roman Kotov, Holly Frances Levin-Aspenson, Camilo Ruggero, Holly Levin-Aspenson, and Katherine Jonas. 2024. https://doi.org/10.17605/osf.io/u25em Interview for the hierarchical taxonomy of psychopathology ( iHiTOP )

    2024 doi

  15. [23]

    Jehyun Kyung, Serin Heo, and Joon-Hyuk Chang. 2024. Enhancing multimodal emotion recognition through asr error compensation and llm fine-tuning. In Proc. Interspeech 2024, pages 4683--4687

    2024

  16. [24]

    May Jorella Lazaro, Sungho Kim, Jaeyong Lee, Jaemin Chun, Gyungbhin Kim, EunJeong Yang, Aigerim Bilyalova, and Myung Yun. 2021. https://doi.org/10.1007/978-3-030-78462-1_15 A review of multimodal interaction in intelligent systems

    2021 doi

  17. [25]

    Hailun Lian, Cheng Lu, Sunan Li, Yan Zhao, Chuangao Tang, and Yuan Zong. 2023. https://doi.org/10.3390/e25101440 A survey of deep learning-based multimodal emotion recognition: Speech, text, and face . Entropy, 25(10)

    2023 doi

  18. [26]

    Daniel M Low, Kate H Bentley, and Satrajit S Ghosh. 2020. Automated assessment of psychiatric disorders using speech: A systematic review. Laryngoscope Investig. Otolaryngol., 5(1):96--116

    2020

  19. [27]

    Martin Lukac. 2024. Speech-based personality prediction using deep learning with acoustic and linguistic embeddings. Sci. Rep., 14(1):30149

    2024

  20. [28]

    Schwartz, Johannes Eichstaedt, Margaret Kern, Michal Kosinski, David Stillwell, Lyle Ungar, and Martin Seligman

    Gregory Park, H. Schwartz, Johannes Eichstaedt, Margaret Kern, Michal Kosinski, David Stillwell, Lyle Ungar, and Martin Seligman. 2014. https://doi.org/10.1037/pspp0000020 Automatic personality assessment through social media language . Journal of personality and social psycho...

    2014 doi

  21. [29]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. https://arxiv.org/abs/2212.04356 Robust speech recognition via large-scale weak supervision . Preprint, arXiv:2212.04356

    2022 arXiv

  22. [30]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . Preprint, arXiv:1908.10084

    2019 arXiv

  23. [31]

    Claire Roman and Philippe Meyer. 2024. https://aclanthology.org/2024.lt4hala-1.12 Analysis of glyph and writing system similarities using S iamese neural networks . In Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ L...

    2024

  24. [32]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. https://arxiv.org/abs/1910.01108 Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter . Preprint, arXiv:1910.01108

    2020 arXiv

  25. [33]

    Giuseppe Sartori and Graziella Orrù. 2023. https://doi.org/10.3389/fpsyg.2023.1279317 Language models and psychological sciences . Frontiers in Psychology, 14

    2023

  26. [34]

    Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. https://arxiv.org/abs/1904.05862 wav2vec: Unsupervised pre-training for speech recognition . Preprint, arXiv:1904.05862

    2019 arXiv

  27. [35]

    Schwartz

    Nikita Soni, Matthew Matero, Niranjan Balasubramanian, and H. Schwartz. 2022. https://doi.org/10.18653/v1/2022.findings-acl.52 Human language modeling . In Findings of the Association for Computational Linguistics: ACL 2022, page 622–636. Association for Computational Linguistics

    2022 doi

  28. [36]

    Andrew Schwartz, Jo \ a o Sedoc, and Niranjan Balasubramanian

    Nikita Soni, H. Andrew Schwartz, Jo \ a o Sedoc, and Niranjan Balasubramanian. 2024. https://doi.org/10.18653/v1/2024.naacl-long.477 Large human language models: A need and the challenges . In Proceedings of the 2024 Conference of the North American Chapter of the Association ...

    2024 doi

  29. [37]

    Andrew Schwartz

    Adithya V Ganesan, Vasudha Varadarajan, Juhi Mittal, Shashanka Subrahmanya, Matthew Matero, Nikita Soni, Sharath Chandra Guntuku, Johannes Eichstaedt, and H. Andrew Schwartz. 2022. https://doi.org/10.18653/v1/2022.clpsych-1.25 WWBP - SQT -lite: Multi-level models and differenc...

    2022 doi

  30. [38]

    Zehui Wu, Ziwei Gong, Lin Ai, Pengyuan Shi, Kaan Donbekci, and Julia Hirschberg. 2024. https://arxiv.org/abs/2407.21315 Beyond silent letters: Amplifying llms in emotion recognition with vocal nuances . Preprint, arXiv:2407.21315

    2024 arXiv

  31. [39]

    Hao Yang, Jinming Zhao, Gholamreza Haffari, and Ehsan Shareghi. 2023. https://arxiv.org/abs/2305.17733 Investigating pre-trained audio encoders in the low-resource condition . Preprint, arXiv:2305.17733

    2023 arXiv

  32. [40]

    Rong Ye, Mingxuan Wang, and Lei Li. 2022. https://arxiv.org/abs/2205.02444 Cross-modal contrastive learning for speech translation . Preprint, arXiv:2205.02444

    2022 arXiv

  33. [41]

    Chuan Zhang, Daoxin Zhang, Ruixiu Zhang, Jiawei Li, and Jianke Zhu. 2023. https://arxiv.org/abs/2302.01555 Bridging the emotional semantic gap via multimodal relevance estimation . Preprint, arXiv:2302.01555

    2023 arXiv

  34. [42]

    Zihan Zhao, Yanfeng Wang, and Yu Wang. 2022. https://arxiv.org/abs/2207.04697 Multi-level fusion of wav2vec 2.0 and bert for multimodal emotion recognition . Preprint, arXiv:2207.04697

    2022 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.