NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech

Daria Diatlova; Egor Spirin; Maksim Borisov

REVIEW 5 major objections 6 minor 41 references

NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 17-hour open dataset brings laughter, coughs, and sighs under text-to-speech control.

desk verdict Useful public NV-TTS dataset with a solid pipeline; the headline parity claim is undercut by a circular automatic metric and an underpowered human test. read the letter →

arxiv 2507.13155 v1 pith:2J6MDGB7 submitted 2025-07-17 cs.LG cs.SD

Maksim Borisov , Egor Spirin , Daria Diatlova This is my paper

classification cs.LGcs.SD

keywords nonverbalvocalizationtext-to-speechexpressivespeechsynthesisdatasetannotationemotionclassificationzero-shotTTSaudioeventdetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces NonverbalTTS (NVTTS), a 17-hour open-access English corpus built by relabeling segments of VoxCeleb and Expresso with 10 types of nonverbal vocalizations (breath, laugh, sniff, cough, throat clearing, sigh, groan, sneeze, snore, grunt) and 8 emotion categories. It argues that the scarcity of such open data is the main bottleneck for expressive speech synthesis, and that its annotation pipeline—automated NV detection, emotion classification, forced alignment, human validation, and majority-vote fusion—produces text-aligned labels good enough to train controllable NV-capable TTS. The central empirical claim is that fine-tuning the open-source CosyVoice model on NVTTS reaches parity with the closed-source CosyVoice2 on speaker similarity, NV fidelity, and human preference. A reader should care because, if this holds, a public dataset can substitute for proprietary training data in a capability previously dominated by closed systems.

What carries the argument

The load-bearing machinery is a four-stage dataset annotation pipeline. BEATs detects 10 NV types at a low threshold, the Montreal Forced Aligner places detected events into word-level alignments, emotion2vec+ assigns 8 emotion labels, and after human annotators correct transcriptions and tags on the Argilla platform, a Pyalign-based merge and majority-vote algorithm fuses multiple annotator revisions into a single text target with bracketed NV tags. This pipeline is what turns noisy, unannotated or weakly annotated speech into training targets that carry positional control over NVs, and its outputs are also used to define the NV Jaccard-distance evaluation metric.

What would settle it

Take the 359-sample test set, transcribe which NVs actually occur by human listening, then score the NVTTS model and CosyVoice2 against those human references with an independent detector; if the parity advantage shrinks or reverses while the original Jaccard scores stay high, the claimed parity is an artifact of shared detector bias.

Watch

Extended reading notes

Core claim

The paper's central claim is that NVTTS is a viable public training resource for zero-shot TTS with controllable nonverbal vocalizations. On a 359-sample test set drawn from held-out VoxCeleb speakers, a CosyVoice-300M model fine-tuned on NVTTS matches CosyVoice2—a system trained on proprietary data—on NV fidelity, speaker similarity, and intelligibility, with human evaluators showing no statistically significant preference (CosyVoice2 35.4% vs NVTTS 33.4%, Wilson 95% CI [30.4, 40.6]). The ablation shows that removing NV tags from the training transcripts degrades NV detection on generated audio, while removing emotion labels does not, supporting the conclusion that explicit NV annotations are the load-bearing ingredient.

Load-bearing premise

The whole argument assumes the automated NV detector, after human validation, is accurate enough to serve as ground truth for both training targets and the NV-fidelity metric—and since the same detector creates the test references and scores the generated audio, a systematic detector bias could make both training and evaluation look better than they are.

Editorial extensions

If this is right

Open-source TTS systems can acquire breath, laughter, cough, and sigh generation from NVTTS instead of proprietary data.
NVTTS gives the field a common public benchmark for measuring NV fidelity and speaker similarity in zero-shot expressive TTS.
Because the pipeline is described and the data released, other unannotated or weakly annotated speech corpora can be enriched for NVs the same way.
The ablation result predicts that future NV-capable TTS efforts should spend annotation effort on NV tags before emotion labels.
Emotion-tag removal slightly improves NV metrics, suggesting emotion and NV annotations interact in ways the paper does not fully resolve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the same BEATs detector creates the reference NV labels and scores the generated audio, the NV-fidelity numbers could be inflated by shared detector bias; an independent detector or human transcription of NVs on the test set would separate model quality from label bias.
With only 16 sneezes, 13 snores, and 7 grunts in the whole dataset, parity claims rest mainly on breath, laughter, cough, and sigh; extending the claim to the rare NV classes would require targeted collection or augmentation.
The pipeline's dependence on an ASR model for transcription means that languages or registers where ASR is weak will inherit transcription errors into the NV tags; applying the same method to lower-resource languages would need an ASR quality gate.
One testable extension is to use NVTTS as a seed to retrieve NV-rich segments from larger unlabeled corpora, in the direction the paper names as future work.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know about this paper because it is one of the few open NV-TTS datasets that ships human-validated labels and a reproducible pipeline. The dataset itself is the contribution: 17 hours of VoxCeleb and Expresso audio, annotated with 10 NV types and 8 emotions, plus a fusion algorithm that merges multiple annotator transcriptions via majority vote. That is a real resource, and the ablation study is honest about what the NV tags contribute. Credit where due: the human-validated labels are a step up from purely automatic detection, and the authors release the dataset and annotation guidelines.

The soft spot is the headline claim. The paper says fine-tuning on NVTTS achieves parity with CosyVoice2, but the support is thinner than the abstract suggests. The main NV fidelity metrics, J, Jbreath, Jlaugh, Jcough, are computed by running BEATs on both reference and generated audio. The training labels were also created with BEATs, then human-validated. So the evaluation references are re-detected with the same detector that shaped the training targets. Any systematic BEATs bias—say, consistently missing quiet coughs or labeling breathy consonants as breathing—is shared between training and evaluation, which can inflate the apparent fidelity of the NVTTS-trained model relative to CosyVoice2, which was trained on different targets. The human preference test is too underpowered to break the tie: 3 annotators, p>0.05, and confidence intervals that comfortably overlap. So the parity claim is not established, though it is also not refuted. The automatic metrics without error bars make this worse.

Other weaknesses are addressable: the test set is almost entirely Neutral and Happy, so emotion generation is barely evaluated; there is no baseline trained on prior open NV datasets, so we cannot tell how much NVTTS adds beyond, say, Expresso's own laughter and breath tags; and the paper withholds code and model checkpoints, which limits reproducibility for a resource paper.

None of this kills the dataset. The construction pipeline is transparent, the human validation is a genuine quality control step, and the release fills a real gap. The paper deserves a serious referee, but the referee should push for a non-circular evaluation of NV fidelity—ideally using the human-validated test labels or a different detector—and for error bars on every automatic metric.

For a TTS researcher who needs NV-annotated training data, this is worth citing and worth trying. For a reviewer, treat the dataset as the contribution and the CosyVoice2 parity claim as unproven. Send it to peer review with a request for revision.

Referee Report

5 major / 6 minor

Summary. The paper introduces NonverbalTTS (NVTTS), a 17-hour English dataset of text-aligned nonverbal vocalizations built by filtering VoxCeleb and Expresso with an automated NV detector, an emotion classifier, and human annotation refinement followed by a majority-vote fusion algorithm. The authors fine-tune CosyVoice-300M on NVTTS with and without emotion/NV tags and report automatic metrics and a small human preference test that they interpret as parity with the closed-source CosyVoice2 system. They release the dataset and annotation guidelines, arguing that NVTTS addresses the scarcity of open NV-enriched TTS corpora.

Significance. If the central parity claim holds, NVTTS is a genuinely useful public resource: it provides 17 hours of transcribed speech with 10 NV types and 8 emotion labels, a reproducible annotation pipeline, and a concrete demonstration that an open model can be fine-tuned to produce breath, laughter, cough, and sigh while remaining competitive with CosyVoice2. The release of the dataset and guidelines is a real contribution to an area where proprietary data dominate. The main risk is the evaluation: the NV fidelity metric is computed with the same BEATs detector that produced the training labels, the human preference test is underpowered, and the test set covers only a subset of the announced NVs and emotions. These are addressable with additional analysis and do not by themselves invalidate the dataset contribution.

major comments (5)

[§3.1 and §5.2 (NV detection and NV Jaccard distance)] The central parity claim rests on NV fidelity metrics computed with the same BEATs detector used to create the training labels. Section 3.1 sets the detection threshold at 0.1 and uses BEATs detections (after human validation) as NV tags; Section 5.2 then computes Jaccard references by running BEATs on the reference audio, bypassing the human-validated labels. Any systematic BEATs error pattern (breath-like consonants counted as breath, missed quiet coughs, etc.) is therefore present both in the training target and in the evaluation metric, which can inflate the apparent fidelity of the NVTTS-trained model relative to a model trained on different proprietary targets. Please report agreement between BEATs-based J scores and human NV judgments on a sample, or evaluate with an independent NV detector; the same shared-detector concern applies to EMO-SIM, which uses emotion2vec+, the same classifier that produced the emotion labels in Section 3.2.
[§5.3.3 (Human evaluation)] The human preference test uses 3 annotators on 359 samples and concludes parity from p>0.05. With the reported Wilson interval [30.4, 40.6] for CosyVoice2's 35.4% preference rate, the data are consistent with a wide range of true preference values; absence of a significant difference is not evidence of equivalence. Please report equivalence bounds, per-annotator agreement, and a larger annotation pool, or qualify the parity claim as 'no significant difference was detected' rather than 'parity'.
[§5.1 and Table 8 (baseline definition)] Table 8 lists 'cosyvoice2-full' and 'cosyvoice2-no-emotion', but Section 5.1 states that all trained models are based on CosyVoice-300M and the human study compares against 'the original CosyVoice2 model'. It is unclear whether these two rows are pretrained CosyVoice2 checkpoints or CosyVoice-300M variants fine-tuned on NVTTS, and what 'no-emotion' means for a model not trained on NVTTS. Specify the exact checkpoint and conditioning used for each row; otherwise the parity comparison is not interpretable.
[§5.2, Tables 5 and 7 (test set coverage)] The evaluation set is strongly skewed: of 359 test samples, emotion tags are almost exclusively Neutral and Happy, and NV counts are 305 breath, 56 laugh, 29 cough, and 4 sigh, with zero samples for the remaining seven NV types. This supports only a narrow parity claim for four NV types and mainly Neutral/Happy emotion; the abstract's wording about 'comprehensive nonverbal capabilities' should be qualified to reflect the actual test coverage.
[§5.3.1 and Table 8 (error bars)] All automatic metrics in Table 8 are reported without confidence intervals or significance tests. The differences that carry the parity claim (e.g., J=0.80 for NVTTS-no-emotion vs J=0.78 for cosyvoice2-no-emotion, Jbreath 0.92 vs 0.89) are small and may be within noise. Please report bootstrap confidence intervals or pairwise significance tests for the automatic metrics, especially J and its per-NV variants.

minor comments (6)

[§3.4, Algorithm 1 and Tables 3–4] The example is inconsistent with the claim in Algorithm 1 line 7 that 'cs and ct are equal' whenever both are non-gap; the merged m3 contains 'dog cat' and 'sofa' from different annotators. Please clarify that m3 is intentionally a superset and explain how mismatching words are handled during alignment and voting.
[Table 1 vs Table 5] The filtered counts in Table 1 (1,804 Expresso + 4,452 VoxCeleb = 6,256) do not match the sum of train/dev/test in Table 5 (3,642 + 46 + 359 = 4,047). Reconcile this discrepancy, likely by clarifying which filtering stage each table describes.
[§5.2 (WER setup)] State the Whisper model variant, decoding parameters, and text normalization used for WER computation; leaving these unspecified makes the 0.18–0.22 WER values hard to reproduce.
[Figure 1] The figure lacks axis labels and the exact percentages behind the bars; include them for reproducibility and to support the reported Wilson interval.
[References and typography] The BEATs citation [19] should be checked against the exact model/checkpoint used, and the manuscript contains formatting artifacts such as 'V oxCeleb' and 'CosyV oice' that should be fixed throughout.
[Dataset release] For a dataset paper, state the licenses of the underlying VoxCeleb and Expresso corpora and the license of the derived annotations; this is essential for users deciding whether to adopt NVTTS.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claim is an empirical dataset contribution benchmarked against an external system.

full rationale

The paper makes no formal derivation or first-principles prediction; its contribution is a new dataset and an empirical comparison after fine-tuning an open-source TTS model. The central parity claim is assessed against CosyVoice2, an external closed-source system trained on proprietary data, so the comparison has independent content. The only structural concern is that the NV-fidelity metrics (J, Jbreath, Jlaugh, Jcough) are computed with the BEATs detector, which was also used to propose candidate NV labels in the annotation pipeline. This shared-detector design is a metric-validity caveat, not a circular reduction: the dataset targets were human-validated, the evaluation re-detects reference audio instead of using the fitted labels, and the paper also reports external speaker-similarity, WER, DNSMOS, and human preference evidence. No self-citation chain, uniqueness import, ansatz smuggling, or renaming of a known result is present. Under the requirement to exhibit a specific equation-level or definition-level reduction, no circular step can be identified.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

This is a dataset construction paper, so the ledger records modeling choices and tool assumptions rather than derived quantities. The only hand-set parameters are the detection threshold and the fusion majority rule; no new theoretical entities are introduced.

free parameters (2)

BEATs NV detection threshold = 0.1
Set to the lowest recommended level from the BEATs paper to maximize recall; directly controls which segments enter the dataset and shapes the NV distribution.
Annotation fusion majority threshold = 2 of 3 annotators
Design choice in the majority-vote fusion algorithm; a letter or NV tag is kept only if at least two of three annotators include it, affecting final label quality.

assumptions (3)

domain assumption Canary ASR produces sufficiently accurate English transcriptions for VoxCeleb audio to support forced alignment and NV tag placement.
Section 3.1 relies on Canary-generated transcriptions because VoxCeleb has none; if transcription errors are systematic, NV tag positions and the text itself are wrong.
domain assumption The 10 chosen NV classes cover the nonverbal events relevant for expressive TTS.
Section 3.1 selects classes from AudioSet based on ontology membership and frequency; events outside this taxonomy are not annotated, potentially missing relevant sounds.
domain assumption BEATs sound-event detections, after human validation, are accurate enough to serve as ground truth.
The pipeline in Section 3.1 uses BEATs detections and Section 3.3 relies on annotators to fix errors; no independent acoustic verification or inter-annotator agreement stats for NV tags are reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech." pith.science (2026). https://pith.science/paper/2J6MDGB7

@misc{pith2026250713155,
  author       = {Pith},
  title        = {Pith review of: NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2J6MDGB7}},
  note         = {Machine review of arXiv:2507.13155}
}

read the original abstract

Current expressive speech synthesis models are constrained by the limited availability of open-source datasets containing diverse nonverbal vocalizations (NVs). In this work, we introduce NonverbalTTS (NVTTS), a 17-hour open-access dataset annotated with 10 types of NVs (e.g., laughter, coughs) and 8 emotional categories. The dataset is derived from popular sources, VoxCeleb and Expresso, using automated detection followed by human validation. We propose a comprehensive pipeline that integrates automatic speech recognition (ASR), NV tagging, emotion classification, and a fusion algorithm to merge transcriptions from multiple annotators. Fine-tuning open-source text-to-speech (TTS) models on the NVTTS dataset achieves parity with closed-source systems such as CosyVoice2, as measured by both human evaluation and automatic metrics, including speaker similarity and NV fidelity. By releasing NVTTS and its accompanying annotation guidelines, we address a key bottleneck in expressive TTS research. The dataset is available at https://huggingface.co/datasets/deepvk/NonverbalTTS.

Figures

Figures reproduced from arXiv: 2507.13155 by the authors.

**Figure 1.** Human evaluation results comparing the nonverbal generation capabilities of CosyVoice (trained on the NVTTS dataset) and CosyVoice2 indicate that the NVTTS-trained model achieves performance comparable to CosyVoice2, the current state-of-the-art method. 5. Experiments This section presents experiments using the NVTTS dataset to train zero-shot TTS models with NV synthesis capability. 5.1. Training and Experimental S… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 28 canonical work pages

[1]

Introduction Expressivity has become a critical component of naturalness in modern text-to-speech (TTS) systems, driving growing inter- est in emotional and prosody-aware speech synthesis. In addi- tion to fine-grained prosodic variation (e.g., pitch, rhythm, in- tensity), nonverbal vocalizations (NVs) such as laughter, sighs, and coughs play a crucial ro...

work page
[2]

Datasets for NV-TTS Datasets containing NVs can be categorized into three cate- gories

Related work 2.1. Datasets for NV-TTS Datasets containing NVs can be categorized into three cate- gories. In the first category, the AMI Meeting Corpus [9] offers 100 hours of meeting recordings with word-level transcriptions, including annotations for laughter and coughs from mostly non- native English speakers. Switchboard [10] contains 2,400 tele- phon...

arXiv 2025
[3]

Dataset Annotation Pipeline In this section, we describe the general data processing pipeline which we applied to the V oxCeleb [7] and Expresso [8] corpora to construct the NonverbalTTS corpus. The proposed pipeline consists of four main steps: (1) detecting nonverbal vocaliza- tions, (2) identifying emotions, (3) human annotation refine- ment, and (4) a...

work page
[4]

This section provides an overview of the original data sources and statistics on NVs and emotion tags in the resulting dataset

NVTTS Dataset NVTTS results from annotating and filtering existing V oxCeleb 1 [24], V oxCeleb 2 [25], and Expresso [8] corpora. This section provides an overview of the original data sources and statistics on NVs and emotion tags in the resulting dataset. 4.1. Data Sources The V oxCeleb dataset was originally designed for speaker ver- ification tasks, ho...

work page
[5]

Merge all annotator hypotheses into a single comprehensive version m3

work page
[6]

Align each individual annotator’s hypothesis with the merged version, receiving a1, a2, a3

work page
[7]

V ote on each letter and NV tag to create the final annotation ttarget. Given the example of initial weak transcription ti, the an- notators’ hypotheses t1, t2, t3: • ti - It’s dog [laugh] on the mat • t1 - It’s a cat [laugh] on the mat • t2 - It’s a cat [laugh] on the sofa • t3 - It’s a cat [sigh] on the mat We apply the merge algorithm 1 to create a fin...

work page
[9]

Experiments This section presents experiments using the NVTTS dataset to train zero-shot TTS models with NV synthesis capability. 5.1. Training and Experimental Setup All our experiments are based on the CosyV oice-300M [3] model. We train only the language model component [3] of the full architecture in a supervised fine-tuning setup. We use the Adam opt...

work page

Show all 41 references

[10]

Conclusion This work addresses a critical bottleneck in expressive speech synthesis – the lack of high-quality, open-source datasets with diverse NVs. To this end, we introduce NVTTS, a 17-hour corpus enriched with 10 NV types and 8 emotion categories, providing a solid founda...
[11]

Psychosocial correlates of interpersonal sensitivity: A meta-analysis,

J. A. Hall, S. A. Andrzejewski, and J. E. Yopchick, “Psychosocial correlates of interpersonal sensitivity: A meta-analysis,” Journal of nonverbal behavior, vol. 33, pp. 149–180, 2009

2009
[12]

Assessing the ability to recognize facial and vocal expressions of emotion: Construction and vali- dation of the emotion recognition index,

K. R. Scherer and U. Scherer, “Assessing the ability to recognize facial and vocal expressions of emotion: Construction and vali- dation of the emotion recognition index,” Journal of Nonverbal Behavior, vol. 35, pp. 305–326, 2011

2011
[13]

Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Ma et al. , “Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,” arXiv preprint arXiv:2407.05407, 2024

2024 arXiv
[14]

H.-H. Guo, K. Liu, F.-Y . Shen, Y .-C. Wu, F.-L. Xie, K. Xie, and K.-T. Xu, “Fireredtts: A foundation text-to-speech framework 2We calculate EMO-SIM on emotional subset of test (i.e without neutral and other) for industry-level generative speech applications,” arXiv preprint a...

2024 arXiv
[15]

Laugh now cry later: Controlling time-varying emotional states of flow- matching-based zero-shot text-to-speech,

H. Wu, X. Wang, S. E. Eskimez, M. Thakker, D. Tompkins, C.-H. Tsai, C. Li, Z. Xiao, S. Zhao, J. Li et al. , “Laugh now cry later: Controlling time-varying emotional states of flow- matching-based zero-shot text-to-speech,” in 2024 IEEE Spoken Language Technology Workshop (SLT)...

2024
[16]

Cosyvoice 2: Scalable stream- ing speech synthesis with large language models,

Z. Du, Y . Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y . Yang, C. Gao, H. Wanget al., “Cosyvoice 2: Scalable stream- ing speech synthesis with large language models,” arXiv preprint arXiv:2412.10117, 2024

2024 arXiv
[17]

V oxceleb: Large-scale speaker verification in the wild,

A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “V oxceleb: Large-scale speaker verification in the wild,”Computer Speech & Language, vol. 60, p. 101027, 2020

2020
[18]

Open automatic speech recognition leaderboard,

V . Srivastav, S. Majumdar, N. Koluguri, A. Moumen, S. Gandhi et al. , “Open automatic speech recognition leaderboard,” https: //huggingface.co/spaces/hf-audio/open asr leaderboard, 2023

2023
[19]

The ami meeting corpus,

W. Kraaij, T. Hain, M. Lincoln, and W. Post, “The ami meeting corpus,” in Proc. International Conference on Methods and Tech- niques in Behavioral Research, 2005, pp. 1–4

2005
[20]

Switchboard: Telephone speech corpus for research and development,

J. J. Godfrey, E. C. Holliman, and J. McDaniel, “Switchboard: Telephone speech corpus for research and development,” in Acoustics, speech, and signal processing, ieee international con- ference on, vol. 1. IEEE Computer Society, 1992, pp. 517–520

1992
[21]

Leveraging both the alignment and timestamps provided by the event detec- tion model, we accurately placed the non-verbal vocalizations within the transcription

to align the audio with its transcription. Leveraging both the alignment and timestamps provided by the event detec- tion model, we accurately placed the non-verbal vocalizations within the transcription. 3.2. Emotion Detection We classified all audio samples into 8 emotion ca...
[22]

The fisher corpus: A resource for the next generations of speech-to-text

C. Cieri, D. Miller, and K. Walker, “The fisher corpus: A resource for the next generations of speech-to-text.” inLREC, vol. 4, 2004, pp. 69–71

2004
[23]

Jvnv: A corpus of japanese emotional speech with verbal content and nonverbal expressions,

D. Xin, J. Jiang, S. Takamichi, Y . Saito, A. Aizawa, and H. Saruwatari, “Jvnv: A corpus of japanese emotional speech with verbal content and nonverbal expressions,” IEEE Access, vol. 12, pp. 19 752–19 764, 2024

2024
[24]

Naturalistic emotional speech collection paradigm with online game and its psychological and acoustical assessment,

Y . Arimoto, H. Kawatsu, S. Ohno, and H. Iida, “Naturalistic emotional speech collection paradigm with online game and its psychological and acoustical assessment,” Acoustical science and technology, vol. 33, no. 6, pp. 359–369, 2012

2012
[25]

Expresso: A benchmark and analysis of discrete expressive speech resynthesis,

T. A. Nguyen, W.-N. Hsu, A. D’Avirro, B. Shi, I. Gat, M. Fazel-Zarani, T. Remez, J. Copet, G. Synnaeve, M. Hassid, F. Kreuk, Y . Adi, and E. Dupoux, “Expresso: A benchmark and analysis of discrete expressive speech resynthesis,” 2023. [Online]. Available: https://arxiv.org/abs...

2023 arXiv
[26]

Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,

H. He, Z. Shang, C. Wang, X. Li, Y . Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi, Y . Wang, K. Chen, P. Zhang, and Z. Wu, “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” 2024. [Online]. Available: https://arxiv.org/abs/2407.05361

2024 arXiv
[27]

Nsv-tts: Non-speech vocalization modeling and transfer in emotional text-to-speech,

H. Zhang, X. Yu, and Y . Lin, “Nsv-tts: Non-speech vocalization modeling and transfer in emotional text-to-speech,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

2023
[28]

NeMo: a toolkit for Conversational AI and Large Language Models

E. Harper, S. Majumdar, O. Kuchaiev, L. Jason, Y . Zhang, E. Bakhturina, V . Noroozi, S. Subramanian, K. Nithin, H. Jocelyn, F. Jia, J. Balam, X. Yang, M. Livne, Y . Dong, S. Naren, and B. Ginsburg, “NeMo: a toolkit for Conversational AI and Large Language Models.” [Online]. A...
[29]

Effective pre-training of audio transformers for sound event detection,

F. Schmid, T. Morocutti, F. Foscarin, J. Schl ¨uter, P. Primus, and G. Widmer, “Effective pre-training of audio transformers for sound event detection,” arXiv preprint arXiv:2409.09546, 2024

2024 arXiv
[30]

Audio set: An ontology and human-labeled dataset for audio events,

J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE inter- national conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017,...

2017
[31]

Montreal forced aligner: Trainable text-speech align- ment using kaldi

M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Son- deregger, “Montreal forced aligner: Trainable text-speech align- ment using kaldi.” in Interspeech, vol. 2017, 2017, pp. 498–502

2017
[32]

emotion2vec: Self-supervised pre-training for speech emotion representation,

Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen, “emotion2vec: Self-supervised pre-training for speech emotion representation,” Proc. ACL 2024 Findings, 2024

2024
[33]

Argilla - Open-source framework for data-centric NLP,

V .-S. Daniel and A. Francisco, “Argilla - Open-source framework for data-centric NLP,” Jan. 2023. [Online]. Available: https: //github.com/argilla-io/argilla

2023
[34]

V oxceleb: A large-scale speaker identification dataset,

A. Nagrani, J. S. Chung, and A. Zisserman, “V oxceleb: A large-scale speaker identification dataset,” in Interspeech 2017 . ISCA, aug 2017. [Online]. Available: http://dx.doi.org/10.21437/ Interspeech.2017-950

2017
[35]

V oxceleb2: Deep speaker recognition,

J. S. Chung, A. Nagrani, and A. Zisserman, “V oxceleb2: Deep speaker recognition,” in Interspeech 2018 . ISCA, sep 2018. [Online]. Available: http://dx.doi.org/10.21437/ Interspeech.2018-1929

2018
[36]

Scaling rich style-prompted text-to-speech datasets,

A. Diwan, Z. Zheng, D. Harwath, and E. Choi, “Scaling rich style-prompted text-to-speech datasets,” 2025. [Online]. Available: https://arxiv.org/abs/2503.04713

2025
[37]

Optimizing speech emotion recognition with ma- chine learning based advanced audio cue analysis,

N. Pallewela, D. Alahakoon, A. Adikari, J. E. Pierce, and M. L. Rose, “Optimizing speech emotion recognition with ma- chine learning based advanced audio cue analysis,” Technologies, vol. 12, no. 7, p. 111, 2024

2024
[38]

Adam: A method for stochastic optimization,

K. Diederik, “Adam: A method for stochastic optimization,” (No Title), 2014

2014
[39]

Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022

2022
[40]

Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise sup- pressors,

C. K. Reddy, V . Gopal, and R. Cutler, “Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise sup- pressors,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 6493–6497

2021
[41]

Robust speech recognition via large- scale weak supervision,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large- scale weak supervision,” 2022. [Online]. Available: https: //arxiv.org/abs/2212.04356

2022 arXiv
[42]

SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,

P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, ˙I. Po- lat, Y . F...

2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Introduction Expressivity has become a critical component of naturalness in modern text-to-speech (TTS) systems, driving growing inter- est in emotional and prosody-aware speech synthesis. In addi- tion to fine-grained prosodic variation (e.g., pitch, rhythm, in- tensity), nonverbal vocalizations (NVs) such as laughter, sighs, and coughs play a crucial ro...

work page

[2] [2]

Datasets for NV-TTS Datasets containing NVs can be categorized into three cate- gories

Related work 2.1. Datasets for NV-TTS Datasets containing NVs can be categorized into three cate- gories. In the first category, the AMI Meeting Corpus [9] offers 100 hours of meeting recordings with word-level transcriptions, including annotations for laughter and coughs from mostly non- native English speakers. Switchboard [10] contains 2,400 tele- phon...

arXiv 2025

[3] [3]

Dataset Annotation Pipeline In this section, we describe the general data processing pipeline which we applied to the V oxCeleb [7] and Expresso [8] corpora to construct the NonverbalTTS corpus. The proposed pipeline consists of four main steps: (1) detecting nonverbal vocaliza- tions, (2) identifying emotions, (3) human annotation refine- ment, and (4) a...

work page

[4] [4]

This section provides an overview of the original data sources and statistics on NVs and emotion tags in the resulting dataset

NVTTS Dataset NVTTS results from annotating and filtering existing V oxCeleb 1 [24], V oxCeleb 2 [25], and Expresso [8] corpora. This section provides an overview of the original data sources and statistics on NVs and emotion tags in the resulting dataset. 4.1. Data Sources The V oxCeleb dataset was originally designed for speaker ver- ification tasks, ho...

work page

[5] [5]

Merge all annotator hypotheses into a single comprehensive version m3

work page

[6] [6]

Align each individual annotator’s hypothesis with the merged version, receiving a1, a2, a3

work page

[7] [7]

V ote on each letter and NV tag to create the final annotation ttarget. Given the example of initial weak transcription ti, the an- notators’ hypotheses t1, t2, t3: • ti - It’s dog [laugh] on the mat • t1 - It’s a cat [laugh] on the mat • t2 - It’s a cat [laugh] on the sofa • t3 - It’s a cat [sigh] on the mat We apply the merge algorithm 1 to create a fin...

work page

[8] [9]

Experiments This section presents experiments using the NVTTS dataset to train zero-shot TTS models with NV synthesis capability. 5.1. Training and Experimental Setup All our experiments are based on the CosyV oice-300M [3] model. We train only the language model component [3] of the full architecture in a supervised fine-tuning setup. We use the Adam opt...

work page