Pith. sign in

REVIEW 3 major objections 6 minor 43 references

Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Quantizing speech at four linguistic levels keeps prosody and emotion that frame-level codebooks discard.

desk verdict A simple, well-posed idea—quantizing HuBERT at four segment levels with pre-pooling—with consistent but underpowered evidence; worth a real referee, not a desk reject. read the letter →

arxiv 2505.15667 v1 pith:DAESZMTP submitted 2025-05-21 eess.AS cs.CLcs.SD

Nicholas Sanders , Yuanchao Li , Korin Richmond , Simon King This is my paper · ORCID
classification eess.AScs.CLcs.SD
keywords discretespeechunitsvectorquantizationHuBERTprosodypreservationparalinguisticinformationemotionrecognitionprominenceclassificationresynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Segmentation-Variant Codebooks (SVCs), a way to quantize self-supervised speech features into discrete units without discarding prosody and emotion. The central claim is that quantizing HuBERT features separately at the frame, phone, word, and utterance levels, with pooling performed before discretization, retains paralinguistic information far better than a single large frame-level codebook at a comparable bitrate. The authors support this with linear-probe emotion recognition, prominence classification, and expressive resynthesis experiments, where SVCs beat frame-level baselines of both k=500 and k=2000 on every classification category. If the paper is right, the information loss in discrete speech units is not an inherent cost of quantization but can be controlled by matching the quantization scale to linguistic structure.

What carries the argument

The load-bearing mechanism is the Segmentation-Variant Codebook setup: a family of four parallel KMeans codebooks, each trained on HuBERT-large embeddings pooled to a distinct linguistic granularity — frame, phone, word, and utterance — with segment boundaries supplied by forced alignment. The critical design choice is pre-pooling: averaging continuous HuBERT features inside each segment before quantization, which retains prosodic patterns that per-frame quantization would otherwise destroy. A supporting mechanism is the bitrate accounting in Equation (1), which shows that the multi-stream factorization buys representation power without the bitrate blow-up of a single large frame codebook. The same pooled-segment structure is then reused to recombine the discrete streams into a frame-length sequence for probing.

What would settle it

Train the same four-stream SVCs on segment boundaries that have been randomly jittered or replaced with fixed-length windows that ignore the transcript; if probing accuracy and style retention stay close to the forced-alignment version, then the benefit does not come from meaningful linguistic units and the alignment dependence is not load-bearing.

Watch

Extended reading notes

Core claim

The central discovery is that prosodic and paralinguistic information survives discretization when the quantization codebook is tailored to the segment length over which that information lives. Instead of quantizing every 25 ms frame with a single codebook, SVCs train four independent KMeans codebooks on HuBERT-large representations: unpooled frames, and mean-pooled phones, words, and utterances derived from forced alignments. This produces four parallel discrete streams, which are then mean-pooled back to frame rate for downstream use. Across probing tasks, SVCs outperform frame-level baselines with larger codebooks (k=2000) at similar or lower bitrate, and pre-pooling before discretization consistently beats post-pooling. In resynthesis on Expresso, the SVC-trained HiFi-GAN preserves expressive style better than the frame-level discrete baselines while keeping intelligibility and quality slightly higher.

Load-bearing premise

The method depends on forced alignments giving accurate phone, word, and utterance boundaries; if those boundaries are noisy, mean pooling mixes frames from different linguistic units and the codebooks learn arbitrary chunks rather than true segment identities.

Editorial extensions

If this is right

  • Speech codecs and text-to-speech front-ends that adopt SVCs should preserve emotion, stress, and speaking style better than frame-level discrete units at comparable bitrates.
  • Pooling before discretization should become the default for segment-level discrete units, since post-pooling loses the very prosodic cues the codebooks are meant to keep.
  • The bitrate-efficiency gain means a four-stream SVC system with k=500 per codebook can beat a single k=2000 frame codebook while transmitting fewer bits per second.
  • Resynthesis with SVCs yields lower word error rate and higher predicted MOS than frame-level discrete baselines, narrowing the gap toward continuous-feature synthesis.
  • Because SVCs combine multiple streams, downstream models can weigh or select streams by granularity instead of treating all discrete units as interchangeable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if SVCs' gain actually comes from pre-pooling rather than from true segment identities, then replacing forced alignments with cheap unsupervised or even random segment boundaries should preserve most of the benefit; testing this would isolate the active ingredient.
  • Beyond the paper: the four streams could be used as independently controlled conditioning inputs for expressive TTS, letting a speaker or style tag modulate only the utterance- or word-level codebook while frame and phone streams carry articulation.
  • Beyond the paper: since emotional speech often blurs phone boundaries through lengthened vowels and creak, the method may degrade on highly expressive input; adapting aligners to expressive data could recover the margin observed here.
  • Beyond the paper: combining SVC streams with learned mixing weights or targeted masking, rather than uniform mean pooling, could push probing accuracy closer to the continuous-feature ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Segmentation-Variant Codebooks (SVCs), which quantize HuBERT features at four segmentation levels (frame, phone, word, utterance) using separate KMeans codebooks, with mean pooling applied before discretization for the segment-level streams. The resulting multi-stream discrete representations are evaluated on speech emotion recognition (IEMOCAP), prominence classification (Naver Prosody Control), and expressive resynthesis (Expresso), comparing against frame-level codebooks of size k=500 and k=2000 and against continuous HuBERT features. The authors report that SVCs outperform the frame-level discrete baselines on probing tasks and resynthesis style accuracy, that pre-pooling before discretization is more effective than post-pooling, and that SVCs improve resynthesis quality and intelligibility relative to the discrete baselines.

Significance. If the reported results hold, SVCs offer a simple and practical way to improve prosodic and paralinguistic preservation in discrete speech units at a modest bitrate increase, and the pre-pooling versus post-pooling comparison is a useful methodological finding for the discrete-unit community. The paper has several strengths: codebooks and probes are trained on training splits only and evaluated on held-out test splits; the comparisons include both frame-level baselines at comparable bitrates and continuous-feature upper bounds; and the probing results are corroborated by a resynthesis experiment with objective style, quality, and intelligibility metrics. The main risks are the unvalidated reliance on forced alignments and the absence of any uncertainty quantification for the reported differences, both of which are load-bearing for the central claims.

major comments (3)
  1. [Section 4.1 and Section 7] The method's factorization into phone, word, and utterance streams depends entirely on forced-alignment boundaries (MFA for NPC and Expresso, HTK for IEMOCAP), yet the paper provides no alignment-quality analysis. On expressive, acted, or conversational speech, aligners are known to be error-prone, especially for word boundaries under emphasis and lengthening; if the boundaries are noisy, the mean-pooled representations do not correspond to genuine linguistic units, and the codebooks may encode arbitrary variable-length chunks. This concern is most acute for prominence classification, where the SVC gain over k=2000 is the smallest (0.305 vs. 0.249 in Table 3) and word boundaries are the very locations where prosodic variation distorts alignments. Please report alignment-quality metrics (e.g., boundary error or frame-level agreement) or provide a sensitivity analysis, such as perturbing boundaries or comparing against oracle alignments on a subset, to show that the conclusions are robust to realistic alignment noise.
  2. [Section 6, Tables 2-4, Abstract] The abstract and Section 6.2 claim that SVCs are 'significantly more effective' than the baselines, but every reported number comes from a single run with no error bars, confidence intervals, or significance tests. For several categories the margins are small (e.g., Ang. 0.614 vs. 0.298 for k=2000 in Table 3; Hap. 0.169 vs. 0.070), and without repeated runs or statistical testing it is impossible to know whether these differences are stable. Because the central claim is a comparative one, this is a load-bearing gap. Please provide multiple seeds or bootstrap estimates, and use an appropriate significance test or confidence intervals for the key comparisons in Tables 2-4.
  3. [Section 5.1 and Table 2] The pre-pooling versus post-pooling comparison is a central finding, but the manuscript does not specify exactly what the post-pooling baselines are for the 'Utterance', 'Word', and 'SVCs' rows of Table 2. If the post-pooling condition uses only a frame-level codebook followed by pooling of discrete indices, then the comparison conflates pooling order with the number of codebooks and the segment-specific codebooks, which would confound the stated conclusion that 'pooling before rather than after discretization better retains segment-level information.' Please describe precisely which representations and codebooks are used in each row, and clarify whether the SVCs post-pooling condition still uses all four codebooks or only the frame-level one.
minor comments (6)
  1. [Section 3.3, Eq. (1)] The bitrate equation is hard to parse: the denominator 'T / S' is written as a ratio, and the symbols N_m, |V_m|, T, and S are not all defined in the text immediately before the equation. Please rewrite the formula and define every symbol.
  2. [Table 1] Table 1 shows bitrate values for SVCs on all three datasets but only shows frame-level baseline bitrates in the IEMOCAP column. If the frame-level bitrates are dataset-independent, state this explicitly; if not, provide the missing values for NPC and Expresso, since Section 6.2 relies on comparing SVC bitrates with k=2000 across datasets.
  3. [Section 6.2, Table 3] The table caption calls the emotion scores 'sub-class Micro F1 scores' but the prominence column is a binary F1 score; please separate these in the caption or table to avoid confusion.
  4. [Section 6.2] There is a typo: 'baslines' should be 'baselines', and 'effecient' should be 'efficient'. Please proofread the results section.
  5. [Section 4.3] The resynthesis section says 'For components without transcriptions, we use Whisper-large to transcribe conversational and long-form utterances,' but it is unclear which components lack transcriptions on Expresso. Please clarify what was transcribed and how this interacts with the forced-alignment pipeline.
  6. [Figure 1] The notation 'Sn refers to the resulting frame-level stream obtained from mean pooling multiple streams of DSUs' is introduced in the caption but not defined in the main text; please define it in Section 3.2 and make the figure self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SVC evaluation is held-out and the method is not constructed from its own outputs; the forced-alignment concern is a robustness limitation, not a circular step.

full rationale

The paper's derivation chain is self-contained against external benchmarks. Continuous HuBERT features are pooled by forced-alignment segment boundaries (Section 3.1), KMeans codebooks are trained only on training splits (Section 4.1), and the resulting discrete streams are probed for emotion and prominence or fed to a HiFi-GAN resynthesizer (Sections 4.2-4.3). All headline quantities (SER micro-F1, prominence F1, style classification accuracy, WER, UTMOS) are computed on held-out test splits or against ground-truth audio, so no fitted parameter is renamed as a prediction and no claimed result reduces by construction to a training target. The self-citations ([5], [7], [8], [18]) are contextual background about SSL representations and prior probing techniques; they do not supply the main result. The one legitimate concern, flagged in the Discussion ('while our current method relies on forced alignments, the inherent flexibility of SVCs could be further explored by investigating automatic and unsupervised segmentation methods'), is a robustness limitation about alignment quality on expressive speech, not a circularity: even if boundaries are noisy, the experimental comparisons remain empirical and unbiased relative to the paper's own definitions. Therefore the score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on design choices (k=500, four codebooks, mean pooling) and on assumptions about alignment accuracy and the interpretability of probing and objective metrics. No invented physical entities are introduced.

free parameters (2)
  • Codebook size k = 500
    All four SVC codebooks use k=500 KMeans clusters, selected by hand rather than tuned; bitrate and capacity of each stream depend on this value.
  • Number and type of segment levels = 4 (frame, phone, word, utterance)
    The choice of four linguistic levels is a design decision, not swept or justified; more levels add streams and bitrate, fewer would reduce expressivity.
assumptions (4)
  • domain assumption KMeans clustering of HuBERT representations yields phone-like discrete units that preserve linguistic content while discarding fine prosodic detail.
    Section 3.1 and Related Work cite [10, 18] for this property; SVCs assume the lost detail can be recovered by segment-level codebooks.
  • domain assumption Forced alignments are accurate on expressive, acted, and conversational speech.
    Section 4.1 uses MFA and HTK alignments; errors would misassign frames to phone, word, or utterance segments.
  • domain assumption Linear probe accuracy on held-out labels reflects the amount of paralinguistic and prosodic information in a representation.
    Section 4.2 uses this standard probing assumption for all comparisons.
  • domain assumption Whisper WER and UTMOS are valid proxies for intelligibility and subjective quality.
    Section 5.3 uses these objective metrics in addition to a style classifier.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information." pith.science (2026). https://pith.science/paper/DAESZMTP

@misc{pith2026250515667,
  author       = {Pith},
  title        = {Pith review of: Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DAESZMTP}},
  note         = {Machine review of arXiv:2505.15667}
}
read the original abstract

Quantization in SSL speech models (e.g., HuBERT) improves compression and performance in tasks like language modeling, resynthesis, and text-to-speech but often discards prosodic and paralinguistic information (e.g., emotion, prominence). While increasing codebook size mitigates some loss, it inefficiently raises bitrates. We propose Segmentation-Variant Codebooks (SVCs), which quantize speech at distinct linguistic units (frame, phone, word, utterance), factorizing it into multiple streams of segment-specific discrete features. Our results show that SVCs are significantly more effective at preserving prosodic and paralinguistic information across probing tasks. Additionally, we find that pooling before rather than after discretization better retains segment-level information. Resynthesis experiments further confirm improved style realization and slightly improved quality while preserving intelligibility.

Figures

Figures reproduced from arXiv: 2505.15667 by the authors.

Figure 1
Figure 1. Segmentation-Variant Codebooks processing with pre-pooling overview. hn refers to continuous hidden representations at the nth frame, Sn refers to the resulting frame-level stream obtained from mean pooling multiple streams of DSUs based on overlapping segmentation. 4.3. Resynthesis For resynthesis, we follow a setup similar to the vocoder-only track in the Interspeech 2024 Challenge on Speech Processing Using Discr… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 31 canonical work pages

  1. [1]

    Introduction With the advent of pre-trained speech foundation models, sig- nificant advancements have been made in the field of speech processing. These models provide universal and comprehensive speech representations, leading to substantial improvements across various tasks, including Automatic Speech Recognition (ASR) [1, 2], Text-to-Speech (TTS) [3, 4...

    work page

  2. [2]

    Are you kidding me?!

    SSL training objectives, such as masked segment predic- tion, which inherently focus on stable and predictable features like phonetic and lexical structures; and 2) KMeans clustering, which tends to group similar phonetic units together [10]. However, while DSUs are capable of preserving linguis- tic information, they struggle to preserve paralinguistic a...

    work page

  3. [3]

    Related Work Speech quantization facilitates efficient downstream process- ing by converting continuous speech representations into dis- crete, structured, and compact units. Although speech quan- tization methods do not have strictly defined categories, they can generally be classified into two major approaches: Vec- tor Quantization (VQ), which is end-t...

    work page Pith review arXiv 2025

  4. [4]

    Segmentation-Variant Codebooks We encode speech into continuous representations using the frozen HuBERT-large model [17]

    Methodology 3.1. Segmentation-Variant Codebooks We encode speech into continuous representations using the frozen HuBERT-large model [17]. Our method operates as fol- lows: first, speech inputs are encoded into frame-wise continu- ous representations via HuBERT. These representations are then pooled across segmentation boundaries (frames, phones, words, a...

    work page 2000

  5. [5]

    Datasets and Alignment Process Naver Prosody Control [21] is a dataset designed for study- ing prosody control in TTS systems

    Experimental Conditions 4.1. Datasets and Alignment Process Naver Prosody Control [21] is a dataset designed for study- ing prosody control in TTS systems. It provides spoken ut- terances with prosodic variations, particularly focusing on con- trastive focus generation, making it suitable for prominence de- tection and prosody modeling. Expresso [11] is a...

    work page 2024

  6. [6]

    Experiments 5.1. Comparison of pooling pre-discretization and post- discretization In this study, we begin by comparing linear probing perfor- mance on features pooled before and after discretization to test whether discretization effectively factors out less salient prosodic and paralinguistic information. Expanding on previ- ous successes in SER probing...

    work page 2000

  7. [7]

    Results 6.1. Factorization of paralinguistic and prosodic informa- tion via pooling For both prominence and emotion, Table 2 shows that pre- pooling is more effective than post-pooling, confirming our hy- pothesis. This effect is particularly pronounced for prominence classification compared to SER, possibly due to differences in task-specific segment len...

    work page 2000

  8. [8]

    The higher the score, the better

    Discussion Our results demonstrate that SVCs efficiently preserve paralin- guistic and prosodic speech qualities, as shown by both probing 1https://yc-li20.github.io/Interspeech2025-SVC-audiosample/ Table 3: Frame-level SER sub-class Micro F1 scores and promi- nence Binary F1 scores. The higher the score, the better. Segmentation Hap. Sad Ang. Neut. Prom....

    work page 2000

Show all 43 references
  1. [9]

    This study contributes to ongoing research on the use of DSUs, demonstrating their po- tential in speech representation learning and downstream tasks

    Conclusion In conclusion, we propose Segmentation-Variant Codebooks, offering a promising approach for efficiently increasing rep- resentation power of DSUs by improving the preservation of paralinguistic and prosodic qualities. This study contributes to ongoing research on th...

  2. [10]

    Acknowledgments This work was supported in part by the UKRI CDT in NLP, funded by the UKRI (grant EP/S022481/1), the University of Edinburgh and Huawei

  3. [11]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,” Advances in neural information processing systems , vol. 33, pp. 12 449–12 460, 2020

    2020

  4. [12]

    Exploring speech recognition, translation, and understanding with discrete speech units: A com- parative study,

    X. Chang, B. Yan, K. Choi, J.-W. Jung, Y . Lu, S. Maiti, R. Sharma, J. Shi, J. Tian, S. Watanabe et al., “Exploring speech recognition, translation, and understanding with discrete speech units: A com- parative study,” in ICASSP 2024-2024 IEEE International Con- ference on Aco...

    2024

  5. [13]

    On the use of self-supervised speech representations in spontaneous speech synthesis,

    S. Wang, G. E. Henter, J. Gustafson, and E. Szekely, “On the use of self-supervised speech representations in spontaneous speech synthesis,” in 12th Speech Synthesis Workshop (SSW), 2023

    2023

  6. [14]

    Selecttts: Syn- thesizing anyone’s voice via discrete unit-based frame selection,

    I. R. Ulgen, S. S. Chandra, J. Lu, and B. Sisman, “Selecttts: Syn- thesizing anyone’s voice via discrete unit-based frame selection,” 2024

    2024

  7. [15]

    Exploration of a self- supervised speech model: A study on emotional corpora,

    Y . Li, Y . Mohamied, P. Bell, and C. Lai, “Exploration of a self- supervised speech model: A study on emotional corpora,” in2022 IEEE Spoken Language Technology Workshop (SLT) , 2023, pp. 868–875

    2023

  8. [16]

    Exploring Wav2vec 2.0 fine tun- ing for improved speech emotion recognition,

    L.-W. Chen and A. Rudnicky, “Exploring Wav2vec 2.0 fine tun- ing for improved speech emotion recognition,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

    2023

  9. [17]

    Layer-wise analysis of self-supervised acoustic word embeddings: A study on speech emotion recognition,

    A. Saliba, Y . Li, R. Sanabria, and C. Lai, “Layer-wise analysis of self-supervised acoustic word embeddings: A study on speech emotion recognition,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing Work- shops (ICASSPW). IEEE, 2024

    2024

  10. [18]

    Crossmodal ASR error correc- tion with discrete speech units,

    Y . Li, P. Chen, P. Bell, and C. Lai, “Crossmodal ASR error correc- tion with discrete speech units,” in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024

    2024

  11. [19]

    Discreteslu: A large language model with self- supervised discrete speech units for spoken language understand- ing,

    S. Shon, K. Kim, Y .-T. Hsu, P. Sridhar, S. Watanabe, and K. Livescu, “Discreteslu: A large language model with self- supervised discrete speech units for spoken language understand- ing,” arXiv preprint arXiv:2406.09345, 2024

    2024 arXiv

  12. [20]

    Analyzing acoustic word embeddings from pre-trained self-supervised speech models,

    R. Sanabria, H. Tang, and S. Goldwater, “Analyzing acoustic word embeddings from pre-trained self-supervised speech models,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023

    2023

  13. [21]

    Ex- presso: A benchmark and analysis of discrete expressive speech resynthesis,

    T. A. Nguyen, W.-N. Hsu, A. d’Avirro, B. Shi, I. Gat, M. Fazel- Zarani, T. Remez, J. Copet, G. Synnaeve, M. Hassid et al., “Ex- presso: A benchmark and analysis of discrete expressive speech resynthesis,” in INTERSPEECH 2023. ISCA, 2023, pp. 4823– 4827

    2023

  14. [22]

    Emo-codec: An in-depth look at emotion preservation capacity of legacy and neural codec models with subjective and objective evaluations,

    W. Ren, Y .-C. Lin, H.-C. Chou, H. Wu, Y .-C. Wu, C.-C. Lee, H.-y. Lee, H.-M. Wang, and Y . Tsao, “Emo-codec: An in-depth look at emotion preservation capacity of legacy and neural codec models with subjective and objective evaluations,” in 2024 Asia Pacific Signal and Informa...

    2024

  15. [23]

    Neural discrete represen- tation learning,

    A. Van Den Oord, O. Vinyals et al., “Neural discrete represen- tation learning,” Advances in neural information processing sys- tems, vol. 30, 2017

    2017

  16. [24]

    vq-wav2vec: Self- supervised learning of discrete speech representations,

    A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self- supervised learning of discrete speech representations,” in Inter- national Conference on Learning Representations, 2020

    2020

  17. [25]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438 , 2022

    2022 arXiv

  18. [26]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 495–507, 2021

    2021

  19. [27]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  20. [28]

    Phonetic analysis of self-supervised representations of english speech,

    D. Wells, H. Tang, and K. Richmond, “Phonetic analysis of self-supervised representations of english speech,” in Interspeech 2022, 2022, pp. 3583–3587

    2022

  21. [29]

    Analysing discrete self supervised speech representation for spoken language modeling,

    A. Sicherman and Y . Adi, “Analysing discrete self supervised speech representation for spoken language modeling,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023

    2023

  22. [30]

    The Interspeech 2024 chal- lenge on speech processing using discrete units,

    X. Chang, J. Shi, J. Tian, Y . Wu, Y . Tang, Y . Wu, S. Watan- abe, Y . Adi, X. Chen, and Q. Jin, “The Interspeech 2024 chal- lenge on speech processing using discrete units,” arXiv preprint arXiv:2406.07725, 2024

    2024 arXiv

  23. [31]

    Controlling prosody in end-to-end TTS: A case study on contrastive focus generation,

    S. Latif, I. Kim, I. Calapodescu, and L. Besacier, “Controlling prosody in end-to-end TTS: A case study on contrastive focus generation,” in Proceedings of the 25th Conference on Compu- tational Natural Language Learning , A. Bisazza and O. Abend, Eds. Online: Association for ...

    2021

  24. [32]

    IEMOCAP: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  25. [33]

    Montreal forced aligner: Trainable text-speech align- ment using Kaldi,

    M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Son- deregger, “Montreal forced aligner: Trainable text-speech align- ment using Kaldi,” in Interspeech 2017, 2017, pp. 498–502

    2017

  26. [34]

    The HTK book,

    S. Young, G. Evermann, D. Kershaw, G. Moore, J. Odell, D. Olla- son, V . Valtchev, and P. Woodland, “The HTK book,”Cambridge University Engineering Department, vol. 3, 2002

    2002

  27. [35]

    k-means++: The advantages of careful seeding,

    D. Arthur and S. Vassilvitskii, “k-means++: The advantages of careful seeding,” Stanford, Tech. Rep., 2006

    2006

  28. [36]

    Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,” Advances in neural information processing systems , vol. 33, pp. 17 022– 17 033, 2020

    2020

  29. [37]

    Su- perb: Speech processing universal performance benchmark,

    S. W. Yang, P. H. Chi, Y . S. Chuang, C. I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G. T. Lin et al. , “Su- perb: Speech processing universal performance benchmark,” in 22nd Annual Conference of the International Speech Communi- cation Association, INTER...

    2021

  30. [38]

    A fine-tuned Wav2vec 2.0/HuBERT benchmark for speech emotion recognition, speaker verification and spoken language understanding,

    Y . Wang, A. Boumadane, and A. Heba, “A fine-tuned Wav2vec 2.0/HuBERT benchmark for speech emotion recognition, speaker verification and spoken language understanding,” arXiv preprint arXiv:2111.02735, 2021

    2021 arXiv

  31. [39]

    Speech emotion di- arization: Which emotion appears when?

    Y . Wang, M. Ravanelli, and A. Yacoubi, “Speech emotion di- arization: Which emotion appears when?” in 2023 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–7

    2023

  32. [40]

    Em- phassess: a prosodic benchmark on assessing emphasis transfer in speech-to-speech models,

    M. de Seyssel, A. D’Avirro, A. Williams, and E. Dupoux, “Em- phassess: a prosodic benchmark on assessing emphasis transfer in speech-to-speech models,” arXiv preprint arXiv:2312.14069 , 2023

    2023 arXiv

  33. [41]

    UTMOS: UTokyo-SaruLab system for voice- mos challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: UTokyo-SaruLab system for voice- mos challenge 2022,” in Proceedings of the Annual Conference of the International Speech Communication Association, INTER- SPEECH, vol. 2022, 2022, pp. 4521–4525

    2022

  34. [42]

    A layer-wise analysis of man- darin and english suprasegmentals in ssl speech models,

    A. de la Fuente and D. Jurafsky, “A layer-wise analysis of man- darin and english suprasegmentals in ssl speech models,” inProc. Interspeech 2024, 2024, pp. 1290–1294

    2024

  35. [43]

    blind speech segmentation: au- tomatic segmentation of speech without linguistic knowledge,

    M. Sharma and R. J. Mammone, “blind speech segmentation: au- tomatic segmentation of speech without linguistic knowledge,” in 4th International Conference on Spoken Language Processing (ICSLP 1996), 1996, pp. 1237–1240

    1996

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.