Pith. sign in

REVIEW 4 major objections 6 minor 72 references

Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vector quantization is not needed for autoregressive speech synthesis; a continuous-latent model beats VALL-E with 10.3% of its parameters.

desk verdict The conceptual claim survives; the SOTA claim against VALL-E does not, because the baseline is a black box. read the letter →

arxiv 2502.01084 v2 pith:PMEYFPHD submitted 2025-02-03 cs.LG cs.SDeess.AS

Weiwei Lin , Chenghan He This is my paper · ORCID
classification cs.LGcs.SDeess.AS
keywords continuousspeechcodecGaussianmixtureVAEautoregressivemodellanguagezero-shottext-to-speechstochasticmonotonicalignmentvectorquantizationalternativesynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that autoregressive speech synthesis does not need vector quantization. It replaces the discrete codebook of a VQ-based codec with a continuous, multi-modal latent space from a Gaussian-mixture VAE, and replaces categorical next-token prediction with a Gaussian-mixture conditional distribution. A stochastic monotonic alignment keeps the encoder-decoder alignment strictly monotonic during training and generation. On zero-shot text-to-speech, the resulting compact model reports lower word error rates and higher naturalness scores than VALL-E while using roughly 10% of VALL-E's parameters. The point of the claim is that continuous speech language models are a simpler, more efficient alternative to quantization-based ones.

What carries the argument

The supporting mechanism is a pair of Gaussian-mixture models. GMM-VAE is a variational autoencoder whose latent prior is a learned mixture of Gaussians; it compresses waveforms into continuous multi-modal latent frames without any codebook. GMM-LM is an autoregressive Conformer that, instead of a softmax over tokens, emits the weights, means, and diagonal variances of a Gaussian mixture for the next latent frame and trains by negative log-likelihood. The third piece is stochastic hard monotonic alignment: attention energies are turned into Bernoulli samples during the forward pass, so every alignment is exactly monotonic, while gradients flow through a Gumbel-Softmax straight-through estimator. The mixture constraint supplies the multi-modality that quantization used to provide, the GMM output supplies a valid continuous probability distribution for sampling, and the monotonic alignment supplies the phonetic stability.

What would settle it

Train the same GMM-LM architecture with the same monotonic alignment on discrete codes from a properly tuned RVQ codec instead of continuous GMM-VAE latents; if its WER reaches the continuous model's roughly 2.7 on LibriSpeech test-clean, the continuous latent representation would not be what drives the result.

Watch

Extended reading notes

Core claim

The paper's central claim is that vector quantization is not a necessary prerequisite for learning an autoregressive speech model. The authors build a two-stage system in which a VAE with a learned Gaussian-mixture constraint compresses speech into a continuous latent space (GMM-VAE), and a Conformer-based autoregressive model predicts the next latent vector as a mixture of Gaussians (GMM-LM), with attention forced to be strictly monotonic by sampling binary alignment decisions from a Bernoulli distribution during the forward pass and backpropagating through a Gumbel-Softmax relaxation. In zero-shot text-to-speech evaluations on LibriSpeech, the authors report that this continuous approach achieves WER of 2.72-3.02 and Q-MOS of 3.87-4.16 across prompt lengths, outperforming the discrete autoregressive VALL-E baseline (WER 6.04-9.68, Q-MOS 3.32-3.58) while using only 51.5M parameters in its mini version, about 10.3% of VALL-E's size. The authors also report better reconstruction quality for GMM-VAE than Encodec, and better teacher-forced reconstruction than VALL-E.

Load-bearing premise

The central outperformance claim rests on the VALL-E baseline being a strong, faithful implementation, and the paper does not say how that baseline was trained or whether an official checkpoint was used.

Editorial extensions

If this is right

  • Discrete tokenization, multi-codebook AR/NAR pipelines, and codebook-specific second-stage models can be dropped entirely; one Conformer models all frames autoregressively, halving theoretical computation relative to VALL-E's two-stage approach.
  • Model size can shrink dramatically: the 51.5M-parameter mini model already beats VALL-E, and scaling to 315M mainly improves speaker similarity rather than content accuracy.
  • The latent space from the GMM-VAE helps alignment: GMM-LM trained on GMM-VAE features aligns faster and performs better than one trained on raw Mel-spectrograms, especially at small model sizes.
  • Monotonic alignment is load-bearing for content: switching from cross-attention to the proposed stochastic hard monotonic alignment lowers WER from 6.6% to 2.72% for the continuous model, and discrete AR models also improve but do not catch up.
  • Continuous autoregressive models support sampling and diversity: with 6 diagonal-covariance mixtures the model produces more diverse samples than VALL-E, StyleTTS2, and HierSpeech++ while keeping WER and SIM strong.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same continuous-latent recipe could be carried to other neural codec applications, such as music or sound-effect generation, where RVQ codebooks are also standard and their artifacts are a known annoyance.
  • The paper leaves implicit that its alignment scheme separates content fidelity from speaker fidelity: scaling the model improves SIM and S-MOS much more than WER, suggesting speaker modeling, not phonetic content, is what benefits from extra capacity.
  • A testable consequence the paper does not pursue is that the learned GMM means alone carry most of the multi-modality, since weights and covariances are fixed; training the full mixture parameters might further improve reconstruction or diversity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a fully continuous autoregressive speech synthesis pipeline. A GMM-VAE codec replaces residual-vector-quantization codecs by imposing a learned mixture-of-Gaussians prior on a deterministic latent; a second-stage GMM-based autoregressive model (GMM-LM) predicts the next latent frame as a mixture of Gaussians, and a hard stochastic monotonic alignment mechanism is trained with a Gumbel straight-through estimator. Experiments on LibriSpeech zero-shot TTS report lower WER and higher MOS than VALL-E, with only 10.3% of VALL-E's parameters, together with reconstruction comparisons, scaling curves, and ablations. The central conceptual claim is that vector quantization is not a necessary prerequisite for autoregressive speech modeling.

Significance. The paper has a genuinely interesting core idea and one clean piece of evidence in its favor: Appendix A.6 compares the continuous GMM-LM against discrete AR baselines under a shared architecture and shows a large WER improvement (2.72% versus 5.35-5.87% with monotonic alignment). This internal control directly supports the claim that discretization is not required for an autoregressive TTS model under the authors' training setup. The scaling experiments and the ablation of the divergence weight also provide useful information. However, the headline claims against VALL-E and the parameter-efficiency figure rest on an undescribed external baseline and an inconsistent parameter accounting; these need to be corrected or substantially reframed before the paper's central advertised claims are supportable.

major comments (4)
  1. [§4, Baseline Models; Table 2] The VALL-E baseline is not described. The text gives training details only for HierSpeech++ and StyleTTS2; for VALL-E there is no checkpoint source, training recipe, hyperparameters, codec configuration, or inference procedure. Since VALL-E's official implementation is not public, the reported WER of 6.04-9.68 and SIM of 0.63-0.73 may come from a non-competitive reproduction. This is load-bearing because the abstract's central claim of significantly outperforming the state-of-the-art autoregressive model VALL-E depends directly on this comparison; the paper's own same-architecture evidence in Appendix A.6 supports only the weaker claim that a GMM-based continuous LM beats discrete AR models in the authors' pipeline. Please either provide a reproducible VALL-E baseline, compare against officially reported VALL-E numbers under the same protocol, or reframe the headline claim to match the Appendix A.6 evidence.
  2. [§3.1, Eq. (4)] The KL divergence in Eq. (4) is not well-defined as written. The posterior is stated to be q(h|x)=δ(h−E(x)), and the KL divergence between a Dirac delta and a Gaussian mixture prior is not finite; one cannot compute it by Monte Carlo sampling without an additional smoothing or density model. If the implemented objective instead maximizes the log-density of the GMM prior at the encoded point, or uses some other approximation to the KL, the paper should state that explicitly and give the actual loss. This is important because the divergence constraint is the only mechanism shaping the continuous latent distribution after quantization is removed, and Section 5.4 shows that without it the downstream model cannot be trained.
  3. [§5.3, Table 3; §A.5] The codec comparison is made against Encodec, while the GMM-VAE is built on the DAC architecture from Kumar et al. (2024), including the same convolutional encoder-decoder and discriminator setup. DAC is a stronger RVQ-based codec than Encodec, so the current comparison does not support the general claim that a continuous GMM-VAE outperforms RVQ codecs. Please add DAC to Table 3, or restrict the claim to the specific comparison against Encodec.
  4. [§1, Table 1 vs §A.5] The parameter-efficiency claim in the abstract (10.3% of VALL-E's parameters) is computed from TTS-model parameters only (51.5M versus 496.5M), but Table 1 reports the proposed method's codec-related parameters as 0 even though the GMM-VAE itself has 76.5M parameters according to Appendix A.5. Counting full system parameters, the proposed method is about 128M versus VALL-E's 513.2M, roughly 25%, not 10.3%. Please make the parameter accounting consistent and report both TTS-only and full-system parameter counts.
minor comments (6)
  1. [§4, Evaluation Protocol] The subjective MOS section specifies the number of samples and listeners but does not state whether listeners were screened, whether the evaluation was blind, or how the Q-MOS and S-MOS ratings were collected. Please provide the full protocol.
  2. [Table 2] The table title says 'P-MOS' while the column headers report 'Q-MOS'; Section 4 defines Q-MOS and S-MOS. Please correct the inconsistency.
  3. [§5.4, Table 5] The diversity evaluation is described in a single sentence, with no detail on the 0-5 scale, judge instructions, or whether failed generations were counted. The statement that VALL-E 'frequently failed' during random sampling is anecdotal and should be quantified or removed.
  4. [§4 vs Table 3] The evaluation protocol defines the metric as SI-SNR, but Table 3 reports 'SI-SDR'. Please use one name consistently.
  5. [§A.4, Table 8] The text and table refer to 'HierSpeech+++' with three plus signs, while the rest of the paper uses 'HierSpeech++'. Please fix the naming.
  6. [§8, Reproducibility Statement] The reproducibility statement says code and pre-trained models will be released 'soon'; they are not available at the time of review. Because the external baselines are not described, this currently prevents independent verification of the central comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are measured against external baselines on held-out data, and the derivation does not reduce to fitted inputs or self-citation.

full rationale

The paper's central claim is that a continuous GMM-VAE codec plus a GMM-output autoregressive model with stochastic hard monotonic alignment can replace RVQ-based discrete speech LMs. The training objective (Eq. 4) is a standard ELBO with a Monte-Carlo KL to a learned GMM prior; the AR stage (Eq. 6) is trained with negative log-likelihood on held-out precomputed latents; the alignment (Eqs. 8-11) is a straight-through Gumbel relaxation of Raffel monotonic attention. None of these quantities is defined in terms of the target metric (WER/SIM/MOS), and no parameter is fitted to the evaluation set. Results are reported on LibriSpeech test-clean with external ASR (wav2vec2-large), external speaker embedding (WeSpeaker), and human MOS, and compared against external baselines VALL-E, HierSpeech++, and StyleTTS2. The paper cites no work by the present authors, so there is no self-citation chain. The underspecified VALL-E reproduction and the use of Mel-distance as both a training loss and an evaluation metric are legitimate experimental-design concerns, but they are not circularity in the derivation: the model's generation pipeline is self-contained and its headline comparisons are not forced by construction. Appendix A.6 actually provides an internal controlled comparison that independently supports the weaker conceptual claim that, under one architecture and training setup, GMM-LM beats discrete AR variants, which strengthens rather than weakens the non-circularity finding.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method rests on the assumption that a continuous, multi-modal Gaussian mixture latent space is as learnable as a discrete codebook by an autoregressive model. The free parameters (mixture counts, lambda, covariance type) are tuned on the evaluation set, which inflates the reported numbers. No code released.

free parameters (5)
  • Number of Gaussian components in GMM-VAE prior = 3
    Selected via ablation in Table 4 as best trade-off for downstream GMM-LM teacher forcing Mel distance; 6 components overfit.
  • Divergence weight lambda = 50
    Chosen from Table 4 after sweeping 0.1, 1, 10, 50, 100; lambda=0.1 prevents GMM-LM training, 50 gives best downstream teacher forcing Mel distance.
  • Number of Gaussian components in GMM-LM output = 6
    Selected from Table 5 on WER, SIM, and diversity; 10 mixtures increased WER, full covariance hurt.
  • Covariance type in GMM-LM = diagonal
    Table 5 shows full covariance worsens WER with more mixtures; diagonal with 6 mixtures gives best combination.
  • Gumbel temperature schedule = not specified
    The paper says temperature s decreases gradually but does not give the schedule, a reproducibility gap.
assumptions (4)
  • domain assumption A deterministic Dirac-delta posterior with a learned mixture-of-Gaussians prior yields a latent space that a downstream autoregressive model can learn.
    Introduced in Section 3.1 as the key replacement for VQ; without the divergence constraint the downstream model cannot be trained (Section 5.4).
  • domain assumption Gumbel-Softmax straight-through gradients through Bernoulli sampling preserve a usable training signal for monotonic alignment.
    Section 3.3 relies on this to train the stochastic hard alignment; no analysis or proof of unbiasedness is given.
  • domain assumption Whisper V2 transcriptions of LibriLight are accurate enough to serve as text labels for TTS training.
    Section 4 uses Whisper-transcribed LibriLight; errors in transcription would hurt content fidelity.
  • domain assumption The LibriSpeech test set and wav2vec2-Large ASR are valid proxies for content and speaker fidelity of generated speech.
    Sections 4 and 5 use WER with wav2vec2 and SIM with WeSpeaker as objective metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis." pith.science (2026). https://pith.science/paper/PMEYFPHD

@misc{pith2026250201084,
  author       = {Pith},
  title        = {Pith review of: Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMEYFPHD}},
  note         = {Machine review of arXiv:2502.01084}
}
read the original abstract

We propose a novel autoregressive modeling approach for speech synthesis, combining a variational autoencoder (VAE) with a multi-modal latent space and an autoregressive model that uses Gaussian Mixture Models (GMM) as the conditional probability distribution. Unlike previous methods that rely on residual vector quantization, our model leverages continuous speech representations from the VAE's latent space, greatly simplifying the training and inference pipelines. We also introduce a stochastic monotonic alignment mechanism to enforce strict monotonic alignments. Our approach significantly outperforms the state-of-the-art autoregressive model VALL-E in both subjective and objective evaluations, achieving these results with only 10.3\% of VALL-E's parameters. This demonstrates the potential of continuous speech language models as a more efficient alternative to existing quantization-based speech language models. Sample audio can be found at https://tinyurl.com/gmm-lm-tts.

Figures

Figures reproduced from arXiv: 2502.01084 by the authors.

Figure 1
Figure 1. Training procedure and architecture difference of typical residual vector quantitation codec [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Training procedure and architecture difference of VALL-E with residual vector quantita [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison on GMM-LM models’ zero-shot TTS performance trained on Mel [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 29 canonical work pages

  1. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proc. Advances in Neural Information Processing Systems, 2020

    work page 2020

  2. [2]

    Better speech synthesis through scaling

    James Betker. Better speech synthesis through scaling. arXiv preprint arXiv:2305.07243, 2023

    arXiv 2023

  3. [3]

    Audiolm: a language modeling approach to audio generation

    Zal \'a n Borsos, Rapha \"e l Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 0 2523--2533, 2023 a

    work page 2023

  4. [4]

    Soundstorm: Efficient parallel audio generation

    Zal \'a n Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636, 2023 b

    arXiv 2023

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    o lge, G \

    Edresson Casanova, Kelly Davis, Eren G \"o lge, G \"o rkem G \"o knar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et al. Xtts: a massively multilingual zero-shot text-to-speech model. arXiv preprint arXiv:2406.04904, 2024

    arXiv 2024

  7. [7]

    A vector quantized approach for text to speech synthesis on real-world spontaneous speech

    Li-Wei Chen, Shinji Watanabe, and Alexander Rudnicky. A vector quantized approach for text to speech synthesis on real-world spontaneous speech. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 12644--12652, 2023

    work page 2023

  8. [8]

    ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Speed

    Meiying Chen and Zhiyao Duan. Controlvc: Zero-shot voice conversion with time-varying controls on pitch and rhythm. arXiv preprint arXiv:2209.11866, 2022

    work page Pith review arXiv 2022

Show all 72 references
  1. [9]

    WavLM : Large-scale self-supervised pre-training for full stack speech processing

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. WavLM : Large-scale self-supervised pre-train...

    2022

  2. [10]

    Monotonic chunkwise attention

    Chung-Cheng Chiu and Colin Raffel. Monotonic chunkwise attention. arXiv preprint arXiv:1712.05382, 2017

    2017 arXiv

  3. [11]

    Self-supervised speech representations are more phonetic than semantic

    Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Livescu, and Shinji Watanabe. Self-supervised speech representations are more phonetic than semantic. arXiv preprint arXiv:2406.08619, 2024

    2024 arXiv

  4. [12]

    Unsupervised speech representation learning using wavenet autoencoders

    Jan Chorowski, Ron J Weiss, Samy Bengio, and A \"a ron Van Den Oord. Unsupervised speech representation learning using wavenet autoencoders. IEEE/ACM transactions on audio, speech, and language processing, 27 0 (12): 0 2041--2053, 2019

    2019

  5. [13]

    Voxceleb2: Deep speaker recognition

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. arXiv preprint arXiv:1806.05622, 2018

    2018 arXiv

  6. [14]

    Simple and controllable music generation

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D \'e fossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [15]

    The road less scheduled

    Aaron Defazio, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, et al. The road less scheduled. arXiv preprint arXiv:2405.15682, 2024

    2024 arXiv

  8. [16]

    High fidelity neural audio compression

    Alexandre D \'e fossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  9. [17]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  10. [18]

    Deep unsupervised clustering with gaussian mixture variational autoencoders

    Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648, 2016

    2016 arXiv

  11. [19]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12873--12883, 2021

    2021

  12. [20]

    Stochastic backpropagation through mixture density distributions

    Alex Graves. Stochastic backpropagation through mixture density distributions. arXiv preprint arXiv:1607.05690, 2016

    2016 arXiv

  13. [21]

    Robust sequence-to-sequence acoustic modeling with stepwise monotonic attention for neural tts

    Mutian He, Yan Deng, and Lei He. Robust sequence-to-sequence acoustic modeling with stepwise monotonic attention for neural tts. arXiv preprint arXiv:1906.00672, 2019

    1906 arXiv

  14. [22]

    Visqol: an objective speech quality model

    Andrew Hines, Jan Skoglund, Anil C Kokaram, and Naomi Harte. Visqol: an objective speech quality model. EURASIP Journal on Audio, Speech, and Music Processing, 2015: 0 1--18, 2015

    2015

  15. [23]

    Reducing the dimensionality of data with neural networks

    Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313 0 (5786): 0 504--507, 2006

    2006

  16. [24]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei - Ning Hsu, Benjamin Bolte, Yao - Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process. , 29: 0 3451--3460, 2021

    2021

  17. [25]

    Prodiff: Progressive fast diffusion model for high-quality text-to-speech

    Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In Proceedings of the 30th ACM International Conference on Multimedia, pp.\ 2595--2605, 2022

    2022

  18. [26]

    Categorical reparameterization with gumbel-softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016

    2016 arXiv

  19. [27]

    Diff-tts: A denoising diffusion model for text-to-speech

    Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. arXiv preprint arXiv:2104.01409, 2021

    2021 arXiv

  20. [28]

    Libri-light: A benchmark for asr with limited or no supervision

    Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazar \'e , Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE Inter...

    2020

  21. [29]

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech

    Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pp.\ 5530--5540. PMLR, 2021

    2021

  22. [30]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014

    2014

  23. [31]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33: 0 17022--17033, 2020

    2020

  24. [32]

    High-fidelity audio compression with improved rvqgan

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36, 2024

    2024

  25. [33]

    Robust training of vector quantized bottleneck models

    Adrian a \'n cucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alum \"a e, and Antoine Laurent. Robust training of vector quantized bottleneck models. In 2020 International Joint Conference on Neural Networks (IJCNN), ...

    2020

  26. [34]

    Voicebox: Text-guided multilingual universal speech generation at scale

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306.15687, 2023

    2023 arXiv

  27. [35]

    Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis

    Sang-Hoon Lee, Ha-Yeong Choi, Seung-Bin Kim, and Seong-Whan Lee. Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. arXiv preprint arXiv:2311.12454, 2023

    2023 arXiv

  28. [36]

    Styletts: A style-based generative model for natural and diverse text-to-speech synthesis

    Yinghao Aaron Li, Cong Han, and Nima Mesgarani. Styletts: A style-based generative model for natural and diverse text-to-speech synthesis. arXiv preprint arXiv:2205.15439, 2022

    2022 arXiv

  29. [37]

    Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models

    Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  30. [38]

    Jets: Jointly training fastspeech2 and hifi-gan for end to end text to speech

    Dan Lim, Sunghee Jung, and Eesung Kim. Jets: Jointly training fastspeech2 and hifi-gan for end to end text to speech. arXiv preprint arXiv:2203.16852, 2022

    2022 arXiv

  31. [39]

    Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans

    Songxiang Liu, Dan Su, and Dong Yu. Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans. arXiv preprint arXiv:2201.11972, 2022

    2022 arXiv

  32. [40]

    Natural language guidance of high-fidelity text-to-speech with synthetic annotations

    Dan Lyth and Simon King. Natural language guidance of high-fidelity text-to-speech with synthetic annotations. arXiv preprint arXiv:2402.01912, 2024

    2024 arXiv

  33. [41]

    Adversarial autoencoders

    Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644, 2015

    2015 arXiv

  34. [42]

    Should you use a probabilistic duration model in tts? probably! especially for spontaneous speech

    Shivam Mehta, Harm Lameris, Rajiv Punmiya, Jonas Beskow, \'E va Sz \'e kely, and Gustav Eje Henter. Should you use a probabilistic duration model in tts? probably! especially for spontaneous speech. arXiv preprint arXiv:2406.05401, 2024 a

    2024 arXiv

  35. [43]

    Matcha-tts: A fast tts architecture with conditional flow matching

    Shivam Mehta, Ruibo Tu, Jonas Beskow, \'E va Sz \'e kely, and Gustav Eje Henter. Matcha-tts: A fast tts architecture with conditional flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 11341--11345. IEEE, 2024 b

    2024

  36. [44]

    Finite scalar quantization: Vq-vae made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023

    2023 arXiv

  37. [45]

    Voxceleb: a large-scale speaker identification dataset

    Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612, 2017

    2017 arXiv

  38. [46]

    Librispeech: An ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In Proc. International Conference on Acoustics, Speech and Signal Processing, ICASSP , pp.\ 5206--5210, 2015

    2015

  39. [47]

    Speech resynthesis from discrete disentangled self-supervised representations

    Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355, 2021

    2021 arXiv

  40. [48]

    Grad-tts: A diffusion probabilistic model for text-to-speech

    Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp.\ 8599--8608. PMLR, 2021

    2021

  41. [49]

    Autovc: Zero-shot voice style transfer with only autoencoder loss

    Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Autovc: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pp.\ 5210--5219. PMLR, 2019

    2019

  42. [50]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  43. [51]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. CoRR, abs/2212.04356, 2022

    2022 arXiv

  44. [52]

    Online and linear-time attention by enforcing monotonic alignments

    Colin Raffel, Minh-Thang Luong, Peter J Liu, Ron J Weiss, and Douglas Eck. Online and linear-time attention by enforcing monotonic alignments. In International conference on machine learning, pp.\ 2837--2846. PMLR, 2017

    2017

  45. [53]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp.\ 8821--8831. Pmlr, 2021

    2021

  46. [54]

    Multi-task self-supervised learning for robust speech recognition

    Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (...

    2020

  47. [55]

    SpeechBrain : A general-purpose speech toolkit, 2021

    Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidon...

    2021 arXiv

  48. [56]

    Fastspeech: Fast, robust and controllable text to speech

    Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019

    2019

  49. [57]

    Fastspeech 2: Fast and high-quality end-to-end text to speech

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558, 2020

    2006 arXiv

  50. [58]

    Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications

    Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017

    2017 arXiv

  51. [59]

    Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions

    Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In International Conference on Acoustics, Speech an...

    2018

  52. [60]

    Vae with a vampprior

    Jakub Tomczak and Max Welling. Vae with a vampprior. In International conference on artificial intelligence and statistics, pp.\ 1214--1223. PMLR, 2018

    2018

  53. [61]

    Givt: Generative infinite-vocabulary transformers

    Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite-vocabulary transformers. In European Conference on Computer Vision, pp.\ 292--309. Springer, 2024

    2024

  54. [62]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017

    2017

  55. [63]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  56. [64]

    Extracting and composing robust features with denoising autoencoders

    Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp.\ 1096--1103, 2008

    2008

  57. [65]

    Neural codec language models are zero-shot text to speech synthesizers, 2023

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers, 2023. URL: https://arxiv. org/abs/2301.02111. doi: doi, 10

    2023 arXiv

  58. [66]

    Wespeaker: A research and production oriented speaker embedding learning toolkit

    Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. Wespeaker: A research and production oriented speaker embedding learning toolkit. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal P...

    2023

  59. [67]

    Tacotron: Towards end-to-end speech synthesis

    Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135, 2017

    2017 arXiv

  60. [68]

    Soundstream: An end-to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 0 495--507, 2021

    2021

  61. [69]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  62. [70]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  63. [71]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  64. [72]

    1.0" encoding=

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2006

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.