Pith. sign in

REVIEW 4 major objections 6 minor 61 references

FlexSpeech: Towards Stable, Controllable and Expressive Text-to-Speech

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FlexSpeech separates rhythm from sound to break the TTS stability-expressiveness trade-off, reaching word error rates as low as 1.20%.

desk verdict A credible decoupled AR-duration + NAR-acoustic TTS with duration-level DPO; the architecture is a solid recombination, but the zero-shot evaluation needs to prove the prompt is not the target utterance. read the letter →

arxiv 2505.05159 v3 pith:5SCWZ6Z7 submitted 2025-05-08 eess.AS

Linhan Ma , Dake Guo , He Wang , Jin Xu , Lei Xie This is my paper · ORCID
classification eess.AS
keywords text-to-speechzero-shotTTSautoregressivedurationpredictionnon-autoregressiveacousticmodelflowmatchingdirectpreferenceoptimizationstyletransferspeechstability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FlexSpeech claims that the long-standing trade-off between stability and expressiveness in text-to-speech can be broken by relocating the autoregressive part of generation to duration prediction alone. The model predicts each phoneme's duration one step at a time with an autoregressive encoder-decoder, then hands a duration-expanded phoneme sequence to a flow-matching acoustic model that renders a mel-spectrogram in parallel. Because audibility no longer depends on the autoregressive path getting every acoustic token right, the system reports word error rates of 1.20% on a Chinese test set and 1.81% on an English test set while matching or beating strong baselines on naturalness and speaker similarity. The same separation makes style adaptation cheap: direct preference optimization on win/lose duration pairs, using about 100 samples, transfers a new speaking style to unseen speakers without retraining the acoustic model.

What carries the argument

The load-bearing mechanism is the decoupling of duration prediction from acoustic rendering: an autoregressive duration predictor, which treats each phoneme's duration as a discrete label (the mel-spectrogram frame count) and predicts it by next-token prediction with causal attention, and a non-autoregressive flow-matching acoustic model that generates a mel-spectrogram of exactly the length implied by the expanded phoneme sequence. The duration predictor also reads a reference mel-spectrogram clip through cross-attention and accepts prompt phoneme-duration pairs for in-context style transfer, while the acoustic model receives only a speaker embedding for timbre. This arrangement lets a lightweight DPO pass on duration pairs change rhythm and style without touching the acoustic model, which is what makes the reported fast style transfer possible.

What would settle it

Run the Seed-TTS English and Chinese benchmarks under three prompt conditions: durations from a different sentence by the same speaker, durations predicted by the model itself, and the target sentence's own ground-truth durations. If the last condition reproduces the paper's WER of 1.20% and 1.81% and clearly beats the other two, the zero-shot numbers are contaminated by oracle rhythm; if all three match, the decoupling claim is confirmed under strict zero-shot conditions.

Watch

Extended reading notes

Core claim

The central discovery claimed is that duration is the right carrier for both Markov dependencies and human-preference optimization in a TTS pipeline. FlexSpeech decomposes synthesis into an autoregressive duration predictor and a non-autoregressive acoustic model. The duration predictor is an encoder-decoder transformer whose decoder uses causal attention and next-token prediction to emit each phoneme's duration as a discrete frame-count label, conditioned on reference-mel prosody through cross-attention. The acoustic model is a diffusion-transformer flow-matching network (about 330M parameters) that maps Gaussian noise to a mel-spectrogram of exactly the length implied by the duration-expanded phoneme sequence, conditioned only on a speaker embedding for timbre. Because the acoustic model never estimates rhythm, it cannot accumulate token-level errors, and because the duration predictor is small, DPO on dozens of winner/loser duration pairs can align predicted rhythms with human preferences; the paper reports the lowest word error rates among the systems compared (1.20% on Seed-TTS test-zh, 1.81% on test-en) and says a storyteller style transfers with roughly 100 pairs while the acoustic model stays frozen.

Load-bearing premise

The zero-shot evaluation assumes that the phoneme and duration prompts fed to the duration model at inference come from a different utterance than the one being synthesized, so the reported WER and SMOS are not inflated by oracle rhythm; Section 4.1 says the target speaker's own phonemes and durations are used as prompts, which leaves this assumption unstated.

Editorial extensions

If this is right

  • If the central claim is right, intelligibility no longer has to be traded against expressiveness: the reported WER of 1.20% (Chinese) and 1.81% (English) comes with CMOS and SMOS scores at or near the top of the compared systems.
  • Style transfer becomes a data-efficient operation: performance plateaus around 50–100 DPO pairs, so a production system could adapt to a new narrative or storytelling style with a few minutes of listening data and no acoustic-model retraining.
  • Because durations are explicit and controllable, users can reliably slow or lengthen individual phonemes or whole sentences by scaling the predicted durations, giving phoneme- and sentence-level rate control as a direct corollary.
  • Ablation results indicate both supervised fine-tuning and DPO contribute to the gain, so the full pipeline—pretraining on internet-scale data, cleaning with SFT, then aligning durations—is part of the claimed recipe, not just the architecture alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the zero-shot evaluation uses the target sentence's own ground-truth durations as prompts, as Section 4.1's wording suggests, the reported scores measure rhythm-aided synthesis rather than open-loop zero-shot synthesis; re-running the benchmarks with prompt durations drawn from a different sentence would settle the gap.
  • The decoupling recipe generalizes beyond speech: any generative task where a small discrete variable controls high-level structure (rhythm, timing, layout) and a separate model renders details could use the same autoregressive-control-plus-parallel-renderer split.
  • A practical system could cache several DPO-tuned duration models for different styles and swap them at inference time while sharing one acoustic model, making style a cheap switchable input rather than a retraining burden.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. FlexSpeech is a two-component text-to-speech system: a flow-matching DiT acoustic model that converts a duration-expanded phoneme sequence plus an ECAPA-TDNN speaker embedding into a mel-spectrogram, and an autoregressive transformer duration model that predicts phoneme durations by next-token prediction while conditioning on prompt durations, prompt phonemes, and a reference mel-spectrogram clip. The duration model is aligned with direct preference optimization (DPO), and the authors report state-of-the-art intelligibility (WER 1.81% on Seed-TTS test-en and 1.20% on test-zh), competitive CMOS/SMOS scores, and rapid style transfer using roughly 100 DPO pairs without retraining the acoustic model. The paper also includes ablations isolating the contributions of supervised fine-tuning and DPO in both modules. The main unresolved point is whether the evaluations are truly zero-shot, since the manuscript does not specify whether the prompt durations and reference mel-spectrogram are taken from the target utterance itself.

Significance. The decoupled design, with an explicitly autoregressive duration predictor feeding a non-autoregressive acoustic model, is a plausible and potentially valuable way to combine synthesis stability with prosodic expressiveness. The claim that style adaptation can be achieved with roughly 100 preference pairs by optimizing only the duration module is interesting and, if the protocol is sound, would be a practical contribution. The paper uses standard objective recognizers (Whisper and Paraformer), standard zero-shot benchmarks, and a clear ablation matrix, and it reports the main architectural hyperparameters. However, the central zero-shot and style-transfer claims are not yet verifiable because the prompt-construction protocol is ambiguous; if target-utterance durations and target-audio references are used as prompts, the reported WER and SMOS numbers would be oracle-conditioned and the comparisons to baselines would be unfair.

major comments (4)
  1. [§4.1, §3.3, Table 1] The zero-shot evaluation protocol is not verifiable as written. Section 4.1 states that during inference 'we use the target speaker's phonemes and durations as prompts, with their mel-spectrogram providing a reference for timbre', and Section 3.3 conditions the duration model on d_prompt, p_prompt, and m_ref. If, for each test sample, the prompt is the target utterance itself, then the duration model receives oracle durations at inference and the acoustic model receives a speaker embedding extracted from the ground-truth target audio; in that case the WER of 1.81% on test-en and 1.20% on test-zh, together with the SMOS values in Table 1, are not zero-shot results and are not comparable to baselines that use a separate reference utterance. The authors must state explicitly whether every test prompt is a distinct non-target utterance, describe how reference utterances were selected for LibriSpeech-PC, Common Voice, and DiDiSpeech, and, if target-utterance prompts were used, rerun the evaluation with held-out prompt utterances.
  2. [§5, Figure 2] The style-transfer experiment leaves the inference protocol underspecified. Section 5 does not state what durations are used as prompts for the 100 test sentences; if these are the ground-truth durations of the test sentences, the WER and SMOS curves in Figure 2 are oracle-conditioned and do not demonstrate transfer to unseen text. In addition, Section 3.4 describes the preference pairs as 'manually annotated preference duration pairs', whereas Section 5 constructs pairs by treating predicted durations as losers and ground-truth durations as winners; these are different data-construction protocols. The paper should clarify which protocol produced the main model in Table 1, how many human-annotated pairs were used, and how the test-set prompts for Figure 2 were obtained.
  3. [§4.4, Table 1] The objective speaker-similarity results contradict the subjective similarity claim and are not discussed. FlexSpeech has the lowest SIM-O on Seed-TTS test-en (0.61/0.62 versus 0.67 for F5-TTS and 0.717 for MaskGCT), on test-zh (0.68 versus 0.76 for F5-TTS and 0.774 for MaskGCT), and on LibriSpeech-PC (0.60 versus 0.66 for F5-TTS), yet the text says 'we achieved superior SMOS scores compared to all baselines' and does not mention SIM-O. The authors should report SIM-O and SMOS together, explain the discrepancy (for example, by noting that SMOS includes prosodic pattern while SIM-O measures timbre only), or temper the speaker-similarity claim.
  4. [§4.4, Table 1] No confidence intervals, significance tests, or sampling variability are reported for any objective or subjective metric. The headline WER gaps are small on Seed-TTS test-en (1.81 versus 1.83 for F5-TTS), and the CMOS comparisons to non-open-source baselines rely on samples downloaded from demo pages whose matching conditions are unverified. The paper should provide error bars or significance tests for WER, SMOS, and CMOS, describe the exact procedure for generating the parallel demo-page samples, and release the evaluation code and prompt lists.
minor comments (6)
  1. [Footnote 1] There is a typo: 'geeneration' should be 'generation'.
  2. [§4.2] The vocoder is inconsistently referred to as both 'BigGAN' and 'BiGVGAN'; the correct name is BigVGAN.
  3. [Figure 2] The axes of Figure 2 are not labeled; the caption and text explain that '0' means no DPO, but the figure should include unit labels for the number of DPO pairs and for the WER and SMOS scales.
  4. [§4.4] The sentence 'outperformed all previous SOTA models of 1.81 in Seed-TTS test-en' is grammatically unclear and should read 'and 1.81% in Seed-TTS test-en'.
  5. [Abstract and §3.1] The abstract says the acoustic model is trained 'given reference audio prosody and phone durations', but Section 3.1 describes inputs consisting only of the expanded phoneme sequence and a speaker embedding; please clarify that reference prosody enters through the duration predictor while the acoustic model receives only duration-expanded phonemes and timbre.
  6. [§3.4] The sentence 'Since annotators only need to listen to each audio sample once to make a preference judgment' appears twice in the same paragraph; one occurrence should be removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FlexSpeech's claims rest on independent training and evaluation, with only a non-load-bearing self-citation.

full rationale

FlexSpeech's derivation chain is not circular. The duration predictor is trained with next-token cross-entropy (Eqs. 12-14) and aligned by DPO (Eq. 16) using win/lose pairs whose positives are ground-truth durations and whose negatives are SFT-model predictions (Section 5); the style-transfer test set is explicitly disjoint from those pairs, so the reported WER/SMOS curves are external measurements, not restatements of the training objective. The acoustic model is a standard OT-CFM generator (Eq. 5) conditioned on expanded phoneme sequence and speaker embedding, and its stability/naturalness scores come from ASR and human listeners, not from the training loss. The only self-citation (Ref. [54], related work) is background and does not carry any load-bearing argument. The zero-shot evaluation wording in Section 4.1 ('we use the target speaker's phonemes and durations as prompts') leaves open whether the prompt is the target utterance itself; if it were, the WER/SMOS numbers would not be zero-shot, but that is an experimental-protocol ambiguity, not a demonstrated Eq-to-Eq reduction, so it does not constitute circularity under the stated rules.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim depends on standard generative-modeling assumptions (flow matching, DPO) and on empirical shortcuts: forced-alignment duration labels, ASR-based WER as a stability proxy, and the assumption that roughly 100 preference pairs transfer style. No new physical or conceptual entity is introduced; the learned feature_ref is an internal representation without a falsifiable handle outside the model, so it is not listed as an invented entity. Hand-chosen hyperparameters are listed as free parameters.

free parameters (5)
  • loss weight lambda_dur = 10.0
    Weights the duration cross-entropy term in Eq. 14 relative to the mask loss; no sensitivity analysis is provided.
  • CFG strength alpha = 2
    Classifier-free guidance extrapolation coefficient in Eq. 6; hand-set for inference.
  • duration sampling hyperparameters = top-k=6, top-p=0.5, temperature=0.9, repetition penalty=1.0
    Used for autoregressive duration sampling in Section 4.2; chosen by hand without ablation.
  • duration cutoff threshold = 99 frames
    All samples with phoneme duration exceeding 99 are skipped during training, which affects long-phoneme behavior.
  • DPO KL coefficient beta
    Beta in Eq. 16 is not reported, yet it controls the strength of preference alignment and is a central tuning choice.
assumptions (6)
  • standard math Flow matching and OT-CFM losses are valid for training the acoustic model
    Invoked in Eqs. 2-5 and credited to Lipman et al.; this is standard background.
  • standard math Bradley-Terry model and DPO reparameterization are valid for preference data
    Invoked in Section 2.2, Eqs. 7-11, and applied to duration pairs in Eq. 16.
  • domain assumption Montreal Forced Aligner produces accurate phoneme boundaries on Emilia and StoryTTS audio
    Used to extract all duration labels in Section 4.2 and to construct ground-truth positive examples in DPO.
  • domain assumption Whisper-large-v3 and Paraformer-zh transcriptions reliably measure intelligibility as WER
    Section 4.3 uses these ASR systems as the sole objective stability metric.
  • domain assumption WavLM speaker verification cosine similarity reflects perceived speaker similarity
    Section 4.3 computes SIM-O with a WavLM-based model and treats it as a speaker similarity metric.
  • domain assumption A reference mel-spectrogram clip carries enough prosody and timbre for style transfer
    The duration model and acoustic model both condition on reference mel features in Sections 3.2 and 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlexSpeech: Towards Stable, Controllable and Expressive Text-to-Speech." pith.science (2026). https://pith.science/paper/5SCWZ6Z7

@misc{pith2026250505159,
  author       = {Pith},
  title        = {Pith review of: FlexSpeech: Towards Stable, Controllable and Expressive Text-to-Speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5SCWZ6Z7}},
  note         = {Machine review of arXiv:2505.05159}
}
read the original abstract

Current speech generation research can be categorized into two primary classes: non-autoregressive and autoregressive. The fundamental distinction between these approaches lies in the duration prediction strategy employed for predictable-length sequences. The NAR methods ensure stability in speech generation by explicitly and independently modeling the duration of each phonetic unit. Conversely, AR methods employ an autoregressive paradigm to predict the compressed speech token by implicitly modeling duration with Markov properties. Although this approach improves prosody, it does not provide the structural guarantees necessary for stability. To simultaneously address the issues of stability and naturalness in speech generation, we propose FlexSpeech, a stable, controllable, and expressive TTS model. The motivation behind FlexSpeech is to incorporate Markov dependencies and preference optimization directly on the duration predictor to boost its naturalness while maintaining explicit modeling of the phonetic units to ensure stability. Specifically, we decompose the speech generation task into two components: an AR duration predictor and a NAR acoustic model. The acoustic model is trained on a substantial amount of data to learn to render audio more stably, given reference audio prosody and phone durations. The duration predictor is optimized in a lightweight manner for different stylistic variations, thereby enabling rapid style transfer while maintaining a decoupled relationship with the specified speaker timbre. Experimental results demonstrate that our approach achieves SOTA stability and naturalness in zero-shot TTS. More importantly, when transferring to a specific stylistic domain, we can accomplish lightweight optimization of the duration module solely with about 100 data samples, without the need to adjust the acoustic model, thereby enabling rapid and stable style transfer.

Figures

Figures reproduced from arXiv: 2505.05159 by the authors.

Figure 1
Figure 1. The overview of our FlexSpeech. (a). The architecture of FlexSpeech acoustic model. The phoneme embeddings, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The WER and SMOS results of rapid style transfer. ‘0’ [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The interface of the annotation system [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Phoneme-level duration control [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Sentence-level duration control [43] Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2024. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers. In The Twelfth International…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 33 canonical work pages

  1. [1]

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, and et al. 2024. Seed-TTS: A Family of High-Quality Versatile Speech Generation Models. CoRR abs/2406.02430 (2024). arXiv:2406.02430

    arXiv 2024

  2. [2]

    Tyers, and Gregor Weber

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common Voice: A Massively-Multilingual Speech Corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020. European Language R...

    work page 2020

  3. [3]

    James Betker. 2023. Better speech synthesis through scaling. CoRR abs/2305.07243 (2023). arXiv:2305.07243

    arXiv 2023

  4. [4]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952), 324–345

    1952

  5. [5]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. 2022. MaskGIT: Masked Generative Image Transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 11305–11315

    work page 2022

  6. [6]

    Chen Chen, Yuchen Hu, Wen Wu, Helin Wang, Eng Siong Chng, and Chao Zhang

    work page

  7. [7]

    Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. 2024. VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers. CoRR abs/2406.05370 (2024). arXiv:2406.05370

    arXiv 2024

  8. [8]

    Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. 2018. Neural Ordinary Differential Equations. In Advances in Neural Information FlexSpeech: Towards Stable, Controllable and Expressive Text-to-Speech Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canad...

    work page 2018

Show all 61 references
  1. [9]

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. 2024. F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. CoRR abs/2410.06885 (2024). arXiv:2410.06885

    2024 arXiv

  2. [10]

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2023. High Fidelity Neural Audio Compression. Trans. Mach. Learn. Res. 2023 (2023)

    2023

  3. [11]

    Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. [n. d.]. ECAPA- TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. In 21st Annual Conference of the International Speech Communication Association, Interspeech 2020, Virt...

    2020

  4. [12]

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. 2024. CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens. CoRR abs/2407.05407 (2024...

    2024 arXiv

  5. [13]

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. 2024. CosyVoice 2: Scalable Streaming Speech Synt...

    2024 arXiv

  6. [14]

    Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, Yanqing Liu, Sheng Zhao, and Naoyuki Kanda. 2024. E2 TTS: Embarrassingly Easy Fully Non- Autoregressive Zero-Shot TTS. InIEEE Spoken Language Tec...

    2024

  7. [15]

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. Scaling Rec- tified Flow Transformers for High-Resolution Image...

    2024

  8. [16]

    Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. 2022. Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition. In 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon,...

    2022

  9. [17]

    Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, and Xiangang Li. 2021. Didis- peech: A Large Scale Mandarin Speech Corpus. InIEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP ...

    2021

  10. [18]

    Jonathan Ho and Tim Salimans. 2022. Classifier-Free Diffusion Guidance. CoRR abs/2207.12598 (2022). arXiv:2207.12598

    2022 arXiv

  11. [19]

    Yuchen Hu, Chen Chen, Siyin Wang, Eng Siong Chng, and Chao Zhang. 2024. Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimization. CoRR abs/2407.02243 (2024). arXiv:2407.02243

    2024 arXiv

  12. [20]

    Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang, Chen Zhang, Pengfei Wei, Chunfeng Wang, Xiang Yin, Zejun Ma, and Zhou Zhao. 2024. Mega-TTS 2: Boosting Prompting Mechanisms for Zero- Shot Speech Synthesis. In The Twelfth International Confere...

    2024

  13. [21]

    Ziyue Jiang, Yi Ren, Ruiqi Li, Shengpeng Ji, Zhenhui Ye, Chen Zhang, Jionghao Bai, Xiaoda Yang, Jialong Zuo, Yu Zhang, Rui Liu, Xiang Yin, and Zhou Zhao

  14. [22]

    Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, Zejun Ma, and Zhou Zhao. 2023. Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias.CoRR abs/2306.03509 (2023). arXiv:2306.03509

    2023 arXiv

  15. [23]

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xi- angyang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and Sheng Zhao

  16. [24]

    Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour

  17. [25]

    Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional Variational Autoen- coder with Adversarial Learning for End-to-End Text-to-Speech. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event (Proceedings of Mac...

    2021

  18. [26]

    In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024

    NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net

    2024

  19. [27]

    Mateusz Lajszczak, Guillermo Cámbara, Yang Li, Fatih Beyhan, Arent van Korlaar, Fan Yang, Arnaud Joly, Álvaro Martín-Cortinas, Ammar Abbas, Adam Michalski, Alexis Moinet, Sri Karlapati, Ewa Muszynska, Haohan Guo, Bartosz Putrycz, Soledad López Gambino, Kayeon Yoo, Elena Sokolo...

  20. [28]

    Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon

  21. [29]

    Yixing Li, Ruobing Xie, Xingwu Sun, Yu Cheng, and Zhanhui Kang. 2024. Con- tinuous Speech Tokenizer in Text To Speech. CoRR abs/2410.17081 (2024). arXiv:2410.17081

    2024 arXiv

  22. [30]

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. 2023. High-Fidelity Audio Compression with Improved RVQGAN. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023,...

    2023

  23. [31]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. 2023. Flow Matching for Generative Modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net

    2023

  24. [32]

    CoRR abs/2402.08093 (2024)

    BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data. CoRR abs/2402.08093 (2024). arXiv:2402.08093

    2024 arXiv

  25. [33]

    Yanqing Liu, Zhihang Xu, Gang Wang, Kuan Chen, Bohan Li, Xu Tan, Jinzhu Li, Lei He, and Sheng Zhao. 2021. DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021. In The Blizzard Challenge 2021, virtual, October 23, 2021. ISCA

    2021

  26. [34]

    In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023

    BigVGAN: A Universal Neural Vocoder with Large-Scale Training. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net

    2023

  27. [35]

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. 2024. Finite Scalar Quantization: VQ-VAE Made Simple. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

    2024

  28. [36]

    Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yijin Xing. 2024. Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis. CoRR abs/2411.01156 (2024). arXiv:2411.01156

    2024 arXiv

  29. [37]

    Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail A. Kudinov. 2021. Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event (Proceedi...

    2021

  30. [38]

    Sen Liu, Yiwei Guo, Xie Chen, and Kai Yu. 2024. StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations. InIEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 20...

    2024

  31. [39]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Lan- guage Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural ...

    2023

  32. [40]

    Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, and Furu Wei

  33. [41]

    CoRR abs/2407.08551 (2024)

    Autoregressive Speech Synthesis without Vector Quantization. CoRR abs/2407.08551 (2024). arXiv:2407.08551

    2024 arXiv

  34. [42]

    Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, R

    Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, R. J. Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu. 2018. Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Pred...

    2018

  35. [43]

    William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Trans- formers. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. IEEE, 4172–4182

    2023

  36. [44]

    Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watan- abe, and Dong Yu. 2024. Preference Alignment Improves Language Model-Based TTS. CoRR abs/2409.12403 (2024). arXiv:2409.12403

    2024 arXiv

  37. [45]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Su- pervision. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA (Proceedings of Ma...

    2023

  38. [46]

    Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. 2017. Tacotron: Towards End-to-End Speech Synthesis. In 18th Annual...

    2017

  39. [47]

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2021. FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021

    2021

  40. [48]

    Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. 2019. FastSpeech: Fast, Robust and Controllable Text to Speech. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 201...

    2019

  41. [49]

    Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, Zhou Zhao, Shinji Watan- abe, and Helen Meng. 2023. UniAudio: An Audio Foundation Model Toward Universal Audio Generation. CoRR abs/2310.00704 (202...

    2023 arXiv

  42. [50]

    Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2024. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers. In The Twelfth International Conference on Learning Representations, ICLR...

    2024

  43. [51]

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2022. SoundStream: An End-to-End Neural Audio Codec. IEEE ACM Trans. Audio Speech Lang. Process. 30 (2022), 495–507

    2022

  44. [52]

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. CoRR abs/2301.02111 (2023). arXiv:2301.02111

    2023 arXiv

  45. [53]

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2024. SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net

    2024

  46. [54]

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Ji- achen Zheng, Qiang Zhang, Shunsi Zhang, and Zhizheng Wu. 2024. MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer. CoRR abs/2409.00750 (2024). arXiv:2409.00750

    2024 arXiv

  47. [55]

    Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. 2023. HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec. CoRR abs/2305.02765 (2023). arXiv:2305.02765

    2023 arXiv

  48. [57]

    Zhen Ye, Zeqian Ju, Haohe Liu, Xu Tan, Jianyi Chen, Yiwen Lu, Peiwen Sun, Jiahao Pan, Weizhen Bian, Shulin He, Wei Xue, Qifeng Liu, and Yike Guo. 2024. Flash- Speech: Efficient Zero-Shot Speech Synthesis. In Proceedings of the 32nd ACM International Conference on Multimedia, M...

    2024

  49. [59]

    Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2024. SpeechAlign: Aligning Speech Generation to Hu- man Preferences. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 202...

    2024

  50. [61]

    Xinfa Zhu, Wenjie Tian, and Lei Xie. 2024. Autoregressive Speech Synthesis with Next-Distribution Prediction. CoRR abs/2412.16846 (2024). arXiv:2412.16846 A Details For Preference Annotation Initially, we utilize automated techniques to pre-select the data. In this stage, samp...

    2024

  51. [2023]

    Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision. Trans. Assoc. Comput. Linguistics 11 (2023), 1703–1718

    2023

  52. [2024]

    CoRR abs/2406.00654 (2024)

    Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback. CoRR abs/2406.00654 (2024). arXiv:2406.00654

    2024 arXiv

  53. [2025]

    CoRR abs/2502.18924 (2025)

    Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis. CoRR abs/2502.18924 (2025). arXiv:2502.18924

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.