Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

UniVoice claims a single 0.4B language model can do both speech recognition and zero-shot speech synthesis, matching or beating task-specific systems while avoiding the information loss of discrete speech tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:26 UTC pith:VACFFFZM

load-bearing objection A genuinely new single-model AR-ASR + flow-matching TTS combination, with results that support 'competitive unified model' but not the abstract's 'achieve or exceed' claim. the 3 major comments →

arxiv 2510.04593 v3 pith:VACFFFZM submitted 2025-10-06 eess.AS cs.SD

UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

Wenhao Guan , Zhikang Niu , Ziyue Jiang , Kaidi Wang , Peijie Chen , Qingyang Hong , Lin Li , Xie Chen This is my paper
classification eess.AS cs.SD
keywords unified speech modelautoregressive ASRflow matching TTSdual attention maskzero-shot voice cloningcontinuous speech representationlarge language modelspeech infilling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to prove that transcribing audio into text and synthesizing speech from text can be done by one transformer rather than separate models. It keeps speech in a continuous representation and alternates between two attention patterns: causal attention for autoregressive recognition, and bidirectional attention for flow-matching generation. Trained on roughly 50,000 hours of speech, the 0.4B-parameter model reports word-error rates of 3.0/6.3 on standard test sets and zero-shot voice cloning with 0.56 speaker similarity and a 4.06 WER on its own synthesized speech. If the results hold, unified speech understanding and generation is practical at small scale, and the usual quantization bottleneck in speech language models can be bypassed. The authors also show the approach outdoes existing unified models on TTS robustness.

Core claim

The central claim is that a single LLM can jointly learn ASR and TTS by operating directly on continuous mel-spectrograms, as long as the model can switch its attention mask for the two tasks and treat text as a prefix condition for speech infilling. On standard English test sets, the unified model reports 3.0% and 6.3% WER, within a small margin of a dedicated ASR variant and far stronger than prior unified models. For synthesis, it reports TTS WER of 4.06, speaker similarity 0.56, and UTMOS 3.72, beating the best unified baseline by 12% relative WER and even exceeding a 7B-parameter unified model. The authors interpret joint training as beneficial to intelligibility, since the unified mode

What carries the argument

The dual attention mask: the same transformer uses a causal mask when decoding text from audio, and a bidirectional mask when denoising speech from a flow-matching path. Supporting it are a pretrained audio encoder feeding continuous features, an adapter network, and a text-prefix-conditioned speech infilling formulation (masking a random 70–100% of mel frames and reconstructing them under a flow-matching loss). The optimal-transport conditional flow-matching objective is combined with a small-weighted autoregressive LM loss.

Load-bearing premise

The results assume the pretrained audio encoder contributes the majority of acoustic and speaker knowledge; the paper never states whether that encoder is frozen or ablates it, so the amount of capability created by the unified training itself remains unmeasured.

What would settle it

Train an identical UniVoice model but replace the pretrained audio encoder with a randomly initialized encoder (or freeze it during training) and measure ASR word error rate and speaker similarity; if WER degrades sharply or similarity collapses, the measured performance is inherited from the encoder rather than the unified LLM training.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Speech-language models can operate on continuous audio without quantization, avoiding the acoustic detail loss of codec tokens.
  • A unified model trained with two losses acts like a regularizer: TTS output is more intelligible (lower WER) than a TTS-only variant.
  • Task-specific performance does not require task-specific models; one small backbone can cover both ends of the speech loop.
  • The dual-attention joint training recipe may extend to other paired speech-text tasks like translation, editing, or dialogue.
  • Strong unified performance at 0.4B parameters and 50K hours suggests scaling up data and model size may further close the gap to specialized systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pretrained audio encoder is frozen, then the unified model's acoustic perception largely inherits from that encoder's pretraining; the paper's contribution would then be the integration and generation side, not raw ASR accuracy. A random-encoder ablation would settle this.
  • The dual attention mask is a general 'task-switchable context' mechanism; it could enable other non-autoregressive generative tasks such as masked speech editing, voice conversion, or partial inpainting within the same LLM with only the mask and loss changed.
  • Continuous mel-spectrogram modeling with flow matching suggests a unified speech LLM need not use a separate codec tokenizer; only the neural vocoder remains as the external conversion stage, potentially simplifying architectures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes UniVoice, a unified 0.4B-parameter speech LLM that performs ASR through autoregressive next-token prediction and zero-shot TTS through flow matching on continuous mel-spectrogram features. The architecture uses a Whisper-large-v3-turbo encoder plus adapter for speech input, a SmolLM2-360M transformer backbone, and a dual attention scheme: causal masking for ASR, bidirectional masking for TTS. TTS is cast as text-prefix-conditioned speech infilling, enabling zero-shot voice cloning from a reference utterance. The model is trained on LibriHeavy (~50K hours) and evaluated on LibriSpeech test-clean/other for ASR and LibriSpeech-PC for zero-shot TTS. The main claimed contributions are the continuous-representation unification of AR and flow-matching paradigms, the dual attention mechanism, and text-prefix-guided infilling, with a stated result of matching or exceeding single-task systems in both modalities.

Significance. If the empirical claims are accurate and reproducible, this is a useful proof of concept: a single small LLM can jointly perform causal ASR and flow-matching TTS on continuous speech representations without discrete tokenization, while remaining competitive with prior unified models. The proposed dual attention mask is a clean solution to the AR/FM incompatibility, and the text-prefix infilling formulation extends Voicebox-style in-context TTS to an LLM backbone. The paper includes useful ablations of the infilling objective, attention mask, and loss weighting, and the authors commit to releasing code and checkpoints. That said, the measured advantages are mostly relative to other unified systems, not to single-task specialized systems, so the broader significance depends on substantially rescoping the paper's central claims.

major comments (3)
  1. [Abstract; Introduction, contributions; Table 1] The abstract and the third contribution bullet state that UniVoice 'can achieve or exceed current single-task modeling methods' in both ASR and zero-shot TTS. This is directly contradicted by Table 1. For TTS, UniVoice reports SIM 0.56, UTMOS 3.72, and WER 4.06, while F5-TTS reports 0.66, 3.84, and 2.54; CosyVoice reports 0.66, 4.17, and 3.59; MaskGCT reports 0.66, 3.85, and 2.49. For ASR, UniVoice reports 3.0/6.3 WER on test-clean/other, while Whisper-large-v3-turbo reports 1.9/3.5, Whisper-large-v2 reports 2.7/5.2, and Zipformer reports 2.0/4.4. The paper's own Introduction wording ('comparable' and 'competitive') is more accurate than the abstract. Because this overclaim is the headline result, the claim must be either removed, restricted to the unified-model comparison, or backed by statistical evidence; no error bars, confidence intervals, or multiple-seed results are reported anywh
  2. [UniVoice Model; Implementation Details; Figure 1] The audio front-end is specified only as 'We use the Whisper-large-v3-turbo encoder as the audio encoder.' The manuscript never states whether this encoder is frozen or fine-tuned during UniVoice training, and no ablation removes, replaces, or freezes it. This matters because the same encoder supplies both the ASR input and the TTS reference-speaker conditioning in Figure 1. As a result, the reported ASR WER (3.0/6.3) and speaker similarity (0.56) could substantially inherit Whisper's pretrained acoustic and speaker representations rather than being attributable to the proposed unified LLM training. The authors should state the freeze/fine-tune status and provide an ablation that isolates the encoder's contribution, for example a randomly initialized encoder or a frozen-versus-trainable comparison.
  3. [Experiments; Evaluation metrics] All comparisons in Table 1 are point estimates from single training runs and single evaluation passes, and the subjective tests use only 8 listeners and 20 utterances. Claims such as 'significant improvements' over unified baselines, 'parity' with single-task TTS on similarity, and 'slight degradation' in ASR are therefore not statistically supported. Since the paper's central message depends on relative comparisons, at least bootstrap confidence intervals for WER/SIM/UTMOS or multiple-seed results should be provided, especially where differences are small (e.g., SIM 0.56 vs 0.55 for NaturalSpeech2, UTMOS 3.72 vs 3.68 for VALL-E).
minor comments (3)
  1. [Eq. (3)] The flow-matching loss in Eq. (3) is missing the squared norm that appears in Eqs. (4) and (6). This is a typo, but it makes the objective inconsistent with the standard conditional flow-matching loss.
  2. [Inference, Eq. (8)] The duration ratio is defined as len(Ygen)/len(Yref), but it is unclear whether 'len' is measured in characters, subword tokens, or some other unit. Please specify, since this directly controls the synthesized speech length.
  3. [Appendix B] The attention-mask ablation in Table 4 is reported only for the TTS-only variant, not for the full unified UniVoice model. The conclusion that bidirectional masks validate the unified design would be stronger if the same comparison were shown in the joint-training setting.

Circularity Check

0 steps flagged

No circular derivation found: UniVoice's results come from training and external benchmark evaluation, not from fitting the claimed output into its own loss or from a load-bearing self-citation chain.

full rationale

The paper's central claims are empirical: UniVoice is trained on LibriHeavy with a combined autoregressive ASR loss and conditional flow-matching TTS loss (Eqs. 2, 6, 7), then evaluated on LibriSpeech test sets and LibriSpeech-PC using external metrics (WER via Whisper-large-v3, SIM via WavLM, UTMOS). No prediction is derived from a fitted parameter that was itself fit to the same target; hyperparameters such as lambda, CFG weight, mask ratio, and NFE are training/inference settings, not inputs that make the benchmark numbers true by construction. The self-comparisons of UniVoice vs. UniVoice-TTS and UniVoice-ASR are legitimate ablations isolating multitask effects. The self-citations (Guan et al. 2024a,b,c; 2025) appear only in related-work context and are not load-bearing for any claim. The use of the Whisper-large-v3-turbo encoder as a front-end is a design choice and an external pretrained component, not a self-cited result; whether it dominates performance is an attribution question, not definitional circularity. The abstract's 'achieve or exceed' language conflicts with the paper's own Table 1, but that is an internal-consistency/correctness concern, not a circular-derivation concern. Accordingly, no circular step can be exhibited with quotes and a specific reduction, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

No new physical entities are introduced. The assumptions are representational and design-level: Whisper and mel features suffice for both tasks, and a single transformer with two attention masks can serve both. The free parameters are hyperparameters tuned by hand or directly on test sets, not constants derived from first principles.

free parameters (4)
  • lambda (loss weight) = 0.005
    Balances ASR LM loss and TTS flow-matching loss in Eq. 7; chosen by comparing 0.005 vs 0.01 on LibriSpeech test sets (Supp. Table 3), not on a held-out validation set.
  • TTS training mask ratio = 70%-100% mel frames
    Randomly masks 70-100% of mel frames during TTS infilling training; a manual choice affecting difficulty and voice-cloning quality.
  • CFG weight and condition drop rates = CFG=2, text drop=0.2, masked speech drop=0.3
    Inference-time classifier-free guidance strength and training-time dropping of text and masked-speech conditions; set by hand, with no ablation reported for these values.
  • TTS inference NFE = 32
    Number of flow-ODE function evaluations for TTS inference; set by hand following F5-TTS practice.
axioms (6)
  • domain assumption The Whisper-large-v3-turbo encoder supplies a shared, information-sufficient continuous representation for both ASR and TTS.
    The entire pipeline consumes Whisper encoder output (Implementation details; Fig. 1). The paper does not state whether the encoder is frozen or fine-tuned, nor does it ablate it.
  • domain assumption 80-bin mel-spectrogram is a sufficient generation target, with BigVGAN as vocoder.
    TTS is trained and inferred in the mel domain; waveform quality is largely delegated to the BigVGAN vocoder.
  • standard math OT-CFM conditional flow matching (Eqs. 5-6) is a valid objective for speech generation.
    Borrowed from Lipman et al. 2022 and F5-TTS; no re-derivation is needed.
  • domain assumption LibriHeavy training transfers to LibriSpeech test-clean/test-other and LibriSpeech-PC for both ASR and zero-shot TTS.
    All experiments use Libri-derived audiobook English; results may not generalize to other domains or languages.
  • ad hoc to paper One transformer can serve both causal ASR and bidirectional flow-matching TTS without architectural separation.
    Core design hypothesis behind the dual attention mechanism; ablations cover mask type for TTS but not encoder sharing or a two-model baseline.
  • domain assumption Reference transcript is available at cloning time, and duration ratio len(Ygen)/len(Yref) controls generated length.
    Inference follows the F5-TTS duration-ratio formula (Eq. 8); assumes proportional durations and that a reference transcript is always available.

pith-pipeline@v1.3.0-alltime-deepseek · 15509 in / 17731 out tokens · 139022 ms · 2026-08-04T11:26:19.892978+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have demonstrated promising performance in both automatic speech recognition (ASR) and text-to-speech (TTS) systems, gradually becoming the mainstream approach. However, most current approaches address these tasks separately rather than through a unified framework. This work aims to integrate these two tasks into one unified model. Although discrete speech tokenization enables joint modeling, its inherent information loss limits performance in both recognition and generation. In this work, we present UniVoice, a unified LLM framework through continuous representations that seamlessly integrates speech recognition and synthesis within a single model. Our approach combines the strengths of autoregressive modeling for speech recognition with flow matching for high-quality generation. To mitigate the inherent divergence between autoregressive and flow-matching models, we further design a dual attention mechanism, which switches between a causal mask for recognition and a bidirectional attention mask for synthesis. Furthermore, the proposed text-prefix-conditioned speech infilling method enables high-fidelity zero-shot voice cloning. Experimental results demonstrate that our method can achieve or exceed current single-task modeling methods in both ASR and zero-shot TTS tasks. This work explores new possibilities for end-to-end speech understanding and generation. Code is available at https://github.com/gwh22/UniVoice.

Figures

Figures reproduced from arXiv: 2510.04593 by Kaidi Wang, Lin Li, Peijie Chen, Qingyang Hong, Wenhao Guan, Xie Chen, Zhikang Niu, Ziyue Jiang.

Figure 1
Figure 1. Figure 1: An overview of UniVoice model. Blue elements (blocks and lines) denote ASR components, while green elements [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Two variants of TTS model designs in UniVoice. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models

    cs.CL 2026-04 conditional novelty 6.0

    Synthetic data for low-resource spoken language models creates a Stability-Expressivity Gap that DGSA and TDSC self-alignment close, enabling SOTA Thai TTS and first Lao zero-shot voice cloning.

Reference graph

Works this paper leans on

68 extracted references · 47 linked inside Pith · cited by 1 Pith paper

  1. [1]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  2. [2]

    B.; Lozhkov, A.; Bakouch, E.; Blázquez, G

    Allal, L. B.; Lozhkov, A.; Bakouch, E.; Blázquez, G. M.; Penedo, G.; Tunstall, L.; Marafioti, A.; Kydlíček, H.; Lajarín, A. P.; Srivastav, V.; Lochner, J.; Fahlgren, C.; Nguyen, X.-S.; Fourrier, C.; Burtenshaw, B.; Larcher, H.; Zhao, H.; Zakka, C.; Morlon, M.; Raffel, C.; von Werra, L.; and Wolf, T. 2025. SmolLM2: When Smol Goes Big -- Data-Centric Traini...

    Pith/arXiv arXiv 2025

  3. [3]

    Anastassiou, P.; Chen, J.; Chen, J.; Chen, Y.; Chen, Z.; Chen, Z.; Cong, J.; Deng, L.; Ding, C.; Gao, L.; et al. 2024. Seed-TTS: A Family of High-Quality Versatile Speech Generation Models. arXiv preprint arXiv:2406.02430

    Pith/arXiv arXiv 2024

  4. [4]

    Ao, J.; Wang, R.; Zhou, L.; Wang, C.; Ren, S.; Wu, Y.; Liu, S.; Ko, T.; Li, Q.; Zhang, Y.; et al. 2021. SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.07205

    Pith/arXiv arXiv 2021

  5. [5]

    Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    Pith/arXiv arXiv 2023

  6. [6]

    Bai, Y.; Chen, J.; Chen, J.; Chen, W.; Chen, Z.; Ding, C.; Dong, L.; Dong, Q.; Du, Y.; Gao, K.; et al. 2024. Seed-ASR: Understanding diverse speech and contexts with llm-based speech recognition. arXiv preprint arXiv:2407.04675

    Pith/arXiv arXiv 2024

  7. [7]

    Borsos, Z.; Marinier, R.; Vincent, D.; Kharitonov, E.; Pietquin, O.; Sharifi, M.; Roblek, D.; Teboul, O.; Grangier, D.; Tagliasacchi, M.; et al. 2023. AudioLM: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 2523--2533

    2023

  8. [8]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Proc. NeurIPS, 33: 1877--1901

    2020

  9. [9]

    Chen, S.; Liu, S.; Zhou, L.; Liu, Y.; Tan, X.; Li, J.; Zhao, S.; Qian, Y.; and Wei, F. 2024 a . VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers. arXiv preprint arXiv:2406.05370

    Pith/arXiv arXiv 2024

  10. [10]

    Chen, S.; Wang, C.; Chen, Z.; Wu, Y.; Liu, S.; Chen, Z.; Li, J.; Kanda, N.; Yoshioka, T.; Xiao, X.; et al. 2022 a . WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6): 1505--1518

    2022

  11. [11]

    Chen, S.; Wu, Y.; Wang, C.; Liu, S.; Tompkins, D.; Chen, Z.; and Wei, F. 2022 b . Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058

    Pith/arXiv arXiv 2022

  12. [12]

    Chen, W.; Ma, Z.; Yan, R.; Liang, Y.; Li, X.; Xu, R.; Niu, Z.; Zhu, Y.; Yang, Y.; Liu, Z.; et al. 2024 b . Slam-Omni: Timbre-controllable voice interaction system with single-stage training. arXiv preprint arXiv:2412.15649

    Pith/arXiv arXiv 2024

  13. [13]

    Chen, Y.; Niu, Z.; Ma, Z.; Deng, K.; Wang, C.; Zhao, J.; Yu, K.; and Chen, X. 2024 c . F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885

    Pith/arXiv arXiv 2024

  14. [14]

    Chen, Y.-C.; Chi, P.-H.; Yang, S.-w.; Chang, K.-W.; Lin, J.-h.; Huang, S.-F.; Liu, D.-R.; Liu, C.-L.; Lee, C.-K.; and Lee, H.-y. 2021. SpeechNet: A universal modularized model for speech processing tasks. arXiv preprint arXiv:2105.03070

    Pith/arXiv arXiv 2021

  15. [15]

    Conneau, A.; Baevski, A.; Collobert, R.; Mohamed, A.; and Auli, M. 2020. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979

    Pith/arXiv arXiv 2020

  16. [16]

    Copet, J.; Kreuk, F.; Gat, I.; Remez, T.; Kant, D.; Synnaeve, G.; Adi, Y.; and D \'e fossez, A. 2024. Simple and controllable music generation. Proc. NeurIPS, 36

    2024

  17. [17]

    R.; Juki \'c , A.; Langman, R.; Balam, J.; and Ginsburg, B

    Dhawan, K.; Koluguri, N. R.; Juki \'c , A.; Langman, R.; Balam, J.; and Ginsburg, B. 2024. Codec-ASR: Training performant automatic speech recognition systems with discrete speech representations. arXiv preprint arXiv:2407.03495

    Pith/arXiv arXiv 2024

  18. [18]

    Du, Z.; Chen, Q.; Zhang, S.; Hu, K.; Lu, H.; Yang, Y.; Hu, H.; Zheng, S.; Gu, Y.; Ma, Z.; et al. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407

    Pith/arXiv arXiv 2024

  19. [19]

    Du, Z.; Wang, J.; Chen, Q.; Chu, Y.; Gao, Z.; Li, Z.; Hu, K.; Zhou, X.; Xu, J.; Ma, Z.; et al. 2023. LauraGPT: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673

    Pith/arXiv arXiv 2023

  20. [20]

    Fu, C.; Lin, H.; Long, Z.; Shen, Y.; Zhao, M.; Zhang, Y.; Dong, S.; Wang, X.; Yin, D.; Ma, L.; et al. 2024. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211

    Pith/arXiv arXiv 2024

  21. [21]

    Gao, Z.; Zhang, S.; McLoughlin, I.; and Yan, Z. 2022. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. arXiv preprint arXiv:2206.08317

    Pith/arXiv arXiv 2022

  22. [22]

    Ghosal, D.; Majumder, N.; Mehrish, A.; and Poria, S. 2023. Text-to-audio generation using instruction-tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731

    Pith/arXiv arXiv 2023

  23. [23]

    Guan, W.; Li, Y.; Li, T.; Huang, H.; Wang, F.; Lin, J.; Huang, L.; Li, L.; and Hong, Q. 2024 a . MM-TTS: Multi-Modal Prompt Based Style Transfer for Expressive Text-to-Speech Synthesis. In Proc. AAAI, volume 38, 18117--18125

    2024

  24. [24]

    Guan, W.; Su, Q.; Zhou, H.; Miao, S.; Xie, X.; Li, L.; and Hong, Q. 2024 b . ReFlow-TTS: A rectified flow model for high-fidelity text-to-speech. In Proc. ICASSP, 10501--10505. IEEE

    2024

  25. [25]

    Guan, W.; Wang, K.; Zhou, W.; Wang, Y.; Deng, F.; Wang, H.; Li, L.; Hong, Q.; and Qin, Y. 2024 c . LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation. In Proc. Interspeech, 4813--4817

    2024

  26. [26]

    Guo, H.-H.; Liu, K.; Shen, F.-Y.; Wu, Y.-C.; Xie, F.-L.; Xie, K.; and Xu, K.-T. 2024 a . Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.03283

    Pith/arXiv arXiv 2024

  27. [27]

    Guo, Y.; Du, C.; Ma, Z.; Chen, X.; and Yu, K. 2024 b . VoiceFlow: Efficient text-to-speech with rectified flow matching. In Proc. ICASSP, 11121--11125. IEEE

    2024

  28. [28]

    He, H.; Shang, Z.; Wang, C.; Li, X.; Gu, Y.; Hua, H.; Liu, L.; Yang, C.; Li, J.; Shi, P.; et al. 2024. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361

    Pith/arXiv arXiv 2024

  29. [29]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. Proc. NeurIPS, 33: 6840--6851

    2020

  30. [30]

    H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A

    Hsu, W.-N.; Bolte, B.; Tsai, Y.-H. H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29: 3451--3460

    2021

  31. [31]

    Jiang, Z.; Ren, Y.; Li, R.; Ji, S.; Ye, Z.; Zhang, C.; Jionghao, B.; Yang, X.; Zuo, J.; Zhang, Y.; et al. 2025. Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis. arXiv preprint arXiv:2502.18924

    Pith/arXiv arXiv 2025

  32. [32]

    Jiang, Z.; Ren, Y.; Ye, Z.; Liu, J.; Zhang, C.; Yang, Q.; Ji, S.; Huang, R.; Wang, C.; Yin, X.; et al. 2023. Mega-TTS: Zero-shot text-to-speech at scale with intrinsic inductive bias. arXiv preprint arXiv:2306.03509

    Pith/arXiv arXiv 2023

  33. [33]

    Ju, Z.; Wang, Y.; Shen, K.; Tan, X.; Xin, D.; Yang, D.; Liu, Y.; Leng, Y.; Song, K.; Tang, S.; et al. 2024. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100

    Pith/arXiv arXiv 2024

  34. [34]

    Kang, W.; Yang, X.; Yao, Z.; Kuang, F.; Yang, Y.; Guo, L.; Lin, L.; and Povey, D. 2024. LibriHeavy: A 50,000 hours ASR corpus with punctuation casing and context. In Proc. ICASSP, 10991--10995. IEEE

    2024

  35. [35]

    Le, M.; Vyas, A.; Shi, B.; Karrer, B.; Sari, L.; Moritz, R.; Williamson, M.; Manohar, V.; Adi, Y.; Mahadeokar, J.; et al. 2024. Voicebox: Text-guided multilingual universal speech generation at scale. Proc. NeurIPS., 36

    2024

  36. [36]

    Lee, S.-g.; Ping, W.; Ginsburg, B.; Catanzaro, B.; and Yoon, S. 2022. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.04658

    Pith/arXiv arXiv 2022

  37. [37]

    T.; Ben-Hamu, H.; Nickel, M.; and Le, M

    Lipman, Y.; Chen, R. T.; Ben-Hamu, H.; Nickel, M.; and Le, M. 2022. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747

    Pith/arXiv arXiv 2022

  38. [38]

    Liu, H.; Chen, Z.; Yuan, Y.; Mei, X.; Liu, X.; Mandic, D.; Wang, W.; and Plumbley, M. D. 2023. AudioLDM: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503

    Pith/arXiv arXiv 2023

  39. [39]

    Liu, J.; Li, C.; Ren, Y.; Chen, F.; and Zhao, Z. 2022. DiffSinger: Singing voice synthesis via shallow diffusion mechanism. In Proc. ICML, volume 36, 11020--11028

    2022

  40. [40]

    Liu, X.; Gong, C.; and Liu, Q. 2022. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003

    Pith/arXiv arXiv 2022

  41. [41]

    Ma, Z.; Yang, G.; Yang, Y.; Gao, Z.; Wang, J.; Du, Z.; Yu, F.; Chen, Q.; Zheng, S.; Zhang, S.; et al. 2024. An embarrassingly simple approach for llm with strong asr capacity. arXiv preprint arXiv:2402.08846

    Pith/arXiv arXiv 2024

  42. [42]

    Mehta, S.; Tu, R.; Beskow, J.; Sz \'e kely, \'E .; and Henter, G. E. 2024. Matcha-TTS: A fast TTS architecture with conditional flow matching. In Proc. ICASSP, 11341--11345. IEEE

    2024

  43. [43]

    Meng, L.; Zhou, L.; Liu, S.; Chen, S.; Han, B.; Hu, S.; Liu, Y.; Li, J.; Zhao, S.; Wu, X.; et al. 2024. Autoregressive speech synthesis without vector quantization. arXiv preprint arXiv:2407.08551

    Pith/arXiv arXiv 2024

  44. [44]

    Peng, P.; Huang, P.-Y.; Li, S.-W.; Mohamed, A.; and Harwath, D. 2024. Voicecraft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint arXiv:2403.16973

    Pith/arXiv arXiv 2024

  45. [45]

    Popov, V.; Vovk, I.; Gogoryan, V.; Sadekova, T.; and Kudinov, M. 2021. Grad-TTS: A diffusion probabilistic model for text-to-speech. In Proc. ICML, 8599--8608. PMLR

    2021

  46. [46]

    W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I

    Radford, A.; Kim, J. W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I. 2023. Robust speech recognition via large-scale weak supervision. In Proc. ICML, 28492--28518. PMLR

    2023

  47. [47]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1--67

    2020

  48. [48]

    K.; Asawaroengchai, C.; Nguyen, D

    Rubenstein, P. K.; Asawaroengchai, C.; Nguyen, D. D.; Bapna, A.; Borsos, Z.; Quitry, F. d. C.; Chen, P.; Badawy, D. E.; Han, W.; Kharitonov, E.; et al. 2023. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925

    Pith/arXiv arXiv 2023

  49. [49]

    Saeki, T.; Xin, D.; Nakata, W.; Koriyama, T.; Takamichi, S.; and Saruwatari, H. 2022. UTMOS: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152

    Pith/arXiv arXiv 2022

  50. [50]

    Shen, K.; Ju, Z.; Tan, X.; Liu, Y.; Leng, Y.; He, L.; Qin, T.; Zhao, S.; and Bian, J. 2023. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116

    Pith/arXiv arXiv 2023

  51. [51]

    P.; Kumar, A.; Ermon, S.; and Poole, B

    Song, Y.; Sohl-Dickstein, J.; Kingma, D. P.; Kumar, A.; Ermon, S.; and Poole, B. 2020. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456

    Pith/arXiv arXiv 2020

  52. [52]

    Tang, C.; Yu, W.; Sun, G.; Chen, X.; Tan, T.; Li, W.; Lu, L.; Ma, Z.; and Zhang, C. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289

    Pith/arXiv arXiv 2023

  53. [53]

    Tian, J.; Chen, W.; Peng, Y.; Shi, J.; Arora, S.; Bharadwaj, S.; Maekaku, T.; Shinohara, Y.; Goto, K.; Yue, X.; et al. 2025. OpusLM: A Family of Open Unified Speech Language Models. arXiv preprint arXiv:2506.17611

    Pith/arXiv arXiv 2025

  54. [54]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    Pith/arXiv arXiv 2023

  55. [55]

    Wang, C.; Chen, S.; Wu, Y.; Zhang, Z.; Zhou, L.; Liu, S.; Chen, Z.; Liu, Y.; Wang, H.; Li, J.; et al. 2023. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111

    Pith/arXiv arXiv 2023

  56. [56]

    Wang, K.; Guan, W.; Jiang, Z.; Huang, H.; Chen, P.; Wu, W.; Hong, Q.; and Li, L. 2025 a . Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion. arXiv preprint arXiv:2505.24291

    Pith/arXiv arXiv 2025

  57. [57]

    Wang, T.; Zhou, L.; Zhang, Z.; Wu, Y.; Liu, S.; Gaur, Y.; Chen, Z.; Li, J.; and Wei, F. 2024 a . VioLA: conditional language models for speech recognition, synthesis, and translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing

    2024

  58. [58]

    Wang, X.; Jiang, M.; Ma, Z.; Zhang, Z.; Liu, S.; Li, L.; Liang, Z.; Zheng, Q.; Wang, R.; Feng, X.; et al. 2025 b . Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710

    Pith/arXiv arXiv 2025

  59. [59]

    Wang, Y.; Zhan, H.; Liu, L.; Zeng, R.; Guo, H.; Zheng, J.; Zhang, Q.; Zhang, X.; Zhang, S.; and Wu, Z. 2024 b . MaskGCT: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750

    Pith/arXiv arXiv 2024

  60. [60]

    Xie, Z.; and Wu, C. 2024. Mini-Omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725

    Pith/arXiv arXiv 2024

  61. [61]

    Yang, D.; Tian, J.; Tan, X.; Huang, R.; Liu, S.; Chang, X.; Shi, J.; Zhao, S.; Bian, J.; Wu, X.; et al. 2023. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704

    Pith/arXiv arXiv 2023

  62. [62]

    Yao, Z.; Guo, L.; Yang, X.; Kang, W.; Kuang, F.; Yang, Y.; Jin, Z.; Lin, L.; and Povey, D. 2023. Zipformer: A faster and better encoder for automatic speech recognition. arXiv preprint arXiv:2310.11230

    Pith/arXiv arXiv 2023

  63. [63]

    Ye, Z.; Ju, Z.; Liu, H.; Tan, X.; Chen, J.; Lu, Y.; Sun, P.; Pan, J.; Bian, W.; He, S.; et al. 2024. FlashSpeech: Efficient zero-shot speech synthesis. In Proc. ACM MM, 6998--7007

    2024

  64. [64]

    Ye, Z.; Zhu, X.; Chan, C.-M.; Wang, X.; Tan, X.; Lei, J.; Peng, Y.; Liu, H.; Jin, Y.; DAI, Z.; et al. 2025. Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis. arXiv preprint arXiv:2502.04128

    Pith/arXiv arXiv 2025

  65. [65]

    Yu, L.; Simig, D.; Flaherty, C.; Aghajanyan, A.; Zettlemoyer, L.; and Lewis, M. 2023. Megabyte: Predicting million-byte sequences with multiscale transformers. Proc. NeurIPS, 36: 78808--78823

    2023

  66. [66]

    Zeghidour, N.; Luebs, A.; Omran, A.; Skoglund, J.; and Tagliasacchi, M. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 495--507

    2021

  67. [67]

    Zhang, D.; Li, S.; Zhang, X.; Zhan, J.; Wang, P.; Zhou, Y.; and Qiu, X. 2023 a . SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000

    Pith/arXiv arXiv 2023

  68. [68]

    Zhang, Y.; Han, W.; Qin, J.; Wang, Y.; Bapna, A.; Chen, Z.; Chen, N.; Li, B.; Axelrod, V.; Wang, G.; et al. 2023 b . Google USM: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037

    Pith/arXiv arXiv 2023