Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TouchTTS claims that LLM-based TTS can be trained on million-hour-scale dirty audio using a simplified pipeline and deployed entirely through standard LLM inference engines.

desk verdict Useful engineering report: the char-unit and flow-mask results are real, but the >50% retention claim is not pinned to S3Tokenizer by the evidence. read the letter →

arxiv 2412.08237 v2 pith:7HICUO3S submitted 2024-12-11 cs.SD cs.CLeess.AS

Xingchen Song , Mengtao Xing , Changwei Ma , Shengqiang Li , Di Wu , Binbin Zhang , Fuping Pan , Dinghao Zhou
show 4 more authors
Yuekai Zhang Shun Lei Zhendong Peng Zhiyong Wu
This is my paper · ORCID
classification cs.SDcs.CLeess.AS
keywords text-to-speechLLM-basedTTSS3TokenizerdataprocessingpipelineflowmatchingstreamingunifiedTTS/ASRretention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to remove the two bottlenecks of LLM-based text-to-speech: the expensive data pipelines that keep only 10–30% of raw audio, and the separate U-Net flow model that blocks reuse of standard LLM serving engines. It claims that a tokenizer trained with an ASR loss (S3Tokenizer) is already tolerant of noise and speaker variation, so denoising, speaker diarization, and punctuation modules can be deleted; a Copilot-ASR cross-validation using two recognizers then filters low-quality clips and keeps 51.6% of 1.26M hours of raw data. On the architecture side, the paper replaces the U-Net flow backbone with a standard LLM (Qwen2), removes the text and token encoders, and uses WeNet-style dynamic chunk masks so one flow model serves both streaming and non-streaming inference on TensorRT and vLLM. It also reports that a single LLM can be trained on the same dirty data for both TTS and ASR, with continuous tokenizer features outperforming discrete ones for ASR. If these claims hold, high-quality TTS training becomes a data-scaling and deployment problem within the standard LLM ecosystem rather than a custom speech-modelling one.

What carries the argument

The load-bearing mechanism has five pieces. First, S3Tokenizer, a speech tokenizer trained with an ASR loss, which the paper argues implicitly ignores background noise and speaker identity. Second, Copilot-ASR cross-validation with Rover, which compares Whisper and Paraformer transcriptions and drops clips whose WER is greater than 10 or PER is greater than 5. Third, Qwen2ForCausalLM used as both the text-to-token LLM and the token-to-waveform flow backbone, replacing the U-Net and making the whole stack executable by standard LLM inference engines. Fourth, WeNet-style dynamic chunk attention masks, which let one flow model see either the full sentence or fixed chunks and thereby unify streaming and non-streaming inference. Fifth, character-level Chinese units alongside English BPE, which give a one-character-one-pronunciation prior and reduce insertion and deletion errors in Chinese synthesis.

What would settle it

Run the simplified pipeline end-to-end twice, once with S3Tokenizer and once with a tokenizer trained without the ASR loss, keeping every other stage identical; if the retention rate and PER stay essentially the same, the ASR-loss tolerance attributed to the tokenizer is not the mechanism that made the 51.6% retention possible.

Watch

Extended reading notes

Core claim

The paper's central claim is that the S3Tokenizer's ASR-loss training makes it implicitly denoise and disentangle speakers, so a TTS data pipeline can be cut down to VAD, ASR, and a Copilot-ASR cross-check; this is what allows 1260k hours of raw audio to become 650k hours (51.6% retention) and a final one-million-hour mixed training set. A second claim is architectural: replacing the U-Net flow backbone with Qwen2ForCausalLM, removing the text/token encoders, and switching Chinese text to character units keeps synthesis quality on Seed-Eval comparable to or lower than CosyVoice on PER while enabling unified streaming and non-streaming inference with the same flow model. A third claim is that the same LLM can be trained jointly for TTS and ASR on the same dirty data: TTS performance stays on par with a standalone TTS model, and ASR works better with continuous S3Tokenizer features than with discrete tokens.

Load-bearing premise

The data-scaling claim depends on the ASR-trained S3Tokenizer actually ignoring background noise and speaker variation well enough that denoising, diarization, and punctuation modules can be dropped; the paper asserts this from prior work and does not isolate the tokenizer's contribution in its own experiments.

Editorial extensions

If this is right

  • A 51.6% retention rate means training-data acquisition for LLM TTS can roughly double in usable volume from the same raw audio, directly attacking the data-scaling bottleneck.
  • Using Qwen2 as the flow backbone lets token-to-waveform synthesis run on standard LLM inference engines such as TensorRT and vLLM, which removes the separate U-Net deployment path and, per the paper's benchmark, keeps first-packet latency under 200 ms.
  • The dynamic chunk mask trained into the flow model supports both streaming and non-streaming inference with one set of weights, so a TTS service does not need two different models for the two modes.
  • Chinese character units plus English BPE reduce insertion and deletion errors in Chinese synthesis, which suggests G2P conversion and byte-level BPE are unnecessary for large-scale Chinese TTS.
  • Training TTS and ASR on the same dirty data with one LLM is feasible without hurting TTS, which implies ASR-grade corpora can be reused as TTS training data rather than discarded.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not run the obvious isolation experiment: swapping S3Tokenizer for a tokenizer trained without ASR loss in the same pipeline would show whether the tokenizer's noise tolerance, rather than the Copilot-ASR filter, explains the 51.6% retention and the PER scores.
  • Because the WER greater than 10 and PER greater than 5 thresholds are hand-set, the pipeline would likely need domain-specific or automatically tuned thresholds; audiobooks and outdoor live streams already show different retention rates in the paper.
  • The punctuation-free training result suggests prosody emerges from data scale without explicit pause labels; a clean ablation at smaller data sizes could test whether this holds before reaching million-hour scale.
  • If unified TTS and ASR training is adopted broadly, ASR datasets could double as TTS pretraining data, which would change how speech corpora are curated for foundation models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This technical report describes TouchTTS, a simplified LLM-based TTS framework. The data pipeline removes denoising, speaker diarization, punctuation, and DNSMOS filtering, relying instead on S3Tokenizer plus a copilot-ASR cross-validation, and reports a 51.6% retention rate on 1,260k hours of raw data. The architecture replaces the text/token encoders and the U-Net flow model with Qwen-based backbones, enabling deployment on standard LLM inference engines and a unified streaming/non-streaming configuration. The paper also explores a unified TTS/ASR model trained on shared data. Experiments on Seed-Eval report PER and SIM for frontend choices, model comparison, streaming flow configurations, inference latency, and a 60k-step unified TTS/ASR trend.

Significance. The paper targets two genuine bottlenecks in LLM-based TTS: low data retention in filtering pipelines and the deployment cost of non-LLM flow backbones. If the central claims held, the simplified pipeline and Qwen-only architecture would be practically valuable, and the >50% retention rate would be a concrete advance over the 10-30% figures cited for existing pipelines. The manuscript is transparent about some limitations, including admitted unfairness in the baseline comparisons, and it provides reproducible architecture details plus objective Seed-Eval results. However, the central data-scaling claim is currently supported mainly by argument rather than by measurement, and several performance claims rest on trends or single configurations rather than on matched comparisons, so the contribution is promising but not yet established.

major comments (4)
  1. [3 (Simplified Data Processing Pipeline; Figure 1)] The >50% retention claim is the paper's central motivation, but the manuscript does not show that S3Tokenizer is the cause. Section 3 states 'we argue' and 'we hypothesize' that the ASR loss makes the tokenizer robust to dirty data; the only quantitative result, 51.6%, comes from the full VAD + ASR + Copilot-ASR/ROVER pipeline with hand-set thresholds (WER > 10, PER > 5), and the text itself notes that retention varies significantly across domains. There is no ablation comparing S3Tokenizer with a tokenizer without ASR loss, no comparison to a traditional pipeline on the same raw corpus, and no isolation of the copilot cross-validation contribution. Moreover, the TouchLLM evaluated in Table 2 is trained on the filtered 650k-hour subset plus additional ASR data, not on raw dirty audio, so downstream quality does not demonstrate dirty-data tolerance. Please add isolating ablations or substantially weaken the attribution in the abstract.
  2. [6.2 (Model comparison with baselines; Table 2)] The architecture claims are evaluated under conditions the authors themselves call unfair: model sizes, training data, and training time differ, while TouchLLM is trained for only one epoch. Table 2 shows FireRedTTS achieving a lower PER on test-zh (0.51 vs. 1.12 for TouchLLM-0.5B-NoPrompt), so the table does not establish parity without matched training. In addition, no human listening test is reported, and the PER metric is computed with Paraformer, the same ASR model family used in the data-filtering copilot. To support the claim that the simplified architecture performs comparably, please provide matched-condition comparisons and/or human evaluation of naturalness and intelligibility.
  3. [4.2, 4.3, and Table 3] The claim that replacing the U-Net flow model with a Qwen backbone is 'effective and efficient' is not supported by a direct comparison with a U-Net flow baseline. Table 3 compares only TouchFlow-170M and TouchFlow-50M under non-streaming and streaming configurations; every row uses the same Qwen backbone, so the table cannot show whether the simplification preserves the quality of CosyVoice's U-Net flow. Please add a matched U-Net baseline (for example, CosyVoice flow trained on the same Emilia subset with the same hyperparameters) or limit the claim to 'our Qwen flow supports streaming without degrading its own non-streaming performance.'
  4. [6.5 (Evaluation on unified TTS & ASR; Figure 5)] The unified TTS/ASR claim is supported only by a 60k-step training trend over roughly 0.4M hours, not by final converged results, and Figure 5 reports error-rate trends without absolute values or confidence intervals. The text asserts that these trends 'are sufficient,' but a trend at 60k steps does not establish that the unified model reaches the same TTS performance as the standalone model at convergence. Additionally, Section 5 says ASR and TTS are trained on 'the same large-scale dirty data,' while Section 3 describes 650k filtered hours plus open-source and internal ASR data; the exact composition of the shared training data needs to be specified. Please report longer-horizon or converged results with the precise data split.
minor comments (5)
  1. [Abstract and Section 3] The phrase 'for the first time' needs a citation or an explicit definition of the retention-rate metric; as written, it is not verifiable whether previous pipelines used the same segmentation and filtering conventions.
  2. [Throughout] There are minor typographical and naming inconsistencies, including 'V AD' in Section 3, 'FishSpech' in Section 4.1, and the mixed use of 'Rover' and 'ROVER.'
  3. [6.4 (Inference benchmark; Tables 4-6)] Table 5 reports only FP32 results for TouchFlow-170M while Table 4 gives both FP16 and FP32 for the LLM; a direct FP16 flow benchmark, together with a U-Net flow baseline on the same hardware, would make the deployment-latency claim more complete.
  4. [4.3 and Table 3] Several hyperparameters are stated without justification or sensitivity analysis, including the 0.5-second minimum chunk length, the 50% sentence-level receptive-field mix, the 5/10-token overlaps, and the 5 flow iterations.
  5. [6.1-6.3] PER and SIM results in Tables 1-3 are reported without confidence intervals; the claim that streaming has 'minimal' impact on quality would be stronger with variance over the five seeds.

Circularity Check

0 steps flagged · score 2.0 of 10

Minor self-citations (WeNet, wespeaker) are not load-bearing; the S3Tokenizer robustness premise is an acknowledged hypothesis rather than a fitted prediction, so there is no significant circularity.

full rationale

The central TTS architecture claims (Qwen-based LLM and flow backbones, unified streaming/non-streaming via dynamic chunk masks, and unified TTS/ASR) are tested on the external Seed-Eval benchmark with PER and SIM comparisons against FireRedTTS and CosyVoice; no parameter is fitted to the evaluation set, so these claims do not reduce to their inputs by construction. The 51.6% retention rate (Section 3) is the empirical output of the stated VAD + ASR + Copilot-ASR pipeline with hand-set WER>10 and PER>5 thresholds; it is not a predicted quantity derived from an equation that already contains the result, and the thresholds are not fitted to the reported retention number. The main evidence gap is that the paper justifies the dirty-data robustness of S3Tokenizer by 'we argue' and 'we hypothesize' (Section 3) without ablating the tokenizer or comparing against a standard pipeline; this is a correctness/validity concern, not a circular reduction. Using Paraformer both in the Copilot-ASR filter and for PER evaluation could bias the metric, but the comparison remains on fixed external test sentences and is not statistically forced by construction. Self-citations (WeNet [20] for chunk-based inference, wespeaker [38] for speaker similarity) serve as technique/tool pointers and are not load-bearing premises, so the paper receives a low circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The simplified pipeline's core claim rests on the untested robustness of S3Tokenizer to dirty audio and on hand-set ASR consistency thresholds. These choices directly determine the reported 51.6% retention rate. No new physical or model entities are proposed.

free parameters (5)
  • Copilot-ASR filter thresholds = WER>10, PER>5
    Hand-set thresholds used to reject training data; no sensitivity analysis is provided. They directly control the reported 51.6% retention rate.
  • Dynamic chunk min length = 0.5 seconds
    Minimum chunk length during training; affects the streaming versus non-streaming trade-off.
  • Flow iterations = 5
    Number of flow matching iterations at inference; affects latency and quality.
  • Receptive field coverage during training = 50% of training data at sentence-level
    Fraction of training data using full receptive field; chosen for stability, not swept.
  • ZeroShot overlap tokens = 5 or 10 tokens between chunks
    Inference hyperparameters for smoothing chunk boundaries in streaming mode.
assumptions (4)
  • domain assumption S3Tokenizer's ASR-loss training makes it robust to noise and speaker variation, allowing TTS training on dirty data.
    Stated in Section 3 and attributed to prior work [1]; not tested here against other tokenizers.
  • domain assumption Cross-validation between two ASR models approximates SNR and DNSMOS quality filtering.
    Explicitly hypothesized in Section 3; no experiment demonstrates the equivalence.
  • domain assumption Dynamic chunk masks with varying receptive fields during training transfer to streaming and non-streaming inference.
    Inspired by WeNet [20]; supported only by the Section 6.3 experiments, with no formal argument.
  • domain assumption PER computed from ASR reflects intelligibility better than WER for this evaluation.
    Argued in Section 6; no comparison of WER versus PER for the authors' own models is presented.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch." pith.science (2026). https://pith.science/paper/7HICUO3S

@misc{pith2026241208237,
  author       = {Pith},
  title        = {Pith review of: TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7HICUO3S}},
  note         = {Machine review of arXiv:2412.08237}
}
read the original abstract

It is well known that LLM-based systems are data-hungry. Recent LLM-based TTS works typically employ complex data processing pipelines to obtain high-quality training data. These sophisticated pipelines require excellent models at each stage (e.g., speech denoising, speech enhancement, speaker diarization, and punctuation models), which themselves demand high-quality training data and are rarely open-sourced. Even with state-of-the-art models, issues persist, such as incomplete background noise removal and misalignment between punctuation and actual speech pauses. Moreover, the stringent filtering strategies often retain only 10-30\% of the original data, significantly impeding data scaling efforts. In this work, we leverage a noise-robust audio tokenizer (S3Tokenizer) to design a simplified yet effective TTS data processing pipeline that maintains data quality while substantially reducing data acquisition costs, achieving a data retention rate of over 50\%. Beyond data scaling challenges, LLM-based TTS systems also incur higher deployment costs compared to conventional approaches. Current systems typically use LLMs solely for text-to-token generation, while requiring separate models (e.g., flow matching models) for token-to-waveform generation, which cannot be directly executed by LLM inference engines, further complicating deployment. To address these challenges, we eliminate redundant modules in both LLM and flow components, replacing the flow model backbone with an LLM architecture. Building upon this simplified flow backbone, we propose a unified architecture for both streaming and non-streaming inference, significantly reducing deployment costs. Finally, we explore the feasibility of unifying TTS and ASR tasks using the same data for training, thanks to the simplified pipeline and the S3Tokenizer that reduces the quality requirements for TTS training data.

Figures

Figures reproduced from arXiv: 2412.08237 by the authors.

Figure 1
Figure 1. Overview of the Simplified Data Processing Pipeline. The blue blocks represent the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Simplified TTS Architecture. The blue blocks represent the common [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the training and inference pipeline for unified streaming and non-streaming [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Overview of Unified TTS and ASR. In the unified architecture, ASR and TTS share the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison between single TTS model and unified TTS/ASR model. For TTS task, we [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech

    eess.AS 2026-08 conditional novelty 7.0 of 10

    Experience-Calibrated Contrastive Decoding, a training-free decoding method that strengthens text alignment signals, reduces speech hallucination errors across four LM-based TTS models and nine languages.

  2. DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis

    eess.AS 2025-07 conditional novelty 6.0 of 10

    Reinforcement learning on duration prediction improves intelligibility and speaker similarity in a 4-step distilled text-to-speech model, and teacher-guided sampling recovers prosodic diversity.

  3. Balalaika: Data-Centric, Prosody-Aware Annotation Pipeline for Russian Speech

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    Balalaika is a data-centric annotation pipeline for Russian speech that combines semantic VAD, ASR ensembling, and prosody enrichment to build a 5.1k-hour corpus showing gains in denoising and TTS.

  4. Interleaved Speech-Text Language Models for Simple Streaming Text-to-Speech Synthesis

    eess.AS 2024-12 conditional novelty 6.0 of 10

    Training a decoder-only LM on fixed-ratio interleaved text and speech tokens yields a simple zero-shot streaming TTS, with a 1:3 text-to-speech chunk ratio keeping WER within about 8 percent relative of non-streaming.

  5. TouchASP: Elastic Automatic Speech Perception that Everyone Can Touch

    eess.AS 2024-12 conditional novelty 6.0 of 10

    TouchASP trains a single elastic mixture-of-experts ASR model on 1M hours of partly pseudo-labeled audio and reports SpeechIO CER dropping from 4.98% to 2.45% while adding multi-task perception.

Reference graph

Works this paper leans on

39 extracted references · 12 canonical work pages · cited by 5 Pith papers

  1. [1]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024

    arXiv 2024

  2. [2]

    Matcha-tts: A fast tts architecture with conditional flow matching

    Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, and Gustav Eje Henter. Matcha-tts: A fast tts architecture with conditional flow matching. In Proc. ICASSP, pages 11341–11345. IEEE, 2024

    work page 2024

  3. [3]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    arXiv 2024

  4. [4]

    Tensorrt-llm, https://github.com/nvidia/tensorrt-llm, 2024

    NVIDIA. Tensorrt-llm, https://github.com/nvidia/tensorrt-llm, 2024

    work page 2024

  5. [5]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  6. [6]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022. 5https://developer.d-robotics.cc 13

    arXiv 2022

  7. [7]

    Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthesis

    Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yijin Xing. Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthesis. arXiv preprint arXiv:2411.01156, 2024

    arXiv 2024

  8. [8]

    Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications

    Hao-Han Guo, Kun Liu, Fei-Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.03283, 2024

    arXiv 2024

Show all 39 references
  1. [9]

    Takin: A cohort of superior quality zero-shot speech generation models

    Sijing Chen, Yuan Feng, Laipeng He, Tianwei He, Wendi He, Yanni Hu, Bin Lin, Yiting Lin, Yu Pan, Pengfei Tan, et al. Takin: A cohort of superior quality zero-shot speech generation models. arXiv preprint arXiv:2409.12139, 2024

    2024 arXiv

  2. [10]

    Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model benchmark, 2024

    Linhan Ma, Dake Guo, Kun Song, Yuepeng Jiang, Shuai Wang, Liumeng Xue, Weiming Xu, Huan Zhao, Binbin Zhang, and Lei Xie. Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model benchmark, 2024

    2024

  3. [11]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361, 2024

    2024 arXiv

  4. [12]

    Autoprep: An automatic preprocessing framework for in-the-wild speech data

    Jianwei Yu, Hangting Chen, Yanyao Bian, Xiang Li, Yi Luo, Jinchuan Tian, Mengyang Liu, Jiayi Jiang, and Shuai Wang. Autoprep: An automatic preprocessing framework for in-the-wild speech data. In Proc. ICASSP, pages 1136–1140. IEEE, 2024

    2024

  5. [13]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  6. [14]

    Dnsmos p

    Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In Proc. ICASSP, pages 886–890. IEEE, 2022

    2022

  7. [15]

    Audiopalm: A large language model that can speak and listen

    Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925, 2023

    2023 arXiv

  8. [16]

    Conformer-1: Robust asr via large-scale semisupervised bootstrapping

    Kevin Zhang, Luka Chkhetiani, Francis McCann Ramirez, Yash Khare, Andrea Vanzo, Michael Liang, Sergio Ramirez Martin, Gabriel Oexle, Ruben Bousbib, Taufiquzzaman Peyash, et al. Conformer-1: Robust asr via large-scale semisupervised bootstrapping. arXiv preprint arXiv:2404.07341, 2024

    2024 arXiv

  9. [17]

    Anatomy of industrial scale multilingual asr

    Francis McCann Ramirez, Luka Chkhetiani, Andrew Ehrenberg, Robert McHardy, Rami Botros, Yash Khare, Andrea Vanzo, Taufiquzzaman Peyash, Gabriel Oexle, Michael Liang, et al. Anatomy of industrial scale multilingual asr. arXiv preprint arXiv:2404.09841, 2024

    2024 arXiv

  10. [18]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. ICML, pages 28492–28518. PMLR, 2023

    2023

  11. [19]

    Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition

    Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. arXiv preprint arXiv:2206.08317, 2022

    2022 arXiv

  12. [20]

    Wenet 2.0: More productive end-to-end speech recognition toolkit

    Binbin Zhang, Di Wu, Zhendong Peng, Xingchen Song, Zhuoyuan Yao, Hang Lv, Lei Xie, Chao Yang, Fuping Pan, and Jianwei Niu. Wenet 2.0: More productive end-to-end speech recognition toolkit. arXiv preprint arXiv:2203.15455, 2022

    2022 arXiv

  13. [21]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  14. [22]

    Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech

    Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech. In International Conference on Machine Learning , pages 5530–5540. PMLR, 2021. 14

    2021

  15. [23]

    FastSpeech: Fast, robust and controllable text to speech

    Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech: Fast, robust and controllable text to speech. In Proc. NeurIPS, pages 3165–3174, 2019

    2019

  16. [24]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  17. [25]

    Flow- TTS: A non-autoregressive network for text to speech based on flow

    Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao. Flow- TTS: A non-autoregressive network for text to speech based on flow. In Proc. ICASSP, pages 7209–7213. IEEE, 2020

    2020

  18. [26]

    An embarrassingly simple approach for llm with strong asr capacity

    Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. An embarrassingly simple approach for llm with strong asr capacity. arXiv preprint arXiv:2402.08846, 2024

    2024 arXiv

  19. [27]

    Lauragpt: Listen, attend, understand, and regenerate audio with gpt

    Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673, 2023

    2023 arXiv

  20. [28]

    Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition

    Ye Bai, Jingping Chen, Jitong Chen, Wei Chen, Zhuo Chen, Chuang Ding, Linhao Dong, Qianqian Dong, Yujiao Du, Kepan Gao, et al. Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition. arXiv preprint arXiv:2407.04675, 2024

    2024 arXiv

  21. [29]

    dmel: Speech tokenization made simple

    He Bai, Tatiana Likhomanenko, Ruixiang Zhang, Zijin Gu, Zakaria Aldeneh, and Navdeep Jaitly. dmel: Speech tokenization made simple. arXiv preprint arXiv:2407.15835, 2024

    2024 arXiv

  22. [30]

    Librispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 5206–5210. IEEE, 2015

    2015

  23. [31]

    Scaling speech-text pre-training with synthetic interleaved data

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data. arXiv preprint arXiv:2411.17607, 2024

    2024 arXiv

  24. [32]

    Spirit-lm: Interleaved spoken and written language model

    Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, et al. Spirit-lm: Interleaved spoken and written language model. arXiv preprint arXiv:2402.05755, 2024

    2024 arXiv

  25. [33]

    Seed-tts: A family of high-quality versatile speech generation models

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024

    2024 arXiv

  26. [34]

    Didispeech: A large scale mandarin speech corpus

    Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. Didispeech: A large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)...

    2021

  27. [35]

    Common voice: A massively-multilingual speech corpus

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019

    1912 arXiv

  28. [36]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024

    2024 arXiv

  29. [37]

    Maskgct: Zero-shot text-to-speech with masked generative codec transformer

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskgct: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750, 2024. 15

    2024 arXiv

  30. [38]

    Wespeaker: A research and production oriented speaker embedding learning toolkit

    Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. Wespeaker: A research and production oriented speaker embedding learning toolkit. In Proc. ICASSP, pages 1–5. IEEE, 2023

    2023

  31. [39]

    Speechcolab leaderboard, https://github.com/speechcolab/leaderboard, 2021

    SpeechColab. Speechcolab leaderboard, https://github.com/speechcolab/leaderboard, 2021. 16

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.