{"id":"0e8d94a3-65bf-4e95-9f97-2e7df8c4331a","arxiv_id":"2412.08237","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TouchTTS reports a 51.6% data retention rate using a noise-robust tokenizer and two-ASR cross-validation, and a Qwen-backbone flow model that unifies streaming and non-streaming synthesis while matching CosyVoice on PER.","lead":"This paper presents a simplified pipeline for LLM-based text-to-speech that keeps over 50% of raw audio as training data, versus the 10-30% typical of prior systems, and a Qwen-based architecture that runs both streaming and non-streaming synthesis on the same flow model. The significance is practical: lower data and deployment costs for building scalable TTS, plus a first attempt to train TTS and ASR on the same data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The >50% retention claim is not shown to be caused by S3Tokenizer robustness; no ablation isolates the tokenizer from the ASR cross-validation or compares to a standard pipeline.","rationale":"The reader's weakest assumption already identifies the S3Tokenizer robustness premise as the soft spot. I agree and sharpen it: Section 3 explicitly says 'we argue' that ASR loss gives S3Tokenizer denoising and speaker-disentangling abilities; the 51.6% retention number is an end-to-end pipeline result, not an isolation of the tokenizer. The paper even acknowledges domain-dependent retention, so raw-data composition is a confound. The strongest counterevidence would be a controlled ablation; without it, the central data-scaling contribution remains plausible but unproven. I also note the architecture claim (LLM-backbone flow) lacks a U-Net flow baseline, but that is secondary because the headline contribution is the simplified data pipeline. Since the reader already issued CONDITIONAL with medium confidence, my analysis does not change the verdict; it reinforces the condition that the ablation be performed. No unsupported claim is treated as fraud; this is a normal missing-baseline issue in a technical report.","tokens_in":12835,"tokens_out":5444,"duration_ms":56879,"concrete_test":"On a fixed raw corpus (e.g., 100k hours of noisy in-the-wild speech), run four variants: (i) current pipeline with S3Tokenizer; (ii) current pipeline with a non-ASR tokenizer (e.g., EnCodec or k-means) while keeping the same LLM/flow recipe; (iii) S3Tokenizer with only VAD and basic SNR filtering, omitting Copilot-ASR/Rover; (iv) traditional pipeline (denoising, diarization, DNSMOS, punctuation) with S3Tokenizer. Report per-variant retention rate and downstream TTS PER/SIM on Seed-Eval. If (i) and (ii) are comparable or (iii) degrades sharply, the tokenizer robustness assumption is unsupported and the retention claim must be reattributed to the ASR filtering or the raw corpus.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The data-scaling headline rests on Section 3's claim that S3Tokenizer's ASR loss makes it robust to dirty audio, so denoising, diarization, and punctuation can be removed. The paper supports this only by 'we argue' and 'we hypothesize' (Section 3), not by measurement. The sole quantitative result, 51.6% retention, is produced by the full VAD + ASR + Copilot-ASR/Rover pipeline with hand-set WER>10 and PER>5 thresholds. There is no ablation removing the copilot cross-validation, no comparison to a traditional pipeline on the same raw corpus, and no comparison to a tokenizer without ASR loss (e.g., EnCodec or k-means quantization). Because the paper itself states retention varies strongly across domains, the average 51.6% may reflect raw-corpus composition or threshold calibration rather than tokenizer robustness. Moreover, the TouchLLM evaluated in Table 2 is trained on the filtered 650k-hour output (plus additional ASR data), not on raw dirty audio, so downstream quality does not demonstrate dirty-data tolerance. If the S3Tokenizer robustness premise is false, the simplified pipeline's advantage reduces to ASR cross-validation thresholds, and the 'first time' >50% retention claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This technical report describes TouchTTS, a simplified LLM-based TTS framework. The data pipeline removes denoising, speaker diarization, punctuation, and DNSMOS filtering, relying instead on S3Tokenizer plus a copilot-ASR cross-validation, and reports a 51.6% retention rate on 1,260k hours of raw data. The architecture replaces the text/token encoders and the U-Net flow model with Qwen-based backbones, enabling deployment on standard LLM inference engines and a unified streaming/non-streaming configuration. The paper also explores a unified TTS/ASR model trained on shared data. Experiments on Seed-Eval report PER and SIM for frontend choices, model comparison, streaming flow configurations, inference latency, and a 60k-step unified TTS/ASR trend.","tokens_in":13107,"tokens_out":5852,"duration_ms":60792,"significance":"The paper targets two genuine bottlenecks in LLM-based TTS: low data retention in filtering pipelines and the deployment cost of non-LLM flow backbones. If the central claims held, the simplified pipeline and Qwen-only architecture would be practically valuable, and the >50% retention rate would be a concrete advance over the 10-30% figures cited for existing pipelines. The manuscript is transparent about some limitations, including admitted unfairness in the baseline comparisons, and it provides reproducible architecture details plus objective Seed-Eval results. However, the central data-scaling claim is currently supported mainly by argument rather than by measurement, and several performance claims rest on trends or single configurations rather than on matched comparisons, so the contribution is promising but not yet established.","major_comments":[{"comment":"The >50% retention claim is the paper's central motivation, but the manuscript does not show that S3Tokenizer is the cause. Section 3 states 'we argue' and 'we hypothesize' that the ASR loss makes the tokenizer robust to dirty data; the only quantitative result, 51.6%, comes from the full VAD + ASR + Copilot-ASR/ROVER pipeline with hand-set thresholds (WER > 10, PER > 5), and the text itself notes that retention varies significantly across domains. There is no ablation comparing S3Tokenizer with a tokenizer without ASR loss, no comparison to a traditional pipeline on the same raw corpus, and no isolation of the copilot cross-validation contribution. Moreover, the TouchLLM evaluated in Table 2 is trained on the filtered 650k-hour subset plus additional ASR data, not on raw dirty audio, so downstream quality does not demonstrate dirty-data tolerance. Please add isolating ablations or substantially weaken the attribution in the abstract.","section":"3 (Simplified Data Processing Pipeline; Figure 1)"},{"comment":"The architecture claims are evaluated under conditions the authors themselves call unfair: model sizes, training data, and training time differ, while TouchLLM is trained for only one epoch. Table 2 shows FireRedTTS achieving a lower PER on test-zh (0.51 vs. 1.12 for TouchLLM-0.5B-NoPrompt), so the table does not establish parity without matched training. In addition, no human listening test is reported, and the PER metric is computed with Paraformer, the same ASR model family used in the data-filtering copilot. To support the claim that the simplified architecture performs comparably, please provide matched-condition comparisons and/or human evaluation of naturalness and intelligibility.","section":"6.2 (Model comparison with baselines; Table 2)"},{"comment":"The claim that replacing the U-Net flow model with a Qwen backbone is 'effective and efficient' is not supported by a direct comparison with a U-Net flow baseline. Table 3 compares only TouchFlow-170M and TouchFlow-50M under non-streaming and streaming configurations; every row uses the same Qwen backbone, so the table cannot show whether the simplification preserves the quality of CosyVoice's U-Net flow. Please add a matched U-Net baseline (for example, CosyVoice flow trained on the same Emilia subset with the same hyperparameters) or limit the claim to 'our Qwen flow supports streaming without degrading its own non-streaming performance.'","section":"4.2, 4.3, and Table 3"},{"comment":"The unified TTS/ASR claim is supported only by a 60k-step training trend over roughly 0.4M hours, not by final converged results, and Figure 5 reports error-rate trends without absolute values or confidence intervals. The text asserts that these trends 'are sufficient,' but a trend at 60k steps does not establish that the unified model reaches the same TTS performance as the standalone model at convergence. Additionally, Section 5 says ASR and TTS are trained on 'the same large-scale dirty data,' while Section 3 describes 650k filtered hours plus open-source and internal ASR data; the exact composition of the shared training data needs to be specified. Please report longer-horizon or converged results with the precise data split.","section":"6.5 (Evaluation on unified TTS & ASR; Figure 5)"}],"minor_comments":[{"comment":"The phrase 'for the first time' needs a citation or an explicit definition of the retention-rate metric; as written, it is not verifiable whether previous pipelines used the same segmentation and filtering conventions.","section":"Abstract and Section 3"},{"comment":"There are minor typographical and naming inconsistencies, including 'V AD' in Section 3, 'FishSpech' in Section 4.1, and the mixed use of 'Rover' and 'ROVER.'","section":"Throughout"},{"comment":"Table 5 reports only FP32 results for TouchFlow-170M while Table 4 gives both FP16 and FP32 for the LLM; a direct FP16 flow benchmark, together with a U-Net flow baseline on the same hardware, would make the deployment-latency claim more complete.","section":"6.4 (Inference benchmark; Tables 4-6)"},{"comment":"Several hyperparameters are stated without justification or sensitivity analysis, including the 0.5-second minimum chunk length, the 50% sentence-level receptive-field mix, the 5/10-token overlaps, and the 5 flow iterations.","section":"4.3 and Table 3"},{"comment":"PER and SIM results in Tables 1-3 are reported without confidence intervals; the claim that streaming has 'minimal' impact on quality would be stronger with variance over the five seeds.","section":"6.1-6.3"}],"recommendation":"major_revision","confidential_remarks":"The data-pipeline claims rely on S3Tokenizer, which the corresponding author maintains, and the PER evaluation uses Paraformer, the same ASR family used in the data-filtering copilot. This is not by itself disqualifying—the final evaluation is on the external Seed-Eval set—but it makes the absence of an independent tokenizer comparison and a non-Paraformer evaluation more consequential. The revision should prioritize the ablations and matched comparisons listed in the major comments; presentational fixes alone will not be sufficient. The paper is otherwise a readable systems report with useful architecture details, and the authors are commendably frank about the unfairness of their baseline comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful engineering report, not a scientific breakthrough, and the 50%-retention headline is weaker than it looks. The two things worth remembering are the character-based Chinese tokenization result and the Qwen-based flow backbone with dynamic chunk masking. Both are new, simple, and supported by the reported numbers. The frontend table (Table 1) shows a clean drop in insertion/deletion errors on test-zh and test-hard when switching from BPE to char for Chinese, holding the rest of the pipeline fixed. That is a real, reproducible-style result. The streaming flow experiments (Table 3) show the dynamic chunk mask works: streaming configurations land within noise of non-streaming PER/SIM. The inference benchmark also supports the deployment story.\n\nThe soft spot is the data-pipeline claim. Section 3 attributes the >50% retention to S3Tokenizer's robustness, but the only support is 'we argue' and 'we hypothesize'—no ablation isolates the tokenizer, no comparison to a standard pipeline on the same corpus, no comparison to a non-ASR-loss tokenizer. The 51.6% number comes from the full VAD+ASR+Copilot-ASR pipeline with hand-set WER/PER thresholds, and the paper itself says retention varies strongly across domains. So the headline 'over 50% for the first time' may be true for this corpus and these thresholds, but it is not shown to be caused by S3Tokenizer. Also, the TouchLLM evaluated in Table 2 is trained on the filtered 650k-hour output plus ASR data, not on raw dirty audio, so downstream quality doesn't demonstrate dirty-data tolerance either. The paper admits its baseline comparisons are unfair on model size and training data, and the unified TTS/ASR section only shows 60k-step trends, which is explicitly acknowledged as 'sufficient' but is not a full training run.\n\nA smaller concern: evaluation uses Paraformer for both filtering and PER computation. That is not fatal—the frontend comparison is controlled—but it could bias the absolute numbers in the model comparison. There are no human listening tests, and no code, data, or audio samples are released, so independent verification is limited.\n\nBottom line: the paper deserves a serious referee. The frontend and flow-mask results are worth publishing; the data-retention claim needs a proper ablation and a baseline pipeline comparison before it can be taken at face value. If I worked on TTS, I'd cite it for the char-unit result and the unified stream/non-stream flow design.","headline":"Useful engineering report: the char-unit and flow-mask results are real, but the >50% retention claim is not pinned to S3Tokenizer by the evidence.","tokens_in":13666,"tokens_out":3547,"would_cite":true,"duration_ms":34942,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TouchTTS claims that LLM-based TTS can be trained on million-hour-scale dirty audio using a simplified pipeline and deployed entirely through standard LLM inference engines.","keywords":["text-to-speech","LLM-based TTS","S3Tokenizer","data processing pipeline","flow matching","streaming TTS","unified TTS/ASR","data retention"],"falsifier":"Run the simplified pipeline end-to-end twice, once with S3Tokenizer and once with a tokenizer trained without the ASR loss, keeping every other stage identical; if the retention rate and PER stay essentially the same, the ASR-loss tolerance attributed to the tokenizer is not the mechanism that made the 51.6% retention possible.","tokens_in":12655,"feed_emoji":"🗣️","tokens_out":9687,"duration_ms":89777,"temperature":0.7,"pith_summary":"This paper is trying to remove the two bottlenecks of LLM-based text-to-speech: the expensive data pipelines that keep only 10–30% of raw audio, and the separate U-Net flow model that blocks reuse of standard LLM serving engines. It claims that a tokenizer trained with an ASR loss (S3Tokenizer) is already tolerant of noise and speaker variation, so denoising, speaker diarization, and punctuation modules can be deleted; a Copilot-ASR cross-validation using two recognizers then filters low-quality clips and keeps 51.6% of 1.26M hours of raw data. On the architecture side, the paper replaces the U-Net flow backbone with a standard LLM (Qwen2), removes the text and token encoders, and uses WeNet-style dynamic chunk masks so one flow model serves both streaming and non-streaming inference on TensorRT and vLLM. It also reports that a single LLM can be trained on the same dirty data for both TTS and ASR, with continuous tokenizer features outperforming discrete ones for ASR. If these claims hold, high-quality TTS training becomes a data-scaling and deployment problem within the standard LLM ecosystem rather than a custom speech-modelling one.","feed_headline":"TTS pipeline keeps 51.6% of raw audio for training","feed_subtitle":"A noise-tolerant tokenizer and two-ASR cross-check replace denoising, diarization, and punctuation modules.","key_machinery":"The load-bearing mechanism has five pieces. First, S3Tokenizer, a speech tokenizer trained with an ASR loss, which the paper argues implicitly ignores background noise and speaker identity. Second, Copilot-ASR cross-validation with Rover, which compares Whisper and Paraformer transcriptions and drops clips whose WER is greater than 10 or PER is greater than 5. Third, Qwen2ForCausalLM used as both the text-to-token LLM and the token-to-waveform flow backbone, replacing the U-Net and making the whole stack executable by standard LLM inference engines. Fourth, WeNet-style dynamic chunk attention masks, which let one flow model see either the full sentence or fixed chunks and thereby unify streaming and non-streaming inference. Fifth, character-level Chinese units alongside English BPE, which give a one-character-one-pronunciation prior and reduce insertion and deletion errors in Chinese synthesis.","core_discovery":"The paper's central claim is that the S3Tokenizer's ASR-loss training makes it implicitly denoise and disentangle speakers, so a TTS data pipeline can be cut down to VAD, ASR, and a Copilot-ASR cross-check; this is what allows 1260k hours of raw audio to become 650k hours (51.6% retention) and a final one-million-hour mixed training set. A second claim is architectural: replacing the U-Net flow backbone with Qwen2ForCausalLM, removing the text/token encoders, and switching Chinese text to character units keeps synthesis quality on Seed-Eval comparable to or lower than CosyVoice on PER while enabling unified streaming and non-streaming inference with the same flow model. A third claim is that the same LLM can be trained jointly for TTS and ASR on the same dirty data: TTS performance stays on par with a standalone TTS model, and ASR works better with continuous S3Tokenizer features than with discrete tokens.","pith_inferences":["The paper does not run the obvious isolation experiment: swapping S3Tokenizer for a tokenizer trained without ASR loss in the same pipeline would show whether the tokenizer's noise tolerance, rather than the Copilot-ASR filter, explains the 51.6% retention and the PER scores.","Because the WER greater than 10 and PER greater than 5 thresholds are hand-set, the pipeline would likely need domain-specific or automatically tuned thresholds; audiobooks and outdoor live streams already show different retention rates in the paper.","The punctuation-free training result suggests prosody emerges from data scale without explicit pause labels; a clean ablation at smaller data sizes could test whether this holds before reaching million-hour scale.","If unified TTS and ASR training is adopted broadly, ASR datasets could double as TTS pretraining data, which would change how speech corpora are curated for foundation models."],"forward_implications":["A 51.6% retention rate means training-data acquisition for LLM TTS can roughly double in usable volume from the same raw audio, directly attacking the data-scaling bottleneck.","Using Qwen2 as the flow backbone lets token-to-waveform synthesis run on standard LLM inference engines such as TensorRT and vLLM, which removes the separate U-Net deployment path and, per the paper's benchmark, keeps first-packet latency under 200 ms.","The dynamic chunk mask trained into the flow model supports both streaming and non-streaming inference with one set of weights, so a TTS service does not need two different models for the two modes.","Chinese character units plus English BPE reduce insertion and deletion errors in Chinese synthesis, which suggests G2P conversion and byte-level BPE are unnecessary for large-scale Chinese TTS.","Training TTS and ASR on the same dirty data with one LLM is feasible without hurting TTS, which implies ASR-grade corpora can be reused as TTS training data rather than discarded."],"supporting_citations":[{"why":"Supplies the S3Tokenizer used for noise-tolerant audio tokenization and the CosyVoice architecture and baseline that this work simplifies.","marker":"[1]"},{"why":"Provides the Qwen2ForCausalLM backbone reused for both the text-to-token LLM and the token-to-waveform flow model.","marker":"[3]"},{"why":"TensorRT-LLM is the inference engine used in the paper's latency benchmarks and a target for deployable Qwen-based flow inference.","marker":"[4]"},{"why":"vLLM is cited as an LLM inference engine that the simplified Qwen-based TTS architecture can directly run on.","marker":"[5]"},{"why":"AudioPaLM gives the prior result that ASR-loss tokenizer training beats k-means quantization in speech translation, which underpins the S3Tokenizer noise-resistance argument.","marker":"[15]"},{"why":"Whisper serves as the main ASR model in the simplified pipeline's Copilot-ASR cross-validation.","marker":"[18]"},{"why":"Paraformer serves as the copilot ASR model in the cross-validation and as the transcription model used for PER evaluation.","marker":"[19]"},{"why":"WeNet 2.0 provides the dynamic chunk training and attention-mask strategy that unifies streaming and non-streaming flow inference.","marker":"[20]"},{"why":"LauraGPT is the comparison point for unified TTS and ASR with continuous features, supporting the paper's claim that continuous features outperform discrete ones for ASR.","marker":"[27]"},{"why":"Seed-Eval is the benchmark providing the test-zh, test-en, and test-hard sets used for PER and speaker-similarity comparisons.","marker":"[33]"}],"fun_headline_variants":["TTS pipeline keeps 51.6% raw audio via noise-robust tokenizer","Unified LLM for TTS and ASR with simple pipeline","Replacing flow U-Net with LLM unifies streaming and batch","Over half of raw audio survives simplified TTS pipeline","TouchTTS: one LLM, no denoising, 51.6% data kept"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The data-scaling claim depends on the ASR-trained S3Tokenizer actually ignoring background noise and speaker variation well enough that denoising, diarization, and punctuation modules can be dropped; the paper asserts this from prior work and does not isolate the tokenizer's contribution in its own experiments.","fun_headline_variants_meta":{"raw":{"variants":["TTS pipeline keeps 51.6% raw audio via noise-robust tokenizer","Unified LLM for TTS and ASR with simple pipeline","Replacing flow U-Net with LLM unifies streaming and batch","Over half of raw audio survives simplified TTS pipeline","TouchTTS: one LLM, no denoising, 51.6% data kept"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000868,"raw_usage":{"total_tokens":3828,"prompt_tokens":1080,"completion_tokens":2748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":696,"completion_tokens_details":{"reasoning_tokens":2647}},"tokens_in":696,"tokens_out":2748,"duration_ms":19513,"temperature":1.0,"reasoning_tokens":2647,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:02:11.826095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the simplified pipeline end-to-end twice, once with S3Tokenizer and once with a tokenizer trained without the ASR loss, keeping every other stage identical; if the retention rate and PER stay essentially the same, the ASR-loss tolerance attributed to the tokenizer is not the mechanism that made the 51.6% retention possible.","supporting_citations":[{"cited_title":"Tensorrt-llm, https://github.com/nvidia/tensorrt-llm, 2024","cited_arxiv_id":null,"evidence_quote":"TensorRT-LLM is the inference engine used in the paper's latency benchmarks and a target for deployable Qwen-based flow inference."},{"cited_title":"Robust speech recognition via large-scale weak supervision","cited_arxiv_id":null,"evidence_quote":"Whisper serves as the main ASR model in the simplified pipeline's Copilot-ASR cross-validation."}],"review_version":1}