REVIEW 14 cited by
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
We introduce a text-to-speech (TTS) model called BASE TTS, which stands for $\textbf{B}$ig $\textbf{A}$daptive $\textbf{S}$treamable TTS with $\textbf{E}$mergent abilities. BASE TTS is the largest TTS model to-date, trained on 100K hours of public domain speech data, achieving a new state-of-the-art in speech naturalness. It deploys a 1-billion-parameter autoregressive Transformer that converts raw texts into discrete codes ("speechcodes") followed by a convolution-based decoder which converts these speechcodes into waveforms in an incremental, streamable manner. Further, our speechcodes are built using a novel speech tokenization technique that features speaker ID disentanglement and compression with byte-pair encoding. Echoing the widely-reported "emergent abilities" of large language models when trained on increasing volume of data, we show that BASE TTS variants built with 10K+ hours and 500M+ parameters begin to demonstrate natural prosody on textually complex sentences. We design and share a specialized dataset to measure these emergent abilities for text-to-speech. We showcase state-of-the-art naturalness of BASE TTS by evaluating against baselines that include publicly available large-scale text-to-speech systems: YourTTS, Bark and TortoiseTTS. Audio samples generated by the model can be heard at https://amazon-ltts-paper.com/.
Forward citations
Cited by 14 Pith papers
-
The NaijaVoices Dataset: Cultivating Large-Scale, High-Quality, Culturally-Rich Speech Data for African Languages
NaijaVoices is a 1,838-hour, 5,455-speaker speech-text corpus for Igbo, Hausa, and Yoruba whose use in fine-tuning cuts Word Error Rates by 42-76% relative to unadapted baselines.
-
MPO: Multidimensional Preference Optimization for Language Model-based Text-to-Speech
MPO improves TTS alignment by constructing multi-dimensional preference pairs and adding cross-entropy regularization to DPO, yielding better intelligibility, speaker similarity, and prosody.
-
Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy
DCAR dynamically schedules chunk-wise token prediction in AR TTS, improving WER by up to 72.27% relative and speeding up inference by up to 2.89x over next-token baselines.
-
Dataset of News Articles with Provenance Metadata for Media Relevance Assessment
A new benchmark dataset and two tasks let researchers test whether AI systems can judge if a news image's recorded location and date match the article, with current chatbots scoring 64-81% on location but 42-58% on date.
-
EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge
A new TTS benchmark with 1,645 LLM-generated challenge prompts and an audio-language-model judge that ranks TTS systems in close agreement with a small human study.
-
VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation
VoiceStar uses a progress-based rotary position embedding and mixed prompt training to give zero-shot voice cloning precise duration control and much longer output than training clips.
-
Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equation
A linear residual that subtracts a speaker-embedding-based projection from WavLM representations reduces speaker information while preserving content, improving voice conversion.
-
Quantize More, Lose Less: Autoregressive Generation from Residually Quantized Speech Representations
QTTS models speech as sequences from a multi-codebook RVQ audio codec whose first codebook is trained with ASR supervision, aiming for higher-fidelity TTS than single-codebook systems.
-
StreamFlow: Streaming Flow Matching with Block-wise Guided Attention Mask for Speech Token Decoding
StreamFlow achieves streaming speech token decoding with 180 ms first-packet latency by using hierarchical block-wise attention masks in a DiT flow matching model.
-
Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling
A hand-designed, training-free step schedule prunes later sampling steps in flow-matching TTS, cutting F5-TTS inference cost by about 4x while keeping quality roughly unchanged.
-
CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloning
A universal adversarial perturbation framework claiming to protect speech against zero-shot voice cloning by degrading cloned outputs while preserving input naturalness.
-
Discrete Audio Representations for Automated Audio Captioning
A supervised audio tokenizer trained with audio tagging labels nearly matches continuous BEATs features for automated audio captioning on Clotho, while conventional discrete tokens degrade performance.
-
Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment
A-DMA adds text and speech alignment losses to F5-TTS training, cutting the number of steps to a target quality while also improving WER and speaker similarity.
-
Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding
A speculative decoding variant with a tolerance-based acceptance rule speeds up CosyVoice 2 inference by 1.4x while keeping subjective quality comparable.
Discussion (0). Sign in to comment.