Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment

Chunhui Wang; Jeongsoo Choi; Ji-Hoon Kim; Joon Son Chung; Xie Chen; Zhikang Niu

REVIEW 4 major objections 5 minor 45 references

Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding two alignment losses to diffusion TTS training halves the number of steps to convergence while improving intelligibility and voice similarity.

desk verdict Solid incremental TTS training-speed paper, but the headline 'doubles convergence' is measured in steps, not wall-clock time, and the paper never reports the per-step overhead. read the letter →

arxiv 2505.19595 v2 pith:TAGCTIWO submitted 2025-05-26 eess.AS cs.SD

Jeongsoo Choi , Zhikang Niu , Ji-Hoon Kim , Chunhui Wang , Joon Son Chung , Xie Chen This is my paper

classification eess.AScs.SD

keywords diffusiontext-to-speechflowmatchingmodalityalignmentCTClossself-supervisedspeechrepresentationszero-shotTTStrainingaccelerationHuBERT

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that diffusion text-to-speech models spend too many training steps learning to connect text to speech on their own. It adds two auxiliary losses to the F5-TTS training objective: a CTC-based loss that forces an intermediate transformer layer to predict the text, and a cosine-similarity loss that nudges another layer toward the representations of a frozen speech model (HuBERT). With these guides, the model reaches the same or better word error rate and speaker similarity in about half the training steps, and the extra modules are discarded at inference time. If the claim holds, the result is a faster, cheaper recipe for zero-shot TTS without changing the deployed model.

What carries the argument

The mechanism is a dual-branch auxiliary objective stacked on the conditional flow matching loss $L_{\text{CFM}}$. The text branch is a CTC compressor: a projection head that maps an intermediate transformer layer to frame-level character posteriors and trains with the CTC loss to align text and speech lengths. The speech branch is an aligner: a linear interpolation plus 1D convolution that maps a hidden layer to the time-resolution of a frozen HuBERT representation, trained with negative cosine similarity. Both branches operate only during training, use frozen SSL parameters for speech, and their gradients are added to the flow-matching gradient with weights $\lambda_{\text{text}}=0.1$ and $\lambda_{\text{speech}}=1$. Layer placement matters: text guidance works best at layers 8–12, speech guidance at layers 12–16, and the best dual configuration places text below speech (8th and 12th), respecting the representational gap between modalities.

What would settle it

Measure wall-clock seconds per training step for F5-TTS with and without A-DMA on the same hardware, then compute total time to reach the baseline's converged WER. If the added per-step cost makes total training time longer or fails to reduce it by roughly half, the acceleration claim fails; the paper does not report these numbers.

Watch

Extended reading notes

Core claim

The central claim is that aligning the hidden states of a diffusion transformer with discriminative features from both modalities accelerates convergence while improving quality. Text-guided alignment applies CTC supervision at an early-to-middle layer (the 8th of 18), forcing that layer to carry enough information to reconstruct the target character sequence; speech-guided alignment applies a negative cosine-similarity loss at a later layer (the 12th or 16th) against the last-layer output of frozen HuBERT, so the intermediate acoustic representation inherits speaker and prosodic structure. The paper reports that adding these two losses to F5-TTS lowers WER from 2.68 to 1.97 on LibriSpeech-PC, raises SIM from 0.60 to 0.62, and reaches comparable quality in roughly half the training steps, as shown in its convergence curves.

Load-bearing premise

The claimed 'doubling of convergence speed' is measured in training steps, not wall-clock time; the added per-step work of a frozen HuBERT forward pass, CTC projection, and interpolation could offset part or all of the step reduction, and the paper reports no per-step latency or total training duration.

Editorial extensions

If this is right

Models trained with A-DMA keep the exact same inference pipeline and parameters, since the projection head, CTC branch, and SSL model are dropped after training; deployed latency is unaffected.
The benefit appears in low-resource settings: on 0.6k hours of LibriTTS, A-DMA-F5-TTS beats the 100k-hour high-resource F5-TTS on WER (1.97 vs 2.42), suggesting alignment losses can substitute for data scale.
The same dual-alignment recipe transfers to E2 TTS, improving it from WER 3.10 to 2.25 and SIM 0.63 to 0.65, so the approach is not tied to the F5-TTS architecture.
The layer-depth results predict a systematic placement rule for future designs: text alignment near layer 8 and speech alignment near layers 12–16 works better than aligning both at the same depth.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the step-level speedup survives wall-clock accounting, A-DMA-type alignment could make training a diffusion TTS on a single GPU or with much smaller data practical, since the expensive SSL module is frozen and shared.
The same recipe—CTC on an inner layer plus SSL cosine alignment on a deeper layer—may transfer to other generative sequence models, such as diffusion vocoders or speech editing, where the modality gap is similar.
A testable extension: replace the fixed layer indices with learned layer selection or progressive alignment that starts at early layers and moves deeper as training proceeds, which the paper's layer sweep suggests could outperform any single fixed placement.
Since the paper reports only steps, a natural follow-up is to report wall-clock time and throughput (samples per second); that would settle whether the added frozen-HuBERT cost is amortized.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, read this paper if you care about training cost for diffusion TTS. The method is simple and the ablations are honest, but the headline claim—"doubles convergence speed"—is measured in training steps, not wall-clock time, and the paper never reports per-step overhead. That is the thing to keep in mind.

What is actually new: the authors add two auxiliary losses to F5-TTS small: a CTC loss on an intermediate transformer layer (text alignment) and a cosine-similarity loss against frozen HuBERT features on a later layer (speech alignment). The layer-depth study shows text alignment works best around layer 12 and speech alignment also around layer 12, but combining them works better with text at layer 8 and speech at layer 16. That asymmetric placement is a reasonable finding, and the ablations are thorough: they compare SSL models (HuBERT vs WavLM), last vs averaged layers, and three loss formulations. The empirical gains are consistent: at the same number of training steps, A-DMA improves WER and SIM over the baseline, and the inference pipeline is untouched. The code is promised. That is a decent incremental contribution, building on prior work (ref [18]) that already aligned hidden states in diffusion transformers; applying it to TTS with dual modalities and the layer analysis is the new part.

The soft spots are real but not fatal. First, the speedup claim does not account for the extra per-step compute. The speech alignment loss requires resampling the audio to 16 kHz, running a frozen HuBERT-large forward pass on a batch of ~1000 seconds of audio, interpolating, and a Conv1D. HuBERT-large forward is not trivial; unless the features are precomputed and cached (the paper does not say), the wall-clock speedup could be far less than 2x. The paper's Section 6 limitation ("does not target model efficiency") is about inference, so it doesn't address this. Second, the evaluation is a single run on a single test set (LibriSpeech-PC test-clean) with layer and hyperparameter choices made on that set. No error bars, no variance. That is common in TTS papers, but it means the "doubles" should be read with caution. There is no circularity problem: the losses are auxiliary and the teacher is external.

My take: the paper deserves serious peer review. The methodology is sound, the ablations are a step above the usual, and the weakness is a missing measurement rather than a conceptual flaw. A good referee would ask for wall-clock training time or an explicit caching strategy, and ideally a second dataset or repeated runs. With that, the paper would be solid.

Recommendation: engage with it. Send to review, but flag the wall-clock issue clearly.

Referee Report

4 major / 5 minor

Summary. The paper proposes A-DMA, a training-time augmentation for diffusion-based text-to-speech models that adds two auxiliary alignment losses to the conditional flow matching (CFM) objective: a CTC-based text alignment loss applied at an intermediate DiT layer and a cosine-similarity speech alignment loss that matches intermediate representations to frozen HuBERT features. The method is evaluated on LibriTTS/LibriSpeech-PC in low-resource settings, applied to F5-TTS and E2-TTS. The central claims are that A-DMA doubles convergence speed relative to the F5-TTS baseline and improves WER and SIM, while leaving the inference pipeline unchanged.

Significance. If the convergence-speedup claim holds in wall-clock terms, this is a practically valuable contribution: it addresses a real bottleneck in diffusion-based TTS training and the method is simple to graft onto existing architectures. The paper has strengths: the design is clearly motivated, the ablation study in Table 2 systematically explores alignment depth for each modality, and the authors commit to releasing code and demo samples. The use of an external, frozen SSL teacher (HuBERT) avoids circularity in the loss design. However, the headline speedup is only demonstrated in training steps on a single benchmark with no wall-clock measurements, no convergence threshold, and no variance reporting, so the empirical support for the main claim is currently incomplete.

major comments (4)

[Sec. 3.3, Figure 2] The 'doubles convergence speed' claim is measured only in training steps. The method adds per-step computation: a CTC projection head, a 16 kHz resampling of target audio, a frozen HuBERT-large forward pass, an interpolation layer, and a Conv1D layer (Secs. 2.2-2.4). With a total batch size of approximately 1k seconds of audio, the HuBERT-large forward pass alone is comparable in scale to the F5-TTS-small forward pass. No wall-clock training time or per-step latency is reported anywhere in Sec. 3.3 or Figure 2. Unless HuBERT features are precomputed and cached offline, which the paper does not state, halving the number of steps does not necessarily halve training time. The paper should report wall-clock convergence curves or explicitly describe a feature-caching scheme.
[Figure 2] The speedup is not defined with a convergence threshold. The figure plots WER and SIM at discrete step counts (50k-400k), but the paper never states what target quality level defines 'convergence' or how the '2x' factor is computed. For example, one could read the claim as reaching the baseline's final WER in half the steps, or as reaching some absolute WER threshold. A precise definition, ideally with error bars or multiple seeds, is needed before the doubling claim can be evaluated.
[Sec. 4.2, Table 2; Sec. 2.4] The key hyperparameters lambda_text, lambda_speech, text_align_layer, and speech_align_layer appear to be selected on the same LibriSpeech-PC test-clean set used for the final comparison. Table 2 is used both to select the alignment depths (e.g., text at 8th layer, speech at 16th layer) and then to support the superior performance claim of the final configuration. A held-out validation split or a separate development set should be used for such model selection, or the authors should report sensitivity of the final WER/SIM to these choices. Without this, the risk of overfitting to the test set is not addressed.
[Table 1, Sec. 4.1] All training/evaluation results appear to come from a single run, with no standard deviations or multiple seeds. The reported differences in Table 1 are small (e.g., F5-TTS WER 2.68 vs. F5-TTS+A-DMA 1.97, SIM 0.60 vs. 0.62) and could be within run-to-run variance. Adding at least two or three seeds with variance bars would substantially strengthen the claims of superior performance. Additionally, the convergence plot in Figure 2 is only shown for F5-TTS; the claimed speedup for E2-TTS is not demonstrated.

minor comments (5)

[Sec. 6] The limitation paragraph says 'this work does not target model efficiency,' but this appears to refer to inference-time model complexity; the paper should clarify that the training-time overhead of the extra modules (especially the frozen HuBERT pass) is a separate consideration and should be discussed.
[Sec. 2.3] The interpolation and Conv1D details are underspecified: the paper should state the interpolation method (e.g., linear or nearest-neighbor), the output dimension of the Conv1D, and the dimension alignment between the TTS hidden state and the HuBERT feature.
[Sec. 3.3] The model setup states 'Training is conducted on 4 RTX A6000 GPUs' but does not report effective batch size per GPU, gradient accumulation, or total training steps; this makes it hard for readers to reproduce the '1k seconds' batch statement.
[Sec. 4.3, Table 3] The notation in Table 3 is inconsistent with the main text: the table uses 'Phi_n' while Eq. (3) uses 'Phi_i', and the 'Avg.' row is not defined precisely in the text; please align notation and explain what averaging over layers means.
[Overall] There are several typographical issues, including inconsistent spacing in 'V oiceBox' in Sec. 1 and 'W2V-Bert' in the references; a careful proofread is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: A-DMA's alignment losses are supervised by external targets and evaluated on independent benchmarks; the F5-TTS baseline self-citation is not load-bearing.

full rationale

The paper's derivation chain is not circular. A-DMA adds two auxiliary losses to the CFM objective (Sec. 2.4): L_text is a CTC loss computed against the input transcript from an intermediate hidden state, and L_speech is a negative cosine similarity between a projected hidden state and features from a frozen HuBERT model on the target audio. Both supervision signals are external to the TTS model's own prediction; neither loss is defined in terms of the final output metric or the claimed speedup. The central convergence-speed claim is empirical: Figure 2 compares WER and SIM at matched training-step checkpoints for A-DMA versus the baseline, and Tables 1-3 report independent benchmark metrics (Whisper-large-V3 WER, WavLM-based SIM, UTMOS). No fitted parameter is renamed as a prediction, and no equation reduces to its input by construction. The baseline F5-TTS (ref [9]) does have overlapping authors with the present paper, but it is a public, externally benchmarked system, and the comparison to it is empirical rather than asserted by citation. The limitation that the paper does not report wall-clock training time is a real evaluation gap for the "doubles convergence speed" claim, but it is a measurement/completeness issue, not a circularity issue, because the step-based speedup is not self-referential. The paper is self-contained against external benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard flow-matching math plus two domain assumptions about alignment losses, and on a post hoc selection of loss weights and layer indices on the same benchmark used for evaluation.

free parameters (5)

lambda_text = 0.1
Weight for text alignment loss in Sec. 2.4; empirically set, no sensitivity analysis reported.
lambda_speech = 1.0
Weight for speech alignment loss in Sec. 2.4; empirically set, no sensitivity analysis reported.
text_align_layer = 8
Layer index for text CTC alignment, selected from layer-wise ablation in Table 2 using the same benchmark as final results.
speech_align_layer = 12
Layer index for speech alignment, selected from Tables 2 and 3; 8th and 16th layers also tried.
ssl_output = Last layer of HuBERT
Table 3 shows average layer output improves SIM but degrades WER; last layer chosen as the best trade-off.

assumptions (4)

standard math Conditional flow matching loss (Eq. 2) accurately trains the diffusion TTS model
Adopted from F5-TTS/VoiceBox; assumed valid for training.
domain assumption CTC loss on intermediate hidden states provides a useful text alignment signal
Inspired by InterCTC in ASR [22,23]; the paper does not prove it helps, only shows empirically.
domain assumption Frozen HuBERT features are a suitable teacher for TTS hidden representations
Follows from refs [24,25]; the paper does not analyze failure modes or distribution mismatch.
ad hoc to paper Selected lambda and layer indices generalize beyond the single LibriSpeech-PC test benchmark
No validation set or cross-validation is reported; choices are post hoc.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment." pith.science (2026). https://pith.science/paper/TAGCTIWO

@misc{pith2026250519595,
  author       = {Pith},
  title        = {Pith review of: Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TAGCTIWO}},
  note         = {Machine review of arXiv:2505.19595}
}

read the original abstract

The goal of this paper is to optimize the training process of diffusion-based text-to-speech models. While recent studies have achieved remarkable advancements, their training demands substantial time and computational costs, largely due to the implicit guidance of diffusion models in learning complex intermediate representations. To address this, we propose A-DMA, an effective strategy for Accelerating training with Dual Modality Alignment. Our method introduces a novel alignment pipeline leveraging both text and speech modalities: text-guided alignment, which incorporates contextual representations, and speech-guided alignment, which refines semantic representations. By aligning hidden states with discriminative features, our training scheme reduces the reliance on diffusion models for learning complex representations. Extensive experiments demonstrate that A-DMA doubles the convergence speed while achieving superior performance over baselines. Code and demo samples are available at: https://github.com/ZhikangNiu/A-DMA

Figures

Figures reproduced from arXiv: 2505.19595 by the authors.

**Figure 1.** Overall training framework. The model uses masked speech, noisy speech, and transcript to predict the masked part of the speech via flow matching. The SSL model and CTC layer are utilized only during training, with the SSL parameters frozen throughout the process. into a character sequence with filler tokens, but E2 TTS suffers from slow convergence and low robustness. To address these limitations, F5-TTS [9] introd… view at source ↗

**Figure 2.** The WER and SIM results showing the effectiveness of the proposed dual modality alignment. sists of 2.2 hours of data from the LibriSpeech [32] test-clean subset, comprising 1,127 samples, each ranging from 4 to 10 seconds in length. It is specifically designed to facilitate comparison across different models in the research community and is publicly available for benchmarking. Specifically, we assess its ability t… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 33 canonical work pages

[18]

Accelerating codec-based speech synthesis with multi- token prediction and speculative decoding,

T. D. Nguyen, J.-H. Kim, J. Choi, S. Choi, J. Park, Y . Lee, and J. S. Chung, “Accelerating codec-based speech synthesis with multi- token prediction and speculative decoding,” in Proc. ICASSP , 2025

work page 2025
[1]

Introduction Text-to-Speech (TTS) aims to synthesize natural human speech x from text inputs y. Recent advancements, driven by scaling up both data and model size, have enabled the generation of nat- ural speech that closely mimics the voice of any given reference speech x′, a capability known as zero-shot TTS. Existing meth- ods for zero-shot TTS can be ...

work page
[2]

Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment

Method As illustrated in Figure 1, our model introduces an improved training paradigm for a diffusion-based text-to-speech (TTS) system, leveraging dual-modality alignment to accelerate con- vergence while maintaining a straightforward inference process and high speech fidelity. In this section, we provide a detailed introduction to our proposed method, h...

work page Pith review arXiv 2025
[3]

Dataset We train our model on the LibriTTS [31] dataset, a multi- speaker English corpus containing approximately 585 hours of read speech sampled at 24 kHz

Experiments 3.1. Dataset We train our model on the LibriTTS [31] dataset, a multi- speaker English corpus containing approximately 585 hours of read speech sampled at 24 kHz. The dataset is derived from the LibriSpeech [32] corpus and curated specifically for TTS research. Utterances with significant background noise are ex- cluded to ensure high-quality ...

work page
[4]

speech align

Results 4.1. Quantitative Comparison Table 1 shows the evaluation results of previous methods and our proposed approach A-DMA on LibriSpeech-PC [36] test- clean dataset. As shown in high-resource data setting, E2 TTS and F5-TTS which are diffusion-based NAR models demon- strate comparable performance to state-of-the-art AR models such as CosyV oice [29] a...

work page
[5]

This ap- proach significantly accelerates model convergence while en- hancing the quality of generated speech

Conclusion In this paper, we propose A-DMA, a novel modality alignment strategy that effectively guides diffusion-based text-to-speech models by leveraging discriminative representations. This ap- proach significantly accelerates model convergence while en- hancing the quality of generated speech. Experimental results validate that our approach outperform...

work page
[6]

In future work, we plan to further op- timize the framework by reducing model complexity and accel- erating inference

Limitation Although our extensive layer-wise experiments demonstrate that the proposed dual modality alignment leads to superior gen- eration performance with faster convergence, this work does not target model efficiency. In future work, we plan to further op- timize the framework by reducing model complexity and accel- erating inference

work page
[7]

Choi, J.-H

Acknowledgements J. Choi, J.-H. Kim, and J. S. Chung were supported by IITP grant funded by the Korea government (MSIT) (RS-2024- 00457882, National AI Research Lab Project). Z. Niu and X. Chen were supported by the National Natural Science Founda- tion of China (No. U23B2018 and No. 62206171), Shanghai Municipal Science and Technology Major Project under...

work page 2024

Show all 45 references

[8]

Neural codec language models are zero-shot text to speech synthesizers,

C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Neural codec language models are zero-shot text to speech synthesizers,” arXiv:2301.02111, 2023

2023 arXiv
[9]

Seed-tts: A family of high-quality versatile speech generation models,

P. Anastassiou, J. Chen, J. Chen, Y . Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao et al. , “Seed-tts: A family of high-quality versatile speech generation models,” arXiv:2406.02430, 2024

2024 arXiv
[10]

V oicecraft: Zero-shot speech editing and text-to-speech in the wild,

P. Peng, P.-Y . Huang, D. Li, A. Mohamed, and D. Harwath, “V oicecraft: Zero-shot speech editing and text-to-speech in the wild,” in Proc. ACL, 2024

2024
[11]

Base tts: Lessons from building a billion-parameter text- to-speech model on 100k hours of data,

M. Łajszczak, G. C ´ambara, Y . Li, F. Beyhan, A. van Korlaar, F. Yang, A. Joly, ´A. Mart ´ın-Cortinas, A. Abbas, A. Michalski et al., “Base tts: Lessons from building a billion-parameter text- to-speech model on 100k hours of data,”arXiv:2402.08093, 2024

2024 arXiv
[12]

Autoregressive speech synthesis without vector quantization,

L. Meng, L. Zhou, S. Liu, S. Chen, B. Han, S. Hu, Y . Liu, J. Li, S. Zhao, X. Wu et al., “Autoregressive speech synthesis without vector quantization,” arXiv:2407.08551, 2024

2024 arXiv
[13]

V oice- box: Text-guided multilingual universal speech generation at scale,

M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V . Manohar, Y . Adi, J. Mahadeokaret al., “V oice- box: Text-guided multilingual universal speech generation at scale,” in NeurIPS, 2024

2024
[14]

Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,

K. Shen, Z. Ju, X. Tan, Y . Liu, Y . Leng, L. He, T. Qin, S. Zhao, and J. Bian, “Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,” in Proc. ICLR, 2024

2024
[15]

E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts,

S. E. Eskimez, X. Wang, M. Thakker, C. Li, C.-H. Tsai, Z. Xiao, H. Yang, Z. Zhu, M. Tang, X. Tan et al., “E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts,” inIEEE Spoken Lan- guage Technology workshop, 2024

2024
[16]

F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,” arXiv:2410.06885, 2024

2024 arXiv
[17]

Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer,

K. Lee, D. W. Kim, J. Kim, and J. Cho, “Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer,” in Proc. ICLR, 2025

2025
[19]

Vall-e r: Robust and efficient zero-shot text-to-speech synthesis via monotonic alignment,

B. Han, L. Zhou, S. Liu, S. Chen, L. Meng, Y . Qian, Y . Liu, S. Zhao, J. Li, and F. Wei, “Vall-e r: Robust and efficient zero-shot text-to-speech synthesis via monotonic alignment,” arXiv:2406.07855, 2024

2024 arXiv
[20]

Vall-t: Decoder-only generative transducer for robust and decoding-controllable text-to-speech,

C. Du, Y . Guo, H. Wang, Y . Yang, Z. Niu, S. Wang, H. Zhang, X. Chen, and K. Yu, “Vall-t: Decoder-only generative transducer for robust and decoding-controllable text-to-speech,” in Proc. ICASSP, 2025

2025
[21]

Flow matching for generative modeling,

Y . Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” inProc. ICLR, 2023

2023
[22]

Scalable diffusion models with transform- ers,

W. Peebles and S. Xie, “Scalable diffusion models with transform- ers,” in Proc. ICCV, 2023

2023
[23]

Efficient diffusion training via min-snr weighting strat- egy,

T. Hang, S. Gu, C. Li, J. Bao, D. Chen, H. Hu, X. Geng, and B. Guo, “Efficient diffusion training via min-snr weighting strat- egy,” inProc. ICCV, 2023

2023
[24]

Fasterdit: Towards faster diffusion transformers training without architecture modification,

J. Yao, C. Wang, W. Liu, and X. Wang, “Fasterdit: Towards faster diffusion transformers training without architecture modification,” in NeurIPS, 2024

2024
[25]

Representation alignment for generation: Training diffu- sion transformers is easier than you think,

S. Yu, S. Kwak, H. Jang, J. Jeong, J. Huang, J. Shin, and S. Xie, “Representation alignment for generation: Training diffu- sion transformers is easier than you think,” in Proc. ICLR, 2025

2025
[26]

U-net: Convolutional networks for biomedical image segmentation,

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Proc. MICCAI, 2015

2015
[27]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, 2017

2017
[28]

Con- nectionist temporal classification: Labelling unsegmented se- quence data with recurrent neural networks,

A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: Labelling unsegmented se- quence data with recurrent neural networks,” in Proc. ICML , 2006

2006
[29]

Intermediate loss regularization for ctc- based speech recognition,

J. Lee and S. Watanabe, “Intermediate loss regularization for ctc- based speech recognition,” in Proc. ICASSP, 2021

2021
[30]

Audio-visual efficient conformer for robust speech recognition,

M. Burchi and R. Timofte, “Audio-visual efficient conformer for robust speech recognition,” in Proc. WACV, 2023

2023
[31]

Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,

X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,” in Proc. ICLR, 2024

2024
[32]

Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,

Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y . Liu, Y . Leng, K. Song, S. Tang et al. , “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” in Proc. ICML, 2024

2024
[33]

Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM Trans. on Audio, Speech, and Language Processing, 2021

2021
[34]

Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, 2022

2022
[35]

W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,

Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in IEEE Automatic Speech Recognition and Understanding work- shop, 2021

2021
[36]

Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Ma et al. , “Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,” arXiv:2407.05407, 2024

2024 arXiv
[37]

Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications,

H.-H. Guo, K. Liu, F.-Y . Shen, Y .-C. Wu, F.-L. Xie, K. Xie, and K.-T. Xu, “Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications,”arXiv:2409.03283, 2024

2024 arXiv
[38]

Libritts: A corpus derived from librispeech for text- to-speech,

H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “Libritts: A corpus derived from librispeech for text- to-speech,” in Proc. Interspeech, 2019

2019
[39]

Lib- rispeech: An asr corpus based on public domain audio books,

V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An asr corpus based on public domain audio books,” in Proc. ICASSP, 2015

2015
[40]

Convnext v2: Co-designing and scaling convnets with masked autoencoders,

S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “Convnext v2: Co-designing and scaling convnets with masked autoencoders,” in Proc. CVPR, 2023

2023
[41]

Decoupled weight decay regulariza- tion,

I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” in Proc. ICLR, 2019

2019
[42]

V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis,

H. Siuzdak, “V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis,” in Proc. ICLR, 2024

2024
[43]

Librispeech-pc: Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr mod- els,

A. Meister, M. Novikov, N. Karpov, E. Bakhturina, V . Lavrukhin, and B. Ginsburg, “Librispeech-pc: Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr mod- els,” in IEEE Automatic Speech Recognition and Understanding workshop, 2023

2023
[44]

Robust speech recognition via large-scale weak su- pervision,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in Proc. ICML, 2023

2023
[45]

Utmos: Utokyo-sarulab system for voicemos challenge 2022,

T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “Utmos: Utokyo-sarulab system for voicemos challenge 2022,” in Proc. Interspeech, 2022

2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [18]

Accelerating codec-based speech synthesis with multi- token prediction and speculative decoding,

T. D. Nguyen, J.-H. Kim, J. Choi, S. Choi, J. Park, Y . Lee, and J. S. Chung, “Accelerating codec-based speech synthesis with multi- token prediction and speculative decoding,” in Proc. ICASSP , 2025

work page 2025

[2] [1]

Introduction Text-to-Speech (TTS) aims to synthesize natural human speech x from text inputs y. Recent advancements, driven by scaling up both data and model size, have enabled the generation of nat- ural speech that closely mimics the voice of any given reference speech x′, a capability known as zero-shot TTS. Existing meth- ods for zero-shot TTS can be ...

work page

[3] [2]

Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment

Method As illustrated in Figure 1, our model introduces an improved training paradigm for a diffusion-based text-to-speech (TTS) system, leveraging dual-modality alignment to accelerate con- vergence while maintaining a straightforward inference process and high speech fidelity. In this section, we provide a detailed introduction to our proposed method, h...

work page Pith review arXiv 2025

[4] [3]

Dataset We train our model on the LibriTTS [31] dataset, a multi- speaker English corpus containing approximately 585 hours of read speech sampled at 24 kHz

Experiments 3.1. Dataset We train our model on the LibriTTS [31] dataset, a multi- speaker English corpus containing approximately 585 hours of read speech sampled at 24 kHz. The dataset is derived from the LibriSpeech [32] corpus and curated specifically for TTS research. Utterances with significant background noise are ex- cluded to ensure high-quality ...

work page

[5] [4]

speech align

Results 4.1. Quantitative Comparison Table 1 shows the evaluation results of previous methods and our proposed approach A-DMA on LibriSpeech-PC [36] test- clean dataset. As shown in high-resource data setting, E2 TTS and F5-TTS which are diffusion-based NAR models demon- strate comparable performance to state-of-the-art AR models such as CosyV oice [29] a...

work page

[6] [5]

This ap- proach significantly accelerates model convergence while en- hancing the quality of generated speech

Conclusion In this paper, we propose A-DMA, a novel modality alignment strategy that effectively guides diffusion-based text-to-speech models by leveraging discriminative representations. This ap- proach significantly accelerates model convergence while en- hancing the quality of generated speech. Experimental results validate that our approach outperform...

work page

[7] [6]

In future work, we plan to further op- timize the framework by reducing model complexity and accel- erating inference

Limitation Although our extensive layer-wise experiments demonstrate that the proposed dual modality alignment leads to superior gen- eration performance with faster convergence, this work does not target model efficiency. In future work, we plan to further op- timize the framework by reducing model complexity and accel- erating inference

work page

[8] [7]

Choi, J.-H

Acknowledgements J. Choi, J.-H. Kim, and J. S. Chung were supported by IITP grant funded by the Korea government (MSIT) (RS-2024- 00457882, National AI Research Lab Project). Z. Niu and X. Chen were supported by the National Natural Science Founda- tion of China (No. U23B2018 and No. 62206171), Shanghai Municipal Science and Technology Major Project under...

work page 2024