pith. sign in

arxiv: 1803.09047 · v1 · pith:2XA2ZQC6new · submitted 2018-03-24 · 💻 cs.CL · cs.LG· cs.SD· eess.AS

Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron

RJ Skerry-Ryan , Eric Battenberg , Ying Xiao , Yuxuan Wang , Daisy Stanton , Joel Shor , Ron J. Weiss , Rob Clark
show 1 more author
Rif A. Saurous
This is my paper
classification 💻 cs.CL cs.LGcs.SDeess.AS
keywords prosodyreferencetacotronembeddingsynthesistransferaudiodifferent
0
0 comments X
read the original abstract

We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody. We show that conditioning Tacotron on this learned embedding space results in synthesized audio that matches the prosody of the reference signal with fine time detail even when the reference and synthesis speakers are different. Additionally, we show that a reference prosody embedding can be used to synthesize text that is different from that of the reference utterance. We define several quantitative and subjective metrics for evaluating prosody transfer, and report results with accompanying audio samples from single-speaker and 44-speaker Tacotron models on a prosody transfer task.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark

    cs.CL 2026-04 unverdicted novelty 7.0

    CAST benchmark shows language models infer correct word stress from discourse context but TTS systems frequently fail to produce it in speech.

  2. Forward-Backward Decoding for Regularizing End-to-End TTS

    eess.AS 2019-07 unverdicted novelty 6.0

    Forward-backward decoding with divergence regularization and bidirectional decoder improves end-to-end TTS robustness and naturalness by addressing exposure bias via joint L2R/R2L training.

  3. M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention

    cs.CV 2019-07 unverdicted novelty 5.0

    M3D-GAN introduces modality subnets, a shared computing body, and a universal attention module to enable translation across text, image, and speech modalities and their internal domains.

  4. Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning

    cs.CY 2019-07 unverdicted novelty 4.0

    The paper outlines considerations, analogies from other fields, and recommendations for ML research norms on synthetic media to mitigate misuse risks without prescribing specific policies.

  5. A Methodology for Controlling the Emotional Expressiveness in Synthetic Speech -- a Deep Learning approach

    eess.AS 2019-07 unverdicted novelty 3.0

    A methodology is proposed for emotional text-to-speech using emotional data collection, transfer-learning-based annotation of expressiveness features, and fine-tuning of a neutral TTS model.