Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TTS-CtrlNet shows that a frozen flow-matching text-to-speech model can be given time-varying emotion control by training only a ControlNet-style side branch, using about 400 hours of public emotional speech.

desk verdict A genuinely useful ControlNet adaptation for time-varying emotion TTS, but the headline Aro-Val metric is partly self-confirming and the time-varying claim isn't directly evaluated. read the letter →

arxiv 2507.04349 v1 pith:3S3OT74L submitted 2025-07-06 cs.SD eess.AS

Jaeseok Jeong , Yuna Lee , Mingi Kwon , Youngjung Uh This is my paper · ORCID
classification cs.SDeess.AS
keywords text-to-speechemotioncontrolNetflowmatchingzero-shotvoicecloningspeechrecognitionarousal-valencetime-varying
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a large, frozen text-to-speech model can be given fine-grained, time-varying emotion control by attaching a trainable copy of part of itself, in the style of ControlNet, that reads arousal–valence emotion features extracted from the reference audio. The authors show this on F5-TTS, a flow-matching zero-shot voice-cloning model, using about 400 hours of public emotional speech data to train only the control branch. They report that the approach achieves higher emotion similarity scores (Emo-SIM and Aro-Val SIM) than full-fine-tuning baselines while keeping the original model's speaker similarity and naturalness, and they give recipes for which blocks to connect, which flow steps to use, and how to scale the control. If correct, this means emotion control can be added to existing TTS systems without retraining the backbone or collecting a huge labeled emotion dataset.

What carries the argument

The central object is a ControlNet-style side branch: a trainable copy of selected DiT transformer blocks from the frozen F5-TTS model, connected to the original blocks through zero-initialized convolutions so that the added conditioning (the smoothed arousal–valence sequence) starts at zero effect and grows during training. Around it sit three design choices that carry the argument: a block-selection rule that avoids connecting blocks whose removal hurts word error rate and speaker similarity; an emotion-specific flow-step interval (t in [0, temo], with temo ≈ 0.1) where emotion is established and where the control branch is active; and a control scale λ that trades emotion expressiveness against intelligibility at inference.

What would settle it

Compute Aro-Val SIM on the same generated samples using a different, independently trained time-varying emotion recognizer (or frame-level human annotations) and check whether the reported gains over baselines persist; if the gains come mostly from sharing the same SER model, the emotion-transfer claim would be weakened.

Watch

Extended reading notes

Core claim

TTS-CtrlNet is built by freezing the pre-trained flow-matching TTS model and adding a trainable copy of its DiT blocks, connected to the original via zero-convolution so that training starts from the identity behavior. The additional input is a time-varying emotion signal — per-token arousal and valence values predicted by a wav2vec-based speech emotion recognition model, smoothed by a temporal window — which is injected after a 1×1 convolution to match the model's channel dimension. During training only the control branch is updated with the flow-matching loss, using a restricted flow-step interval [0, temo] that the authors identify as the range where emotion is determined. At inference a control scale λ lets the user interpolate between the original model's behavior and full emotion control. The paper reports that this setup yields the best Emo-SIM and Aro-Val SIM on the JVNV S2ST and EMO-Change benchmarks while preserving zero-shot voice cloning and naturalness, with word-error rates competitive on the English EMO-Change task.

Load-bearing premise

The wav2vec-based emotion recognizer, trained only on utterance-level arousal–valence labels, is assumed to give a faithful time-varying emotion signal once its token outputs are smoothed over a window; if that signal is wrong, both the conditioning that drives ControlNet and the headline Aro-Val SIM metric degrade together.

Editorial extensions

If this is right

  • Emotion control can be added to a large flow-matching TTS model without fine-tuning its weights, using roughly 400 hours of public emotional speech, so the cost of adding a new condition drops sharply.
  • Because the backbone is frozen, zero-shot voice cloning and naturalness of the original model are inherited, which means the same technique could be layered onto other large TTS models.
  • Restricting the control branch to the emotion-specific flow-step interval makes the added computation small at inference and improves emotion transfer.
  • The control scale λ gives a practical knob to balance intelligibility and emotional expressiveness, letting deployers tune per use case.
  • On the JVNV S2ST and EMO-Change benchmarks, the method reaches higher Emo-SIM and Aro-Val SIM scores than utterance-level emotion-conditioning baselines and full-fine-tuning approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same ControlNet recipe may transfer to other flow-matching generative models (image, video, or speech-translation systems) where a time-varying condition must be aligned with the content; the paper only tests TTS, but the mechanism is not TTS-specific.
  • Because the emotion representation comes from an utterance-level SER model, its windowed interpolation may smear or miss fast emotional changes (the authors note laughter and crying are not captured); a purpose-built time-varying emotion encoder would likely improve both control fidelity and the Aro-Val SIM numbers.
  • The emotion-specific flow-step finding suggests a general principle for flow-matching controllers: identify the flow interval where the target attribute is determined and apply conditioning only there, which could be investigated for prosody, speaking rate, or style.
  • A strict test of the approach would be to compute Aro-Val SIM with a different, independently trained emotion recognizer, since the paper's reported gains use the same family of SER model that provides the conditioning signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. TTS-CtrlNet proposes to add fine-grained, time-varying emotion control to a frozen flow-matching TTS model (F5-TTS) using a ControlNet-style trainable branch. The conditioning signal is a per-frame arousal/valence embedding extracted by a wav2vec-based speech emotion recognition model, and training uses roughly 400 hours of public emotional speech while the base model remains frozen. The paper contributes practical recipes: selective connection of DiT blocks, an emotion-specific flow-step interval [0, t_emo], and an inference-time control scale. Experiments on JVNV S2ST and EMO-Change report improved Emo-SIM and Aro-Val SIM over several baselines, with subjective MOS supporting preserved naturalness and speaker similarity.

Significance. If the central claims hold, the method offers a low-cost, general recipe for adding emotion control to large pre-trained TTS systems without full fine-tuning, and the block-level and flow-step analyses are useful design guidance for future controllability work. The paper is honest about computing the free parameters from ablations and about the limitations of the SER model. The main scientific value is conditional on the evaluation being trustworthy: Emo-SIM is an independent signal, but the headline Aro-Val SIM result is potentially circular, and the time-varying claim is not directly tested. These issues do not invalidate the architecture, but they do mean the state-of-the-art claim is not yet secure.

major comments (4)
  1. [§3.2, §4.2, Table 5] The Aro-Val SIM metric is not independent of the conditioning signal: both the emotion embedding fed into ControlNet (§3.2, using the wav2vec-based SER model [36]) and the Aro-Val SIM evaluation (§4.2, using a sliding-window approach following [36]) rely on the same wav2vec SER model. Since the controller is trained to reproduce that model's arousal/valence outputs, reporting Aro-Val SIM from the same model can overstate perceived emotion transfer. The authors' own limitation statement concedes that the SER misses non-verbal cues such as laughter and crying, and the supplementary instructs listeners not to account for non-verbal transfer. Please add an independent check of emotional similarity (e.g., a different SER model, discrete emotion classification of the output, or listener-based emotion ratings) and report the gains on that metric separately.
  2. [Table 5, Table 6] The main quantitative comparison lacks error bars, confidence intervals, and significance tests, and most baseline rows (B1–B6) are adopted from [21] rather than reproduced under the authors' protocol. The reported differences are often small (e.g., EMO-Change Emo-SIM 0.724 vs 0.692 for F5-TTS), and the MOS results in Table 6 are given without significance testing. In addition, the comparison to EmoCtrl-TTS is asymmetric because that system uses both an emotion encoder and a nonverbal encoder, while TTS-CtrlNet uses only the emotion encoder; this asymmetry is disclosed only in the supplementary material. Please provide variance estimates for objective metrics, state clearly which rows were measured in-house, and justify the use of external numbers.
  3. [§4.1.2, §4.2, EMO-Change] The central 'time-varying' claim is not directly evaluated. EMO-Change is constructed by concatenating two utterances with different emotions, but the paper reports only whole-utterance Emo-SIM and Aro-Val SIM over sliding windows; there is no segment-level alignment or transition analysis. This makes it difficult to distinguish true time-varying emotional transfer from a single averaged emotion over the utterance. Please report per-segment similarities (e.g., first half vs second half of the generated speech) or an explicit boundary/transition metric to substantiate the time-varying control claim.
  4. [§3.2, Limitations] The time-varying emotion features are produced by interpolating utterance-level SER outputs over a window (W_emo=30), but the paper does not validate that these interpolated features are causally faithful time-varying emotion annotations. The feature extraction recipe is a heuristic: the SER is trained with mean pooling on utterance-level labels, and at inference token-level outputs are windowed and interpolated. The regime where this representation is acknowledged to fail (non-verbal cues such as laughter and crying) is exactly the regime excluded from the supplementary evaluation. Please add a validation of the temporal alignment of the conditioning features, or at least a quantitative analysis of how the interpolation affects the fidelity of the emotion trajectory.
minor comments (5)
  1. [Global] There are several typos and inconsistencies: 'valance' should be 'valence' (e.g., Figure 3), 'pre-traineded' in §2.2, 'enligsh' and 'datset' in §4.4.
  2. [§1] The list of practical recipes in the Introduction contains the numeral '3)' twice; remove the duplicate or renumber the items.
  3. [§3.1.1, Eq. (3)] Equation (3) uses the binary mask m in the block-output combination without a clear definition at that point; clarify whether this is the same temporal mask used for the infilling task and how it is applied during inference.
  4. [References] References [6] and [22] are the same F5-TTS paper; consolidate them to avoid duplicate citations.
  5. [Supplementary Table 7] The baseline IDs in Supplementary Table 7 do not match the row ordering of Table 5 (e.g., B4/B5 definitions shift between tables); align them for clarity.

Circularity Check

1 steps flagged · score 5.0 of 10

Aro-Val SIM is partially circular: the same wav2vec SER model both conditions TTS-CtrlNet and defines the headline metric.

  1. fitted input called prediction [Sec. 3.2 (emotion conditioning) and Sec. 4.2 (Aro-Val SIM metric); reported in Tables 1, 4, 5]
    "We use a wav2vec-based emotion recognition model [36], which predicts arousal-valence-dominance. ... The token is interpolated with the emotion window size Wemo and returns e ∈ R^{D_emo × T}. ... Aro-Val SIM evaluates alignment in continuous emotion space—arousal and valence—estimated using a sliding window approach following [36]."

    The ControlNet conditioning signal e is computed by the wav2vec-based SER model [36] with sliding-window interpolation, and the Aro-Val SIM evaluation metric is computed by the same [36] model with a sliding-window arousal/valence estimate. The controller is trained to make generated speech follow e (Eq. 4, with e extracted from the reference audio), so the evaluation function used for Aro-Val SIM is the same function that produced the conditioning target. A model that reproduces the conditioning SER trajectory in its output will score high on Aro-Val SIM even if human listeners perceive the emotion transfer differently; the large Aro-Val SIM gaps in Table 5 are therefore not an independent confirmation of the headline SOTA claim.

full rationale

The paper's core method—freezing F5-TTS and training a ControlNet copy on public emotion speech—is otherwise self-contained and is compared against external baselines, and no load-bearing self-citation chain is present. The main circular dependency is that the same wav2vec SER model [36] generates the emotion conditioning features (Sec. 3.2) and defines the Aro-Val SIM evaluation metric (Sec. 4.2). Since the flow-matching training objective conditions on e = SER(s_ref) and Aro-Val SIM measures SER-based similarity between reference and generated audio, the headline Aro-Val SIM improvements are confounded by the shared feature extractor. However, the paper also reports Emotion2Vec-based Emo-SIM and human EMOS, which are independent and still show improvements (e.g., Emo-SIM 0.692 to 0.724 on EMO-Change; EMOS 2.69 to 4.02 on EMO-Change in the supplementary), so the central claim retains independent content. The authors' own limitation that the SER cannot reliably capture non-verbal cues such as laughter and crying further weakens Aro-Val SIM as a standalone measure but does not invalidate the method. The score is set to 5 to reflect one significant partial circularity in the evaluation, not a fully self-referential derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on hyperparameters tuned on the same-style evaluation data (t_emo, W_emo, block subset, lambda), on the frozen-backbone assumption, and on the wav2vec SER model serving both as controller and as metric. No new physical or conceptual entities are introduced.

free parameters (4)
  • t_emo (emotion-specific flow step upper bound) = 0.1
    Upper bound of the training/inference flow-step interval; selected from RAVDESS ablation (Table 1) to maximize Emo-SIM and Aro-Val SIM while keeping WER low.
  • W_emo (emotion window size) = 30
    Window size for interpolating wav2vec features to obtain time-varying emotion; selected from ablation (Table 2) balancing WER and emotion similarity.
  • Selected DiT block subset = not enumerated in paper
    A subset of the 22 DiT blocks is connected to ControlNet based on a layer-skipping WER/speaker-similarity analysis (Sec 4.3.1, Table 3); the exact block indices are not reported.
  • lambda (control scale) = not reported for final results; ablated 0.0 to 1.0
    Inference-time scale for the ControlNet output, ablated in Table 4; the value used for the main comparisons in Table 5 is not stated.
assumptions (5)
  • standard math Conditional flow matching / OT-CFM provides a valid training objective for speech synthesis (Lipman et al., Eq 2).
    Adopted from prior work as the training loss; not derived in this paper.
  • domain assumption Freezing the original F5-TTS parameters while training the ControlNet copy preserves zero-shot voice cloning and naturalness.
    Zero-convolution initialization is assumed to yield graceful integration; supported only by the paper's own experiments.
  • domain assumption The wav2vec-based SER model [36] provides a causally adequate time-varying emotion representation via sliding-window interpolation.
    The conditioning input e is computed this way (Sec 3.2) and the same model is used to compute Aro-Val SIM (Sec 4.2).
  • ad hoc to paper The emotion-specific flow-step interval [0, 0.1] identified on RAVDESS transfers to other speakers, languages, and datasets.
    t_emo is chosen from a RAVDESS ablation (Table 1) and applied to all experiments without independent justification.
  • domain assumption A subset of DiT blocks can be excluded from ControlNet without hurting emotion control.
    Block selection is based on an empirical layer-skip analysis (Sec 4.3.1), not a theoretical guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet." pith.science (2026). https://pith.science/paper/3S3OT74L

@misc{pith2026250704349,
  author       = {Pith},
  title        = {Pith review of: TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3S3OT74L}},
  note         = {Machine review of arXiv:2507.04349}
}
read the original abstract

Recent advances in text-to-speech (TTS) have enabled natural speech synthesis, but fine-grained, time-varying emotion control remains challenging. Existing methods often allow only utterance-level control and require full model fine-tuning with a large emotion speech dataset, which can degrade performance. Inspired by adding conditional control to the existing model in ControlNet (Zhang et al, 2023), we propose the first ControlNet-based approach for controllable flow-matching TTS (TTS-CtrlNet), which freezes the original model and introduces a trainable copy of it to process additional conditions. We show that TTS-CtrlNet can boost the pretrained large TTS model by adding intuitive, scalable, and time-varying emotion control while inheriting the ability of the original model (e.g., zero-shot voice cloning & naturalness). Furthermore, we provide practical recipes for adding emotion control: 1) optimal architecture design choice with block analysis, 2) emotion-specific flow step, and 3) flexible control scale. Experiments show that ours can effectively add an emotion controller to existing TTS, and achieves state-of-the-art performance with emotion similarity scores: Emo-SIM and Aro-Val SIM. The project page is available at: https://curryjung.github.io/ttsctrlnet_project_page

Figures

Figures reproduced from arXiv: 2507.04349 by the authors.

Figure 1
Figure 1. Overview of TTS-CtrlNet Controlling signal is processed through ControlNet and fed into the subset of blocks in original model. x1. With µ0(x1) = 0 and σ0(x1) = 1, µ1(x1) = x1 and σ1(x1) = 0, a vector field of the flow is dψt (x0)/dt = ut (ψt (x0) | x1). Then, CFM loss becomes : LCFM(θ) = Et, q(x1), p(x0) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Layer ablation with quantitative results We measure WER and speaker similarity after skipping each block during inference and find that some blocks greatly increase WER and decrease speaker similarity when skipped. vocalizations. For S2ST translation, we first perform speech-to-text translation on the Japanese utterances to obtain English transcripts. These transcripts served as text prompts, while the original Japa… view at source ↗
Figure 3
Figure 3. Flow step experiments [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Survey page 1st on JVNV-S2ST and 2nd on EMO-Change, showing comparable performance to EmoCtrl-TTS. Considering the confidence intervals, the difference between 1st and 2nd place is not statistically significant. Compared to its baseline model F5-TTS, our method achieve…
Figure 5
Figure 5. Figure 5: Illustration of emotion-specific flow steps. (a) Slight perturbation of x0 preserves emotion similarity. (b) Perturbation beyond the emotion-specific flow step leads to failed emotion reconstruc￾tion and emotion shift. case. Perturbing x0 in a noise-like direction move…
Figure 6
Figure 6. Figure 6: Emotion window step Using window size = 1 fails to convey emotion properly, while setting a proper window size improves prediction consistency D More explanation of emotion window size In Section 4.3.3, we propose not to use emotion window size = 1 in the speech emotio…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation

    eess.AS 2026-04 unverdicted novelty 6.0 of 10

    Emotion embedding similarities are unsuitable for zero-shot evaluation of emotional expressiveness in speech generation due to confounding by non-emotional acoustic features.

Reference graph

Works this paper leans on

46 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [36]

    Dawn of the transformer era in speech emotion recognition: Closing the valence gap

    Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W Schuller. Dawn of the transformer era in speech emotion recognition: Closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–13, 2023

    work page 2023

  2. [21]

    Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech

    Haibin Wu, Xiaofei Wang, Sefik Emre Eskimez, Manthan Thakker, Daniel Tompkins, Chung-Hsien Tsai, Canrun Li, Zhen Xiao, Sheng Zhao, Jinyu Li, et al. Laugh now cry later: Controlling time-varying emotional states of flow-matching-based zero-shot text-to-speech. arXiv preprint arXiv:2407.12229, 2024

    work page Pith review arXiv 2024

  3. [1]

    Adding conditional control to text-to-image diffusion models, 2023

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. URL https://arxiv.org/abs/2302.05543

    arXiv 2023

  4. [2]

    An emotion speech synthesis method based on VITS

    Wei Zhao and Zheng Yang. An emotion speech synthesis method based on VITS. Applied Sciences, 13(4): 2225, 2023

    work page 2023

  5. [3]

    V oiceBox: Text-guided multilingual universal speech generation at scale

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. V oiceBox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024

    work page 2024

  6. [4]

    Making flow-matching-based zero-shot text-to-speech laugh as you like

    Naoyuki Kanda, Xiaofei Wang, Sefik Emre Eskimez, Manthan Thakker, Hemin Yang, Zirun Zhu, Min Tang, Canrun Li, Steven Tsai, Zhen Xiao, et al. Making flow-matching-based zero-shot text-to-speech laugh as you like. arXiv preprint arXiv:2402.07383, 2024

    arXiv 2024

  7. [5]

    E2 tts: Embarrass- ingly easy fully non-autoregressive zero-shot tts

    Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, Yanqing Liu, Sheng Zhao, and Naoyuki Kanda. E2 tts: Embarrass- ingly easy fully non-autoregressive zero-shot tts. 2024. URL https://api.semanticscholar.org/ CorpusID:270738197

    work page 2024

  8. [7]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024

    arXiv 2024

Show all 46 references
  1. [8]

    Tacotron: Towards end-to-end speech synthesis

    Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135, 2017

    2017 arXiv

  2. [9]

    Fastspeech: Fast, robust and controllable text to speech

    Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019

    2019

  3. [10]

    ED-TTS: Multi-scale emo- tion modeling using cross-domain emotion diarization for emotional speech synthesis

    Haobin Tang, Xulong Zhang, Ning Cheng, Jing Xiao, and Jianzong Wang. ED-TTS: Multi-scale emo- tion modeling using cross-domain emotion diarization for emotional speech synthesis. arXiv preprint arXiv:2401.08166, 2024. 10

    2024 arXiv

  4. [11]

    EmoDiff: Intensity controllable emotional text-to-speech with soft-label guidance

    Yiwei Guo, Chenpeng Du, Xie Chen, and Kai Yu. EmoDiff: Intensity controllable emotional text-to-speech with soft-label guidance. In ICASSP 2023, pages 1–5. IEEE, 2023

    2023

  5. [12]

    EmoMix: Emotion mixing via diffusion models for emotional speech synthesis

    Haobin Tang, Xulong Zhang, Jianzong Wang, Ning Cheng, and Jing Xiao. EmoMix: Emotion mixing via diffusion models for emotional speech synthesis. arXiv preprint arXiv:2306.00648, 2023

    2023 arXiv

  6. [13]

    Speech synthesis with mixed emotions

    Kun Zhou, Berrak Sisman, Rajib Rana, Björn W Schuller, and Haizhou Li. Speech synthesis with mixed emotions. IEEE Transactions on Affective Computing, 2022

    2022

  7. [14]

    QI-TTS: Questioning intonation control for emotional speech synthesis

    Haobin Tang, Xulong Zhang, Jianzong Wang, Ning Cheng, and Jing Xiao. QI-TTS: Questioning intonation control for emotional speech synthesis. In ICASSP 2023, pages 1–5. IEEE, 2023

    2023

  8. [15]

    MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis

    Yi Lei, Shan Yang, Xinsheng Wang, and Lei Xie. MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:853–864, 2022

    2022

  9. [16]

    Text-driven emotional style control and cross-speaker style transfer in neural tts

    Yookyung Shin, Younggun Lee, Suhee Jo, Yeongtae Hwang, and Taesu Kim. Text-driven emotional style control and cross-speaker style transfer in neural tts. arXiv preprint arXiv:2207.06000, 2022

    2022 arXiv

  10. [17]

    Emotional end-to-end neural speech synthesizer

    Younggun Lee, Azam Rabiee, and Soo-Young Lee. Emotional end-to-end neural speech synthesizer. arXiv preprint arXiv:1711.05447, 2017

    2017 arXiv

  11. [18]

    Controllable emotion transfer for end-to-end speech synthesis

    Tao Li, Shan Yang, Liumeng Xue, and Lei Xie. Controllable emotion transfer for end-to-end speech synthesis. In ISCSLP 2021, pages 1–5. IEEE, 2021

    2021

  12. [19]

    Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition

    Xiong Cai, Dongyang Dai, Zhiyong Wu, Xiang Li, Jingbei Li, and Helen Meng. Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition. In ICASSP 2021, pages 5734–5738. IEEE, 2021

    2021

  13. [20]

    Emosphere- tts: Emotional style and intensity modeling via spherical emotion vector for controllable emotional text-to-speech

    Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim, Sang-Hoon Lee, and Seong-Whan Lee. Emosphere- tts: Emotional style and intensity modeling via spherical emotion vector for controllable emotional text-to-speech. In Interspeech 2024, pages 1810–1814, 2024. doi: 10.21437/Interspeech...

    2024 doi

  14. [22]

    F5- tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5- tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024

    2024 arXiv

  15. [23]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021

    2021

  16. [24]

    wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020

    Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020. URL https://arxiv.org/abs/2006.11477

    2020 arXiv

  17. [26]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022

    2022

  18. [27]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022

    2022 arXiv

  19. [28]

    Black Forest Labs. Flux. https://github.com/black-forest-labs/flux , 2024

    2024

  20. [29]

    Scaling rectified flow transformers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL https://arxiv. org/abs/2403.03206, 2

    2024 arXiv

  21. [30]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023

    2023

  22. [31]

    Stable flow: Vital layers for training-free image editing

    Omri Avrahami, Or Patashnik, Ohad Fried, Egor Nemchinov, Kfir Aberman, Dani Lischinski, and Daniel Cohen-Or. Stable flow: Vital layers for training-free image editing. arXiv preprint arXiv:2411.14430, 2024. 11

    2024 arXiv

  23. [32]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR 2023, 2023. URL https://openreview.net/forum?id=PqvMRDCJT9t

    2023

  24. [33]

    Jointly predicting arousal, valence and dominance with multi-task learning

    Srinivas Parthasarathy and Carlos Busso. Jointly predicting arousal, valence and dominance with multi-task learning. In Interspeech 2017, pages 1103–1107, 2017. doi: 10.21437/Interspeech.2017-1494

    2017 doi

  25. [34]

    Automatic speech emotion recognition using recurrent neural networks with local attention

    Seyedmahdad Mirsamadi, Emad Barsoum, and Cha Zhang. Automatic speech emotion recognition using recurrent neural networks with local attention. 03 2017. doi: 10.1109/ICASSP.2017.7952552

    2017

  26. [35]

    Dia- loguegcn: A graph convolutional neural network for emotion recognition in conversation, 2019

    Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. Dia- loguegcn: A graph convolutional neural network for emotion recognition in conversation, 2019. URL https://arxiv.org/abs/1908.11540

    2019 arXiv

  27. [37]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT...

    2024

  28. [38]

    Libritts: A corpus derived from librispeech for text-to-speech

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882, 2019

    1904 arXiv

  29. [39]

    Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings

    Reza Lotfian and Carlos Busso. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions on Affective Computing, 10(4): 471–483, 2017

    2017

  30. [40]

    EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation

    Julius Richter, Yi-Chiao Wu, Steven Krenn, Simon Welker, Bunlong Lay, Shinjii Watanabe, Alexander Richard, and Timo Gerkmann. EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation. In ISCA Interspeech, pages 4873–4877, 2024

    2024

  31. [41]

    Iemocap: Interactive emotional dyadic motion capture database

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335–359, 2008

    2008

  32. [42]

    Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset

    Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 920–924. IEEE, 2021

    2021

  33. [43]

    Expresso: A benchmark and analysis of discrete expressive speech resynthesis

    Tu Anh Nguyen, Wei-Ning Hsu, Antony D’Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, Felix Kreuk, Yossi Adi, and Emmanuel Dupoux. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. InIntersp...

    2023

  34. [44]

    The ryerson audio-visual database of emotional speech and song (RA VDESS): A dynamic, multimodal set of facial and vocal expressions in north american english

    Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (RA VDESS): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391, 2018

    2018

  35. [45]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023

    2023

  36. [46]

    Seamless: Multilingual expressive and streaming speech translation

    Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul- Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, et al. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint arXiv:2312.05187, 2023

    2023 arXiv

  37. [47]

    supple_demo/index.html

    Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, ShiLiang Zhang, and Xie Chen. emo- tion2vec: Self-supervised pre-training for speech emotion representation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Li...

    2024 doi

  38. [2023]

    doi: 10.21437/Interspeech.2023-1905

    2023 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.