Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Emotional Face-to-Speech

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DEmoFace claims that a single expressive face can supply speaker identity and emotion for text-to-speech, matching or beating speech-prompted systems.

desk verdict New task framing for face-only emotional TTS, but the identity-transfer claim is undercut by a train/inference mismatch and a speaker-overlap test split. read the letter →

arxiv 2502.01046 v1 pith:UI6YYH6O submitted 2025-02-03 cs.SD cs.CVeess.AS

Jiaxin Ye , Boyuan Cao , Hongming Shan This is my paper · ORCID
classification cs.SDcs.CVeess.AS
keywords emotionalface-to-speechdiscretediffusionmodelresidualvectorquantizationmultimodaltransformercurriculumlearningpredictor-freeguidanceidentityembeddingalignmentface-driventext-to-speech
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a new task, emotional face-to-speech: given a text prompt and a single expressive face image, synthesize speech whose words match the text, whose voice sounds like the person in the image, and whose emotional tone matches the facial expression, without using any voice recording of that person. To solve it, the authors build DEmoFace, a discrete diffusion transformer that generates speech tokens from a multi-level neural audio codec, with curriculum learning and an enhanced predictor-free guidance for combining multiple conditions. The paper's central claim is that a face image alone carries enough information about identity and emotion to drive high-quality speech, and that the resulting output is more natural and consistent than existing face-to-speech methods, even outperforming methods that condition on an actual speech prompt. If true, this would let virtual characters be voiced directly from their faces and would offer a new interface for people with expressive language disorders.

What carries the argument

The load-bearing machinery is the cross-modal identity alignment combined with a 12-level residual vector quantization (RVQ) token space. The identity encoder merges ArcFace and FaceNet outputs through an MLP to form $c_{\mathrm{id}}$, trained with cosine-similarity, $\ell^1$, and $\ell^2$ losses against the GE2E speech embedding $c_{\mathrm{ge2e}}$; the reverse diffusion process then refines masked RVQ tokens, with one linear head per RVQ level predicting concrete scores. Curriculum learning exposes the model to low-frequency (low-level) tokens first and adds higher levels every three epochs, matching the codec's frequency hierarchy. Enhanced predictor-free guidance multiplies per-condition and joint guidance terms so that identity, emotion, and text conditions can be controlled separately or together.

What would settle it

Measure speaker identity similarity on a held-out set of speakers whose faces and voices never appeared in any training split, while holding emotion and text fixed; if speaker similarity drops to the level of an average-voice baseline, the cross-modal identity alignment is the broken link.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that identity and emotion can be decoupled from a single face and used as the only vocal-style conditions for a discrete diffusion speech generator. The identity embedding is obtained by concatenating ArcFace and FaceNet face embeddings, then aligning them to GE2E speaker embeddings extracted from ground-truth audio during training; at inference, the aligned face embedding replaces the speech embedding entirely. The emotion embedding is an identity-agnostic label embedding taken from a facial expression recognition network. The paper reports that DEmoFace improves emotion similarity, speaker similarity, F0 accuracy, and mel-cepstral distortion over prior face-to-speech and acoustic-guided baselines, while using far less training data than the main visual baseline.

Load-bearing premise

The load-bearing assumption is that the face-to-voice alignment learned from training speakers transfers to faces never seen in training, so that an unseen face still yields the right identity instead of an average or memorized voice.

Editorial extensions

If this is right

  • Virtual character dubbing can be driven directly by a character's face and facial expression, with no enrollment voice recording.
  • For people with expressive language disorders, a face image plus text could produce a personalized, emotionally colored voice in contexts where recording is impractical.
  • Speech-prompt-free conditioning removes the need to store or transmit reference audio, simplifying privacy and data handling in face-to-speech systems.
  • The reported gains over acoustic-guided baselines suggest that, at least within this evaluation setup, visual identity and emotion cues are not a degraded substitute for speech cues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the identity alignment generalizes beyond training speakers, face-based voice cloning without any enrollment audio becomes feasible; the same capability also makes impersonation and deepfake generation easier, so the consent caveat in the paper becomes a deployment requirement.
  • The reported speaker-similarity numbers are more convincing if they survive a strict held-out-speaker split; because MELD-FAIR follows original splits with speaker overlap, a re-run with zero speaker overlap would be the sharpest test of the transfer claim.
  • The paper's own limitation section says the model tends to produce average-sounding speech because of visual-voice biases in the data; if that bias dominates, the practical ceiling for face-only identity reconstruction is lower than the headline comparisons suggest.
  • A manuscript note: the appendix's description of the HPM baseline contains an unresolved citation marker for EmoFAN, which affects reproducibility of that baseline comparison but not the core eF2S claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces emotional face-to-speech (eF2S), a task in which a face image and text condition the synthesis of speech that matches the speaker's identity and the emotion expressed in the face. The proposed system, DEmoFace, tokenizes speech with an RVQ codec, trains a discrete diffusion transformer with multimodal conditioning, uses a coarse-to-fine curriculum over RVQ levels, and employs an enhanced predictor-free guidance (EPFG) scheme for multi-condition sampling. The model is trained on RAVDESS, MEAD, and MELD-FAIR, and evaluated with objective metrics (EmoSim, SpkSim, MCD, RMSE, WER) and a small subjective MOS study. The authors report that DEmoFace outperforms visual-guided baselines and even surpasses several acoustic-guided speech-driven methods.

Significance. If the central claim holds, DEmoFace would be a meaningful first step toward face-only emotional speech synthesis: the task formulation is clear, the use of discrete diffusion on RVQ tokens is technically current, and the ablations indicate that curriculum learning, identity alignment, and EPFG each contribute to the reported performance. The paper also provides demo audio and a reasonably detailed appendix. However, the headline claim of surpassing speech-driven methods rests on an identity-conditioning path that is trained with ground-truth speech embeddings and evaluated with a test split that contains speaker overlap, so the evidence as presented does not yet establish generalizable face-only identity transfer.

major comments (4)
  1. [Section 4.4] The training/inference mismatch for the identity conditioner is load-bearing and needs explicit treatment. The face encoder is trained to align cid to cge2e via Lalign, but the diffusion model itself is trained with the ground-truth speech embedding cge2e and only at inference receives the aligned face embedding cid. The model therefore denoises under a conditioning distribution it never saw during training. The reported gap between DEmoFace (SpkSim 0.668) and DEmoFace* (SpkSim 0.799) is consistent with incomplete cross-modal transfer, but the paper does not analyze this gap or test whether the face-conditioned pathway is actually calibrated. I ask for an experiment that either fine-tunes the model on cid conditioning or otherwise measures the distribution mismatch, and for a discussion of what the residual gap implies for the claim that face-only conditioning surpasses speech-driven methods.
  2. [Section 5.1 / Appendix D.1] The evaluation protocol does not isolate identity generalization from memorization. The paper states that MELD-FAIR follows its original splits, which contain speaker overlap between training and test, and the quantitative results in Table 1 are reported on the combined test set. For overlapping speakers, the face recognizer and the diffusion model may exploit speaker-specific associations learned during training, which would inflate SpkSim without demonstrating transfer to unseen faces. Please report metrics separately for RAVDESS/MEAD (where speakers are disjoint) and MELD-FAIR, and ideally for MELD-FAIR utterances whose speakers do not appear in training. Without this breakdown, the central claim of face-driven identity preservation is not supported.
  3. [Table 1 and Table 2] Objective results are reported as point estimates without confidence intervals or significance tests, and the subjective study uses only 15 participants and 10 randomly generated test samples. For a claim as strong as 'surpassing speech-driven methods,' the reader needs to know whether differences such as EmoSim 0.6965 versus 0.7010, or MCD 6.86 versus 6.89, are within noise. Please provide error bars or statistical tests on the objective metrics, and report the full subjective evaluation protocol including the number of ratings per sample.
  4. [Appendix H] The manuscript itself contains a limitation statement that undercuts the identity-preservation claim: it says that due to visual-voice biases the model 'tend[s] to produce average-sounding speech.' This is in tension with the abstract and Section 5.2 claims of preserving speaker identity and surpassing speech-driven methods. The authors should quantify this bias, for example by reporting speaker-similarity distributions relative to the ground-truth ceiling, and explain how the reported SpkSim values should be interpreted in light of this acknowledged averaging behavior.
minor comments (5)
  1. [Appendix C, Eq. (A-12)] The derivation contains typographical artifacts: the crossed-out terms 'XXXXXX' in the displayed equation should be removed or replaced with proper cancellation notation.
  2. [Equation (5)] The index notation is inconsistent: the text says 'wi for 1 ≤ k ≤ K' but the product uses wk; please unify the symbols.
  3. [Section 5.4] The ablation text says 'Fig. 3 shows' when discussing curriculum-learning results; the referenced figure appears to be Fig. 5. Please correct the cross-reference.
  4. [Section 4.3] The term 'enhanced predictor-free guidance' is introduced with minimal comparison to the existing PFG formulation of Nisonoff et al.; a short explicit statement of what is new beyond the joint-condition term would help readers assess the contribution.
  5. [Table 3] The ablation table would be easier to check if each row labeled with the removed component (e.g., 'w/o curriculum', 'w/o identity alignment', 'w/o EPFG') in addition to the current (a)-(c) labels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EPFG is a validation-tuned guidance variant, and identity alignment is a train/inference mapping with a stated limitation, not a fitted prediction.

full rationale

Walked the derivation chain. The core generator is a discrete diffusion transformer trained with the DSE objective (Eq. 3) on RVQ tokens, with separate identity, emotion, and text conditioners. EPFG (Eq. 5, Appendix C) is obtained from Bayes rule plus an energy-based product composition, which is a stated modeling choice; its guidance weights are selected by grid search on the validation set (Sec. 5.4, Fig. 6), not by fitting the reported test metrics. The identity mapper is trained with Lalign = 1 - cos(cid, cge2e) + L1 + L2 to a GE2E speech embedding, and DEmoFace is trained with cge2e but switched to cid at inference. This is a genuine train/inference distribution mismatch and a real generalization risk, especially with MELD-FAIR's original speaker-overlapping splits, but it is not circular: SpkSim is measured on generated speech with x-vectors, and the reported gap between DEmoFace (SpkSim 0.668) and DEmoFace* (SpkSim 0.799) shows the face-conditioned output is not forced to equal the speech-conditioned output. The only apparent self-citation, emotion2vec (Ma et al. 2024, which includes author Ye), is used as an external evaluation embedding, not as a fitted input or as justification of the method. Appendix H's admitted limitation that the model tends to produce average-sounding speech due to visual-voice biases further confirms the identity result is empirical, not definitional. No load-bearing step reduces to its own input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The framework rests on several external tools and one novel modeling assumption. The pre-trained codec, face and emotion encoders, GE2E, and SpeechT5 are taken from prior work without independent verification here; the identity alignment assumes these representations are compatible. The EPFG is justified only by an appendix derivation and an ablation, not by a formal correctness theorem. Guidance scales and the curriculum interval are tuned on validation data, adding free parameters that are not derived.

free parameters (3)
  • EPFG guidance scales (w0, w1, w2, w3) = w0=1.9, w1=1.0, w2=1.0, w3=1.6
    Chosen by grid search on the validation set (Sec. 5.4, Fig. 6a); the paper reports performance decays with lower scales.
  • Curriculum learning schedule (RVQ level entry period) = add one RVQ level every 3 epochs
    Hand-selected schedule; no sensitivity analysis for the 3-epoch interval is presented (Sec. 4.4).
  • Sampling steps = 96 steps (with acceptable quality at 32)
    Selected to balance quality and efficiency based on Fig. 6b; the paper uses 96 for reported results.
assumptions (5)
  • standard math The discrete diffusion framework with absorbing-state [MASK] and denoising score entropy (DSE) from Lou et al. (2024) correctly models the reverse process.
    Background method used without modification in Sec. 3; assumed sound based on the cited paper.
  • domain assumption The RVQ-based neural codec (Wang et al., 2024) provides a 12-level token representation where low levels capture semantics and high levels capture acoustic detail.
    Used to justify curriculum learning (Sec. 4.4, Fig. 5); cited to Nishimura et al. but not verified in this paper.
  • domain assumption Face recognition models ArcFace and FaceNet provide an identity embedding, and Poster2 predicts the emotion label used as the emotion condition.
    These pretrained models are taken as reliable visual encoders (Sec. 4.3); if emotion labels are noisy, generated emotion is degraded.
  • domain assumption GE2E speaker embeddings are a valid target for voice identity, and the face embedding can be aligned to them with the stated losses.
    Identity alignment in Sec. 4.4 relies on this cross-modal supervision; the paper provides no analysis of alignment error on unseen speakers.
  • ad hoc to paper The EPFG formula (Eq. 5) yields a valid conditional distribution for multi-condition sampling in discrete diffusion.
    The derivation in Appendix C uses an energy-based-model analogy to justify multiplying compositional and joint score ratios; this is a new modeling assumption not validated theoretically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Emotional Face-to-Speech." pith.science (2026). https://pith.science/paper/UI6YYH6O

@misc{pith2026250201046,
  author       = {Pith},
  title        = {Pith review of: Emotional Face-to-Speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UI6YYH6O}},
  note         = {Machine review of arXiv:2502.01046}
}
read the original abstract

How much can we infer about an emotional voice solely from an expressive face? This intriguing question holds great potential for applications such as virtual character dubbing and aiding individuals with expressive language disorders. Existing face-to-speech methods offer great promise in capturing identity characteristics but struggle to generate diverse vocal styles with emotional expression. In this paper, we explore a new task, termed emotional face-to-speech, aiming to synthesize emotional speech directly from expressive facial cues. To that end, we introduce DEmoFace, a novel generative framework that leverages a discrete diffusion transformer (DiT) with curriculum learning, built upon a multi-level neural audio codec. Specifically, we propose multimodal DiT blocks to dynamically align text and speech while tailoring vocal styles based on facial emotion and identity. To enhance training efficiency and generation quality, we further introduce a coarse-to-fine curriculum learning algorithm for multi-level token processing. In addition, we develop an enhanced predictor-free guidance to handle diverse conditioning scenarios, enabling multi-conditional generation and disentangling complex attributes effectively. Extensive experimental results demonstrate that DEmoFace generates more natural and consistent speech compared to baselines, even surpassing speech-driven methods. Demos are shown at https://demoface-ai.github.io/.

Figures

Figures reproduced from arXiv: 2502.01046 by the authors.

Figure 1
Figure 1. Tasks comparison. (a) Conventional Face-to-Speech (F2S). (b) The introduced Emotional Face-to-Speech (eF2S). Given text and face prompts, the model is expected to generate speech that aligns with both the facial identity and emotional expression. Our eF2S offers a novel perspective for generating consistent speech without relying on any vocal cues. ation process utilizing neural audio codec with Residual Vector Quan… view at source ↗
Figure 2
Figure 2. Overall framework of DEmoFace. The MM-DiT inputs masked token x r1:r12 t , time t, and condition set c to synthesize speech, consisting of N blocks for conditioning and 12 linear heads to predict concrete scores. During training, we propose a curriculum learning that first inputs low-level tokens and refines them by adding high-level tokens progressively. During sampling, an Euler sampler with our EPFG refines the t… view at source ↗
Figure 3
Figure 3. Speech qualitative results. The red rectangles highlight key regions with acoustic differences or over-smoothing issues, and the red dotted circle shows similar F0 contours with ground truth. Zoom in for more details. improves naturalness and consistency with fewer data. Notably, we observe that the visual-guided DEmoFace even outperforms the acoustic-guided methods, which are the most efficient for speech generatio… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: t-SNE visualization of x-vectors from synthesis speeches. Each color represents a different speaker. Visualization of speaker embeddings. To explore the speaker diversity, we utilize t-SNE technique (Van der Maaten & Hinton, 2008) to visualize the distribution of x-vec…
Figure 5
Figure 5. Figure 5: Ablation study on curriculum learning. (a) Feature distribution across RVQ levels, with low-level features showing low-frequency patterns. (b)-(d) For the baseline without curriculum learning, we vary the number of training epochs compared with three metrics on the val…
Figure 6
Figure 6. Figure 6: (a) Parameters grid search for the EPFG, with [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model

    eess.AS 2026-07 conditional novelty 5.0 of 10

    A lightweight face adapter plus soft-tuning aligns face embeddings to a frozen StyleTTS 2 style space, yielding natural zero-shot face-to-speech and language-agnostic transfer to Spanish.

Reference graph

Works this paper leans on

76 extracted references · 62 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  3. [3]

    S., and Zisserman, A

    Afouras, T., Chung, J. S., and Zisserman, A. LRS3-TED: a large-scale dataset for visual speech recognition. CoRR, abs/1809.00496, 2018

    arXiv 2018

  4. [4]

    SpeechT5 : Unified -modal encoder-decoder pre-training for spoken language processing

    Ao, J., Wang, R., Zhou, L., Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q., Zhang, Y., Wei, Z., Qian, Y., Li, J., and Wei, F. SpeechT5 : Unified -modal encoder-decoder pre-training for spoken language processing. In Proc. Annu. Meeting Assoc. Comput. Linguistics , pp.\ 5723--5738. Association for Computational Linguistics, 2022

    work page 2022

  5. [5]

    D., Ho, J., Tarlow, D., and van den Berg, R

    Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in discrete state-spaces. In Adv. Neural Inform. Process. Syst. , pp.\ 17981--17993, 2021

    work page 2021

  6. [6]

    W., Fidler, S., and Kreis, K

    Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog. , pp.\ 22563--22575, 2023

    work page 2023

  7. [7]

    Carneiro, H. C. C., Weber, C., and Wermter, S. Whose emotion matters? speaking activity localisation without prior knowledge. Neurocomputing, 545: 0 126271, 2023

    work page 2023

  8. [8]

    and Zisserman, A

    Carreira, J. and Zisserman, A. Quo vadis, action recognition? A new model and the kinetics dataset. In IEEE Conf. Comput. Vis. Pattern Recog. , pp.\ 4724--4733, 2017

    work page 2017

Show all 76 references
  1. [9]

    V2C: Visual voice cloning

    Chen, Q., Tan, M., Qi, Y., Zhou, J., Li, Y., and Wu, Q. V2C: Visual voice cloning. In IEEE Conf. Comput. Vis. Pattern Recog. , pp.\ 21210--21219, 2022

    2022

  2. [10]

    S., Nagrani, A., and Zisserman, A

    Chung, J. S., Nagrani, A., and Zisserman, A. VoxCeleb2 : Deep speaker recognition. In Yegnanarayana, B. (ed.), Annu. Conf. Int. Speech Commun. Assoc. , pp.\ 1086--1090, 2018

    2018

  3. [11]

    Learning to dub movies via hierarchical prosody models

    Cong, G., Li, L., Qi, Y., Zha, Z., Wu, Q., Wang, W., Jiang, B., Yang, M., and Huang, Q. Learning to dub movies via hierarchical prosody models. In IEEE Conf. Comput. Vis. Pattern Recog. , pp.\ 14687--14697, 2023

    2023

  4. [12]

    StyleDubber : Towards multi-scale style learning for movie dubbing

    Cong, G., Qi, Y., Li, L., Beheshti, A., Zhang, Z., van den Hengel, A., Yang, M., Yan, C., and Huang, Q. StyleDubber : Towards multi-scale style learning for movie dubbing. In Findings Proc. Annu. Meeting Assoc. Comput. Linguistics , pp.\ 6767--6779, 2024

    2024

  5. [13]

    High fidelity neural audio compression

    D \' e fossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Trans. Mach. Learn. Res., 2023, 2023

    2023

  6. [14]

    Arcface: Additive angular margin loss for deep face recognition

    Deng, J., Guo, J., Yang, J., Xue, N., Kotsia, I., and Zafeiriou, S. Arcface: Additive angular margin loss for deep face recognition. IEEE Trans. Pattern Anal. Mach. Intell. , 44 0 (10): 0 5962--5979, 2022

    2022

  7. [15]

    and Shutov, V

    Diatlova, D. and Shutov, V. EmoSpeech : Guiding FastSpeech2 towards emotional text to speech. In ISCA Speech Synthesis Worksh. , pp.\ 106--112, 2023

    2023

  8. [16]

    Speaker adaptive text-to-speech with timbre-normalized vector-quantized feature

    Du, C., Guo, Y., Chen, X., and Yu, K. Speaker adaptive text-to-speech with timbre-normalized vector-quantized feature. IEEE ACM Trans. Audio Speech Lang. Process. , 31: 0 3446--3456, 2023

    2023

  9. [17]

    Efficient emotional adaptation for audio-driven talking-head generation

    Gan, Y., Yang, Z., Yue, X., Sun, L., and Yang, Y. Efficient emotional adaptation for audio-driven talking-head generation. In Int. Conf. Comput. Vis. , pp.\ 22577--22588, 2023

    2023

  10. [18]

    Improving adversarial energy-based model via diffusion process

    Geng, C., Han, T., Jiang, P., Zhang, H., Chen, J., Hauberg, S., and Li, B. Improving adversarial energy-based model via diffusion process. In Int. Conf. on Mach. Learn., 2024

    2024

  11. [19]

    Face2Speech : Towards multi-speaker text-to-speech synthesis using an embedding vector predicted from a face image

    Goto, S., Onishi, K., Saito, Y., Tachibana, K., and Mori, K. Face2Speech : Towards multi-speaker text-to-speech synthesis using an embedding vector predicted from a face image. In Annu. Conf. Int. Speech Commun. Assoc. , pp.\ 1321--1325, 2020

    2020

  12. [20]

    EGC: Image generation and classification via a diffusion energy-based model

    Guo, Q., Ma, C., Jiang, Y., Yuan, Z., Yu, Y., and Luo, P. EGC: Image generation and classification via a diffusion energy-based model. In Int. Conf. Comput. Vis., pp.\ 22895--22905, 2023 a

    2023

  13. [21]

    Emodiff : Intensity controllable emotional text-to-speech with soft-label guidance

    Guo, Y., Du, C., Chen, X., and Yu, K. Emodiff : Intensity controllable emotional text-to-speech with soft-label guidance. In IEEE Conf. Acoust. Speech Signal Process. , pp.\ 1--5, 2023 b

    2023

  14. [22]

    An investigation of multi-speaker training for wavenet vocoder

    Hayashi, T., Tamamori, A., Kobayashi, K., Takeda, K., and Toda, T. An investigation of multi-speaker training for wavenet vocoder. In IEEE Autom. Speech Recognit. Understanding Worksh. , pp.\ 712--718, 2017

    2017

  15. [23]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. In Adv. Neural Inform. Process. Syst. Worksh , pp.\ 1--14, 2021

    2021

  16. [24]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Adv. Neural Inform. Process. Syst. , 2020

    2020

  17. [25]

    and Johnson, L

    Ito, K. and Johnson, L. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/, 2017

    2017

  18. [26]

    Jang, Y., Kim, J., Ahn, J., Kwak, D., Yang, H., Ju, Y., Kim, I., Kim, B., and Chung, J. S. Faces that speak: Jointly synthesising talking face and speech from text. In IEEE Conf. Comput. Vis. Pattern Recog. , pp.\ 8818--8828, 2024

    2024

  19. [27]

    Face-stylespeech: Improved face-to-voice latent mapping for natural zero-shot speech synthesis from a face image

    Kang, M., Han, W., and Yang, E. Face-stylespeech: Improved face-to-voice latent mapping for natural zero-shot speech synthesis from a face image. CoRR, abs/2311.05844, 2023

    2023 arXiv

  20. [28]

    Kelly, F. P. Reversibility and stochastic networks. Cambridge University Press, 2011

    2011

  21. [29]

    Speak, read and prompt: High -fidelity text-to-speech with minimal supervision

    Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., and Zeghidour, N. Speak, read and prompt: High -fidelity text-to-speech with minimal supervision. Trans. Assoc. Comput. Linguistics, 11: 0 1703--1718, 2023

    2023

  22. [30]

    and Bengio, Y

    Kim, T. and Bengio, Y. Deep directed generative models with energy-based probability estimation. CoRR, abs/1606.03439, 2016

    2016 arXiv

  23. [31]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Int. Conf. Learn. Represent. , 2015

    2015

  24. [32]

    Kingma, D. P. and Dhariwal, P. Glow : Generative flow with invertible 1x1 convolutions. In Adv. Neural Inform. Process. Syst. , pp.\ 10236--10245, 2018

    2018

  25. [33]

    S., and Chung, S

    Lee, J., Chung, J. S., and Chung, S. Imaginary voice: Face-styled diffusion model for text-to-speech. In IEEE Conf. Acoust. Speech Signal Process. , pp.\ 1--5, 2023

    2023

  26. [34]

    Hear your face: Face -based voice conversion with F0 estimation

    Lee, J., Oh, Y., Hwang, I., and Lee, K. Hear your face: Face -based voice conversion with F0 estimation. CoRR, abs/2408.09802, 2024

    2024 arXiv

  27. [35]

    Li, X., Cheng, Z., He, J., Peng, X., and Hauptmann, A. G. MM-TTS: A unified framework for multimodal, prompt-induced emotional text-to-speech synthesis. CoRR, abs/2404.18398, 2024

    2024 arXiv

  28. [36]

    A., Han, C., Raghavan, V

    Li, Y. A., Han, C., Raghavan, V. S., Mischler, G., and Mesgarani, N. StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. In Adv. Neural Inform. Process. Syst. , 2023

    2023

  29. [37]

    Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable diffusion models. In Eur. Conf. Comput. Vis. , volume 13677, pp.\ 423--439, 2022

    2022

  30. [38]

    Towards a simultaneous and granular identity-expression control in personalized face generation

    Liu, R., Ma, B., Zhang, W., Hu, Z., Fan, C., Lv, T., Ding, Y., and Cheng, X. Towards a simultaneous and granular identity-expression control in personalized face generation. In IEEE Conf. Comput. Vis. Pattern Recog. , pp.\ 2114--2123, 2024

    2024

  31. [39]

    Livingstone, S. R. and Russo, F. A. The ryerson audio-visual database of emotional speech and song ( RAVDESS ): A dynamic, multimodal set of facial and vocal expressions in north american english. PLOS ONE, 13 0 (5): 0 e0196391, 2018

    2018

  32. [40]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In Int. Conf. Learn. Represent. , 2019

    2019

  33. [41]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Lou, A., Meng, C., and Ermon, S. Discrete diffusion modeling by estimating the ratios of the data distribution. In Int. Conf. on Mach. Learn. , 2024

    2024

  34. [42]

    emotion2vec: Self-supervised pre-training for speech emotion representation

    Ma, Z., Zheng, Z., Ye, J., Li, J., Gao, Z., Zhang, S., and Chen, X. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings Proc. Annu. Meeting Assoc. Comput. Linguistics , pp.\ 15747--15760. Association for Computational Linguistics, 2024

    2024

  35. [43]

    POSTER V2: A simpler and stronger facial expression recognition network

    Mao, J., Xu, R., Yin, X., Chang, Y., Nie, B., and Huang, A. POSTER V2: A simpler and stronger facial expression recognition network. CoRR, abs/2301.12149, 2023

    2023 arXiv

  36. [44]

    Mehta, S., Kirkland, A., Lameris, H., Beskow, J., Sz \' e kely, \' E ., and Henter, G. E. Overflow: Putting flows on top of neural transducers for better TTS . In Annu. Conf. Int. Speech Commun. Assoc. , pp.\ 4279--4283, 2023

    2023

  37. [45]

    Concrete score matching: Generalized score matching for discrete data

    Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data. In Adv. Neural Inform. Process. Syst. , 2022

    2022

  38. [46]

    HALL-E: hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis

    Nishimura, Y., Hirose, T., Ohi, M., Nakayama, H., and Inoue, N. HALL-E: hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis. CoRR, abs/2410.04380, 2024

    2024 arXiv

  39. [47]

    Unlocking guidance for discrete state-space diffusion and flow models

    Nisonoff, H., Xiong, J., Allenspach, S., and Listgarten, J. Unlocking guidance for discrete state-space diffusion and flow models. CoRR, abs/2406.01572, 2024

    2024 arXiv

  40. [48]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. CoRR, abs/2406.03736, 2024

    2024 arXiv

  41. [49]

    Visual form predictions facilitate auditory processing at the n1

    Paris, T., Kim, J., and Davis, C. Visual form predictions facilitate auditory processing at the n1. Neuroscience, 343: 0 157--164, 2017

    2017

  42. [50]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Int. Conf. Comput. Vis. , pp.\ 4172--4182, 2023

    2023

  43. [51]

    Hearing faces: Target speaker text-to-speech synthesis from a face

    Pl \" u ster, B., Weber, C., Qu, L., and Wermter, S. Hearing faces: Target speaker text-to-speech synthesis from a face. In IEEE Autom. Speech Recognit. Understanding Worksh. , pp.\ 757--764, 2021

    2021

  44. [52]

    W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I

    Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In Int. Conf. on Mach. Learn. , volume 202, pp.\ 28492--28518, 2023

    2023

  45. [53]

    A., Bengio, Y., and Courville, A

    Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F. A., Bengio, Y., and Courville, A. C. On the spectral bias of neural networks. In Int. Conf. on Mach. Learn. , volume 97, pp.\ 5301--5310, 2019

    2019

  46. [54]

    FastSpeech 2: Fast and high-quality end-to-end text to speech

    Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T. FastSpeech 2: Fast and high-quality end-to-end text to speech. In Int. Conf. Learn. Represent. , 2021

    2021

  47. [55]

    J., Jin, Q., and Guo, B

    Ruan, L., Ma, Y., Yang, H., He, H., Liu, B., Fu, J., Yuan, N. J., Jin, Q., and Guo, B. MM -diffusion: Learning multi-modal diffusion models for joint audio and video generation. In IEEE Conf. Comput. Vis. Pattern Recog. , pp.\ 10219--10228, 2023

    2023

  48. [56]

    Facenet: A unified embedding for face recognition and clustering

    Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In IEEE Conf. Comput. Vis. Pattern Recog. , pp.\ 815--823, 2015

    2015

  49. [57]

    NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers

    Shen, K., Ju, Z., Tan, X., Liu, E., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In Int. Conf. Learn. Represent. , 2024

    2024

  50. [58]

    Denoising diffusion implicit models

    Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In Int. Conf. Learn. Represent. , 2021

    2021

  51. [59]

    Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  52. [60]

    Attention is all you need in speech separation

    Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., and Zhong, J. Attention is all you need in speech separation. In IEEE Conf. Acoust. Speech Signal Process. , pp.\ 21--25, 2021

    2021

  53. [61]

    Score-based continuous-time discrete diffusion models

    Sun, H., Yu, L., Dai, B., Schuurmans, D., and Dai, H. Score-based continuous-time discrete diffusion models. In Int. Conf. Learn. Represent. , 2023

    2023

  54. [62]

    and Fostick, L

    Taitelbaum-Swead, R. and Fostick, L. Auditory and visual information in speech perception: A developmental perspective. Clinical linguistics & phonetics, 30 0 (7): 0 531--545, 2016

    2016

  55. [63]

    and Hinton, G

    Van der Maaten, L. and Hinton, G. Visualizing data using t- SNE . J. Mach. Learn. Res., 9 0 (11), 2008

    2008

  56. [64]

    and Vanathi, P

    Vasuki, A. and Vanathi, P. A review of vector quantization techniques. IEEE Potentials , 25 0 (4): 0 39--47, 2006

    2006

  57. [65]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Adv. Neural Inform. Process. Syst. , pp.\ 5998--6008, 2017

    2017

  58. [66]

    Generalized end-to-end loss for speaker verification

    Wan, L., Wang, Q., Papir, A., and L \' o pez - Moreno, I. Generalized end-to-end loss for speaker verification. In IEEE Conf. Acoust. Speech Signal Process. , pp.\ 4879--4883, 2018

    2018

  59. [67]

    Neural codec language models are zero-shot text to speech synthesizers

    Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L., Zhao, S., and Wei, F. Neural codec language models are zero-shot text to speech synthesizers. CoRR, abs/2301.02111, 2023

    2023 arXiv

  60. [68]

    Wang, K., Wu, Q., Song, L., Yang, Z., Wu, W., Qian, C., He, R., Qiao, Y., and Loy, C. C. MEAD: A large-scale audio-visual dataset for emotional talking-face generation. In Eur. Conf. Comput. Vis. , volume 12366 of Lecture Notes in Computer Science, pp.\ 700--717, 2020

    2020

  61. [69]

    J., Battenberg, E., Shor, J., Xiao, Y., Jia, Y., Ren, F., and Saurous, R

    Wang, Y., Stanton, D., Zhang, Y., Skerry - Ryan, R. J., Battenberg, E., Shor, J., Xiao, Y., Jia, Y., Ren, F., and Saurous, R. A. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In Int. Conf. on Mach. Learn. , volume 80, pp.\ 5167...

    2018

  62. [70]

    MaskGCT : Zero-shot text-to-speech with masked generative codec transformer

    Wang, Y., Zhan, H., Liu, L., Zeng, R., Guo, H., Zheng, J., Zhang, Q., Zhang, S., and Wu, Z. MaskGCT : Zero-shot text-to-speech with masked generative codec transformer. CoRR, abs/2409.00750, 2024

    2024 arXiv

  63. [71]

    DCTTS: discrete diffusion model with contrastive learning for text-to-speech generation

    Wu, Z., Li, Q., Liu, S., and Yang, Q. DCTTS: discrete diffusion model with contrastive learning for text-to-speech generation. In IEEE Conf. Acoust. Speech Signal Process. , pp.\ 11336--11340. IEEE , 2024

    2024

  64. [72]

    Foundationtts: Text-to-speech for ASR customization with generative language model

    Xue, R., Liu, Y., He, L., Tan, X., Liu, L., Lin, E., and Zhao, S. Foundationtts: Text-to-speech for ASR customization with generative language model. CoRR, abs/2303.02939, 2023

    2023 arXiv

  65. [73]

    Diffsound: Discrete diffusion model for text-to-sound generation

    Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE ACM Trans. Audio Speech Lang. Process. , 31: 0 1720--1733, 2023

    2023

  66. [74]

    SoundStream : An end-to-end neural audio codec

    Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. SoundStream : An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process. , 30: 0 495--507, 2022

    2022

  67. [75]

    SpeechTokenizer : Unified speech tokenizer for speech language models

    Zhang, X., Zhang, D., Li, S., Zhou, Y., and Qiu, X. SpeechTokenizer : Unified speech tokenizer for speech language models. In Int. Conf. Pattern Recog. , 2024

    2024

  68. [76]

    Srcodec: Split -residual vector quantization for neural speech codec

    Zheng, Y., Tu, W., Xiao, L., and Xu, X. Srcodec: Split -residual vector quantization for neural speech codec. In IEEE Conf. Acoust. Speech Signal Process. , pp.\ 451--455, 2024

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.