Pith. sign in

REVIEW 4 major objections 6 minor 47 references

FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FaceSpeak can synthesize a voice from any-style portrait by decoupling identity and emotion, and its EM2TTS dataset makes such training possible.

desk verdict Promising architecture and a useful first step toward style-agnostic portrait-driven TTS, but the dataset's circularity and confounded decoupling test mean the headline claims are not yet established. read the letter →

arxiv 2501.03181 v2 pith:IXRYPLR2 submitted 2025-01-02 cs.SD cs.AIeess.AS

Tian-Hao Zhang , Jiawei Zhang , Jun Wang , Xinyuan Qian , Xu-Cheng Yin This is my paper · ORCID
classification cs.SDcs.AIeess.AS
keywords face-driventext-to-speechmulti-styleportraitsidentity-emotiondisentanglementexpressivespeechsynthesismulti-modaldatasetmutualinformationminimizationVITS2visualpromptTTS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FaceSpeak claims that a single portrait image—real, cartoon, fantasy, line art, or any other style—carries enough identity and emotional information to synthesize a voice that listeners match back to the face. To make that work, the paper introduces a pipeline that splits the portrait into two separate control codes: one for who the speaker is and one for how the speaker feels. It also builds a new dataset, EM2TTS, by transferring real face videos into multiple styles and by generating portraits from speech-derived descriptions, because existing TTS data lack style-diverse image–speech pairs. If the claim holds, speech synthesis for games, virtual assistants, and animation no longer needs a real face or a recorded voice sample as the prompt.

What carries the argument

The load-bearing mechanism is the decoupling of visual features into identity embedding $\alpha_i$ and emotion embedding $\beta_i$. A face-oriented visual encoder maps any-style portrait to $e_i \in \mathbb{R}^{512}$; two lightweight adapters project that vector, a gradient-reversal classifier strips emotion from $\alpha_i$, and minimization of the vCLUB mutual-information upper bound on $I(\alpha_i;\beta_i)$ keeps the two codes from duplicating each other. The two embeddings are then injected into the VITS2 generator's posterior encoder, flow, decoder, and duration predictor, making the synthesized voice follow whichever image supplied each code.

What would settle it

Take portraits whose style transfer has changed apparent gender, age, or emotional expression (for example, a line-art rendering that makes an adult appear childlike), have human raters label the perceived persona of each portrait, and compare those labels with the model's voice output: if the synthesized voice tracks the original dataset labels rather than the perceived persona whenever they conflict, the alignment claim fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that identity and emotion can be pulled apart from a portrait's visual features and used independently to condition a VITS2-based speech synthesizer. Identity embedding $\alpha_i$ and emotion embedding $\beta_i$ are extracted by two adapters from face-focused visual features; a gradient-reversal layer stops the identity embedding from carrying emotion information, and a vCLUB mutual-information upper bound pushes the two embeddings apart. During inference the two codes can come from different images, so one portrait can set the speaker's voice and another can set the emotional tone. Evaluations claim 98.6% identity matching and 92.1% emotion matching when identity and emotion are supplied by separate portraits, with naturalness scores close to ground truth on in-domain data.

Load-bearing premise

The entire training signal rests on the assumption that a portrait produced by changing an image's style, or by drawing a new image from a text description, still shows the same identity and the same emotion as the original photo or label; if the portrait's perceived persona shifts during that process, the model learns to tie voices to image artifacts rather than to a character.

Editorial extensions

If this is right

  • Users can independently swap voice identity and emotional tone by choosing two different portraits as prompts, so a single framework covers character voice design without retraining.
  • Non-photorealistic portraits—cartoon, fantasy art, line art, neonpunk, cinematic—become valid TTS prompts, opening synthesis to games, animation, and virtual worlds.
  • The EM2TTS dataset provides style-diverse image–speech pairs that let a TTS model trained on it generalize to out-of-domain virtual portraits.
  • Because the model learns from image labels rather than a voice sample, a silent image can act as a complete speaker prompt, removing the need for reference recordings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the decoupling generalizes beyond the tested styles, the same two-code scheme could be applied to other face-related attributes, such as age or accent, letting a portrait control multiple vocal dimensions at once.
  • A natural next test is to give the system a portrait whose style transfer changed the apparent gender or age of the face; the current evaluation does not separate those changes from identity, so how the model responds would clarify what 'identity' means to it.
  • The dataset construction suggests a recipe others could reuse: combine a face-aware feature extractor with LLM-expanded captions and text-to-image generation, then treat the resulting pairs as weak labels; if the image generator introduces systematic biases, those biases will be learned as voice traits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes FaceSpeak, a multi-modal TTS system that synthesizes speech from portrait images of arbitrary styles (real, fantasy art, cinematic, neonpunk, line art). The method extracts a FaRL image embedding, then uses Identity and Expression Adapter Modules, an emotion classifier with gradient reversal, and vCLUB-based mutual-information minimization to decouple identity and emotion embeddings. These embeddings are injected into a VITS2 backbone. To address the lack of style-diverse portrait-speech data, the authors introduce EM2TTS, with a MEAD subset created by PhotoMaker style transfer and an ESD-EmovDB subset created by human annotation, ChatGPT text expansion, and DALL-E-3 image generation. Experiments compare FaceSpeak with VITS2, MM-StyleSpeech, and MM-TTS on intra- and out-of-domain data using MOS, AXY preference, MCD, emotion/gender classification, and speaker similarity, reporting consistent subjective wins and claiming 98.6% identity and 92.1% emotion matching in a combined-portrait control test.

Significance. If the central claims hold, FaceSpeak would extend portrait-conditioned TTS beyond real photographs to stylized and virtual characters, and would demonstrate independent identity/emotion control from separate images—both practically useful and technically interesting. The paper's strengths include a clear disentanglement architecture (IAM/EAM with gradient reversal and MI minimization), a novel multi-style dataset construction pipeline, and demo samples that support the naturalness advantage over the compared baselines. However, the empirical validation of the two load-bearing claims—portrait-aligned voice and decoupled identity/emotion control—rests on dataset pairings and evaluation protocols that currently admit much simpler explanations, as detailed below. The work is therefore promising but not yet established.

major comments (4)
  1. [EM2TTS-ESD-EmovDB (Dataset)] The image-speech pairing in the ESD-EmovDB subset is circular for identity and emotion learning: a human expert labels age, gender, and characteristics by listening to each utterance, ChatGPT expands those labels, and DALL-E-3 renders a face that is then paired with the original utterance. The generated face cannot contain the speaker's timbre; it instantiates coarse category labels derived from the same audio that later serves as emotion ground truth (Table 3). Consequently, training IAM/EAM on these pairs can learn label-to-voice associations rather than face-voice correspondences, and the out-of-domain emotion results do not establish portrait-aligned emotion control. Please either exclude this subset from identity/emotion claims and treat it as style/label-conditioned data, or provide a human-verification study showing that the generated faces preserve identity/emotion independently of the audio-derived labels.
  2. [Results of Decoupled Identity and Emotion Information] The combined-portrait test is confounded by construction: X and Y are required to differ in gender and emotion, so a listener can match identity by selecting the gender of X and emotion by selecting the emotion class of Y without any fine-grained disentanglement. The reported 98.6% identity and 92.1% emotion accuracies are therefore not evidence of decoupled identity/expression control. Please rerun the test with same-gender X/Y pairs, with multiple identities per emotion and multiple emotions per identity, and with distractor images matched on gender and emotion.
  3. [Table 3] MCD and speaker similarity are reported only for FaceSpeak (MCD 3.32, SS 0.95), with no corresponding values for VITS2, MM-StyleSpeech, MM-TTS, or a reference-speech upper bound. Without baseline values, these numbers cannot support the claim that FaceSpeak is superior in spectral or speaker similarity. Please add baseline MCD/SS values and describe the reference used for MCD, since synthesized TTS samples are not time-aligned with the reference waveform.
  4. [Table 3, out-of-domain Acc_emo] The out-of-domain emotion accuracy is 31.32%, far below the intra-domain 60.92%, and no chance-level or per-emotion breakdown is given. This is consistent with the model learning coarse, dataset-specific cues from the synthetic EM2TTS-ESD-EmovDB pairs rather than transferable face-emotion associations. Please report chance accuracy, per-emotion confusion, and the same emotion classifier applied to the baseline models.
minor comments (6)
  1. [General] No code, trained models, or dataset release link is mentioned; given that the paper's main contribution includes a new dataset and a new model, availability details are needed for reproducibility.
  2. [Table 3 heading] Table 3 is labeled 'Subjective results' but contains only objective metrics (MCD, classification accuracy, speaker similarity); the subjective results are in Table 1. Please correct the label.
  3. [Table 4 heading] The heading contains the typo 'Intro-domain' instead of 'Intra-domain'.
  4. [Proposed Method / Figure 3] Figure 3 includes components labeled 'SAM', 'Style Conversion', and 'Style Speech Expert' that are not described in the text; please clarify their roles or remove them from the diagram.
  5. [Eq. (6) and Experimental Setup] The loss weights λ1, λ2, and λ3 are introduced as hyper-parameters, but their values are deferred to an appendix that is not present in this version; including them and the network configuration is necessary for reproducibility.
  6. [Related Work] The names 'MMTTS' and 'MM-TTS' are used inconsistently; please unify the notation across the paper.

Circularity Check

2 steps flagged · score 6.0 of 10

EM2TTS-ESD-EmovDB portraits are DALL-E renderings of labels inferred from the same audio used as supervision, so the emotion-alignment and the gender/emotion-confounded decoupling test reduce to label-conditioned generation rather than genuine face-voice correspondence.

  1. self definitional [Proposed EM2TTS Dataset, subsection EM2TTS-ESD-EmovDB (Figure 2 caption and dataset-construction paragraph)]
    "1) Manual annotation: we explore a human expert to label age, gender, and characteristics by listening to each speech data; 2) Text expansion: we use the Large Language Model (LLM) model e.g., ChatGPT to expand the label words into texts with varying contents but similar meanings; 3)Text-driven image generation: the enriched texts were then fed into DALL-E-3, a text-to-image model capable of generating a multitude of images in distinct styles."

    The visual prompt is constructed from labels obtained by listening to the very utterance that later serves as the synthesis target and as the ground truth for emotion/speaker evaluation. DALL-E-3 renders an image of a fictional person whose age/gender/characteristics are taken from that audio, so the image cannot supply independent identity information about the real speaker. Training FaceSpeak’s IAM/EAM on these pairs (with Lemo and Lgrl in Eq. 3 using the image-derived emotion label) teaches the model to recover the coarse demographic/emotion labels that generated the image, not to associate a real face with a real voice. For this subset, 'portrait-aligned voice' reduces by construction to label-conditioned resynthesis of the same audio.

  2. other [Experiments, 'Results of Decoupled Identity and Emotion Information', 'Speech synthesis controlled by combined portraits' (Figure 6)]
    "we define X as the image that provides identity embedding and Y as the image that provides emotion embedding, ensuring that X and Y have different genders and emotions. We let listeners discriminate the synthesized speech by deciding whether it is matched to the X-image or Y-image in terms of identity and emotion, respectively."

    The protocol explicitly forces X and Y to differ in gender and emotion. Because the EM2TTS-ESD-EmovDB images used in training encode exactly those categories (they were generated from gender/emotion/characteristic labels extracted from the target audio), a listener can identify the 'correct' image by matching coarse gender or emotion class. The 98.6% identity-match and 92.1% emotion-match numbers are therefore explainable as label-class selection and do not demonstrate fine-grained identity or expression disentanglement; the test cannot separate the claimed decoupling from the category cues that the dataset construction made trivially available.

full rationale

The paper has two data paths with very different epistemic status. The EM2TTS-MEAD path is independent: PhotoMaker transfers genuine MEAD faces of known speakers and MEAD provides the emotion labels, so identity/emotion supervision there is not circular. However, the EM2TTS-ESD-EmovDB path is load-bearing for the paper's claims about multi-style portraits and about independent identity/emotion control, and it is circular for the evaluated attributes: the generated image is a function of labels extracted from the same audio that is later used as training target and evaluation ground truth. The combined-portrait test in Sec. 4.4 is additionally confounded because X and Y are required to differ in gender and emotion, so the reported 98.6% and 92.1% accuracies can be achieved by selecting gender/emotion categories rather than by true identity/expression decoupling. Table 3's out-of-domain Acc_emo collapse (31.32% vs. 60.92% intra-domain) is consistent with the learned emotion signal being tied to the synthetic label-conditioned distribution. No self-citation chain or author-imported uniqueness argument was found; citations to FaRL, PhotoMaker, VITS2, and vCLUB are external and independently checkable. The circularity is partial, since the MEAD real-face portion and the real-portrait out-of-domain evaluation retain independent content, but the synthetic subset and the decoupling evaluation reduce the central multi-style/decoupling claims to label-conditioned generation by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities such as particles or fields are introduced. EM2TTS is a dataset, not an entity, though its release and license are unstated. The free parameters are limited to the loss balancing weights, and the central assumptions concern the validity of generated image-speech pairings and the FaRL feature extractor.

free parameters (1)
  • Loss weights lambda1, lambda2, lambda3
    Hyperparameters balancing Lvits, Lmi, Lemo and Lgrl in Eq. 6. No values are reported in the paper, and the results depend on these choices.
assumptions (4)
  • domain assumption FaRL, pretrained on real face-image/text data, produces face-related features that remain valid for non-photorealistic styles.
    Section 'Multi-Style Image Feature Disentanglement' applies FaRL to both real and stylized images as the only visual feature source. If style transfer breaks face representation, identity and emotion extraction fails.
  • domain assumption PhotoMaker style transfer preserves identity and expression of the source frame, so EM2TTS-MEAD images inherit the emotion labels.
    The Dataset section describes using PhotoMaker to generate four styles of images per speaker; emotion labels are inherited from MEAD frames without re-verification.
  • domain assumption DALL-E-3 images generated from ChatGPT-expanded labels contain the age, gender, emotion and character factors claimed, so the speech labels match the generated images.
    EM2TTS-ESD-EmovDB construction assigns generated images to speech based on text labels. If DALL-E ignores or misrenders label factors, the image-speech pairing is corrupted.
  • standard math The variational q_theta(beta|alpha) provides a valid upper bound or reliable estimator of mutual information at each training step.
    vCLUB is cited as an upper bound when q_theta is chosen to maximize log-likelihood. The paper does not verify this condition empirically in the training loop.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles." pith.science (2026). https://pith.science/paper/IXRYPLR2

@misc{pith2026250103181,
  author       = {Pith},
  title        = {Pith review of: FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IXRYPLR2}},
  note         = {Machine review of arXiv:2501.03181}
}
read the original abstract

Humans can perceive speakers' characteristics (e.g., identity, gender, personality and emotion) by their appearance, which are generally aligned to their voice style. Recently, vision-driven Text-to-speech (TTS) scholars grounded their investigations on real-person faces, thereby restricting effective speech synthesis from applying to vast potential usage scenarios with diverse characters and image styles. To solve this issue, we introduce a novel FaceSpeak approach. It extracts salient identity characteristics and emotional representations from a wide variety of image styles. Meanwhile, it mitigates the extraneous information (e.g., background, clothing, and hair color, etc.), resulting in synthesized speech closely aligned with a character's persona. Furthermore, to overcome the scarcity of multi-modal TTS data, we have devised an innovative dataset, namely Expressive Multi-Modal TTS, which is diligently curated and annotated to facilitate research in this domain. The experimental results demonstrate our proposed FaceSpeak can generate portrait-aligned voice with satisfactory naturalness and quality.

Figures

Figures reproduced from arXiv: 2501.03181 by the authors.

Figure 1
Figure 1. Our proposed multi-modal speech synthesis frame [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our image generation pipeline of 1) EM2TTS-MEAD subset (top): we specify the desired output style and transfer the real human image to images of different styles using PhotoMaker. 2) EM2TTS-ESD-EmovDB subset (bottom): we use a human expert to label the character factors for chatGPT to create the descriptive text, which is utilized by DALL-E-3 to produce images that are highly aligned with the specified parameters. d… view at source ↗
Figure 3
Figure 3. Block diagram of our proposed FaceSpeak which generates speech given the input text [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visualization of emotion embeddings (colors index [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Visualization of identity embeddings (colors index identities; M: male; F: female). [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The accuracy in matching the synthesized speech [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 38 canonical work pages

  1. [1]

    Busso, C.; Bulut, M.; Lee, C.-C.; Kazemzadeh, A.; Mower, E.; Kim, S.; et al. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42: 335--359

    work page 2008

  2. [2]

    D.; Junior, A

    Casanova, E.; Weber, J.; Shulby, C. D.; Junior, A. C.; et al. 2022. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In Proc. of Int. Conf. on Machine Learning, 2709--2720. PMLR

    work page 2022

  3. [3]

    Cheng, P.; Hao, W.; Dai, S.; Liu, J.; Gan, Z.; et al. 2020. CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event

    work page 2020

  4. [4]

    Deaton, A. 2010. Understanding the mechanisms of economic development. Journal of Economic Perspectives, 24(3): 3--16

    work page 2010

  5. [5]

    Gao, R.; and Grauman, K. 2021. Visualvoice: Audio-visual speech separation with cross-modal consistency. In Proc. of Int. Conf. on Computer Vision and Pattern Recognition, 15490--15500. IEEE

    work page 2021

  6. [6]

    Grzybowska, J.; and Kacprzak, S. 2016. Speaker age classification and regression using i-vectors. In Proc. of Interspeech, 1402--1406

    work page 2016

  7. [7]

    Guan, W.; Li, Y.; Li, T.; Huang, H.; Wang, F.; Lin, J.; Huang, L.; Li, L.; and Hong, Q. 2024. MM-TTS: Multi-modal Prompt based Style Transfer for Expressive Text-to-Speech Synthesis

    work page 2024

  8. [8]

    Guo, Z.; Leng, Y.; Wu, Y.; Zhao, S.; and Tan, X. 2023. PromptTTS: Controllable text-to-speech with text descriptions. In Proc. of IEEE Int. Conf. on Audio, Speech and Signal Processing , 1--5. IEEE

    work page 2023

Show all 47 references
  1. [9]

    J.; Laver, J.; and Gibbon, F

    Hardcastle, W. J.; Laver, J.; and Gibbon, F. E. 2012. The handbook of phonetic sciences

    2012

  2. [10]

    Huang, R.; Ren, Y.; Liu, J.; Cui, C.; and Zhao, Z. 2022. Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech. Advances in Neural Information Processing Systems, 35: 10970--10983

    2022

  3. [11]

    Ji, S.; Zuo, J.; Fang, M.; Jiang, Z.; Chen, F.; et al. 2024. Textrolspeech: A text style control speech corpus with codec language text-to-speech models. In Proc. of IEEE Int. Conf. on Audio, Speech and Signal Processing , 10301--10305. IEEE

    2024

  4. [12]

    Jiang, Z.; Ren, Y.; Ye, Z.; Liu, J.; Zhang, C.; Yang, Q.; Ji, S.; et al. 2023. Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias. arXiv preprint arXiv:2306.03509

    2023 arXiv

  5. [13]

    Kong, J.; Park, J.; Kim, B.; Kim, J.; et al. 2023. VITS2 : Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design. arXiv preprint arXiv:2307.16430

    2023 arXiv

  6. [14]

    Kubichek, R. 1993. Mel-cepstral distance measure for objective speech quality assessment. In Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, 1: 125--128

    1993

  7. [15]

    S.; et al

    Lee, J.; Chung, J. S.; et al. 2023. Imaginary voice: Face-styled diffusion model for text-to-speech. In Proc. of IEEE Int. Conf. on Audio, Speech and Signal Processing , 1--5. IEEE

    2023

  8. [16]

    Lee, J.; Chung, S.-W.; Kim, S.; Kang, H.-G.; and Sohn, K. 2021. Looking into your speech: Learning cross-modal affinity for audio-visual speech separation. In Proc. of Int. Conf. on Computer Vision and Pattern Recognition, 1336--1345

    2021

  9. [17]

    Lei, Y.; and Cao, H. 2023. Audio-visual emotion recognition with preference learning based on intended and multi-modal perceived labels. IEEE Transactions on Affective Computing

    2023

  10. [18]

    Li, S.; Dabre, R.; Lu, X.; Shen, P.; Kawahara, T.; and Kawai, H. 2019. Improving transformer-based speech recognition systems with compressed structure and speech attributes augmentation. In Proc. of Interspeech, 4400--4404

    2019

  11. [19]

    Li, X.; Wen, Y.; Yang, M.; Wang, J.; Singh, R.; and Raj, B. 2023 a . Rethinking Voice-Face Correlation: A Geometry View. In Proc. of ACM Int. Conf. on Multimedia , 2458--2467

    2023

  12. [20]

    Li, Z.; Cao, M.; Wang, X.; Qi, Z.; et al. 2023 b . Photomaker: Customizing realistic human photos via stacked id embedding. arXiv preprint arXiv:2312.04461

    2023 arXiv

  13. [21]

    R.; and Russo, F

    Livingstone, S. R.; and Russo, F. A. 2018. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PloS one, 13(5): e0196391

    2018

  14. [22]

    Lu, J.; Sisman, B.; Liu, R.; Zhang, M.; and Li, H. 2022. VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over. In Proc. of IEEE Int. Conf. on Audio, Speech and Signal Processing . IEEE

    2022

  15. [23]

    B.; Yang, E.; and Hwang, S

    Min, D.; Lee, D. B.; Yang, E.; and Hwang, S. J. 2021. Meta-stylespeech: Multi-speaker adaptive text-to-speech generation. In Proc. of Int. Conf. on Machine Learning, 7748--7759. PMLR

    2021

  16. [24]

    S.; Morerio, P.; Mahmood, A.; et al

    Nawaz, S.; Saeed, M. S.; Morerio, P.; Mahmood, A.; et al. 2021. Cross-modal speaker verification and recognition: A multilingual perspective. In Proc. of Int. Conf. on Computer Vision and Pattern Recognition, 1682--1691

    2021

  17. [25]

    A.; Hsu, W.-N.; d'Avirro, A.; Shi, B.; Gat, I.; Fazel-Zarani, M.; et al

    Nguyen, T. A.; Hsu, W.-N.; d'Avirro, A.; Shi, B.; Gat, I.; Fazel-Zarani, M.; et al. 2023. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv preprint arXiv:2308.05725

    2023 arXiv

  18. [26]

    Qian, Y.; Chen, Z.; and Wang, S. 2021. Audio-visual deep neural network for robust person verification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 1079--1092

    2021

  19. [27]

    Ravanelli, M.; and Bengio, Y. 2018. Speaker recognition from raw waveform with sincnet. In 2018 IEEE Spoken Language Technology Workshop (SLT), 1021--1028. IEEE

    2018

  20. [28]

    Ren, Y.; Hu, C.; Tan, X.; Qin, T.; Zhao, S.; Zhao, Z.; and Liu, T.-Y. 2020. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558

    2020 arXiv

  21. [29]

    Singh, R.; Keshet, J.; Gencaga, D.; and Raj, B. 2016. The relationship of voice onset time and voice offset time to physical age. In Proc. of IEEE Int. Conf. on Audio, Speech and Signal Processing , 5390--5394. IEEE

    2016

  22. [30]

    Sisman, B.; and Yamagishi, J. 2021. An overview of voice conversion and its challenges: From statistical modeling to deep learning. In IEEE/ACM Trans. on Audio, Speech and Language Processing, 132--157

    2021

  23. [31]

    Skerry-Ryan, R.; and Battenberg, E. 2018. Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron. Proceedings of the 35th International Conference on Machine Learning(ICML), 80: 3--16

    2018

  24. [32]

    G.; Alrashoud, M.; et al

    Verde, L.; Giuseppe De Pietro, A. G.; Alrashoud, M.; et al. 2021. Exploring the Use of Artificial Intelligence Techniques to Detect the Presence of Coronavirus Covid-19 Through Speech and Voice Analysis. IEEE Access

    2021

  25. [33]

    Wang, K.; Wu, Q.; Song, L.; Yang, Z.; Wu, W.; Qian, C.; et al. 2020. Mead: A large-scale audio-visual dataset for emotional talking-face generation. In Proc. of European Conf. on Computer Vision, 700--717. Springer

    2020

  26. [34]

    Wang, Z.-Q.; and Tashev, I. 2017. Learning utterance-level representations for speech emotion and age/gender recognition using deep neural networks. In Proc. of IEEE Int. Conf. on Audio, Speech and Signal Processing , 5150--5154. IEEE

    2017

  27. [35]

    Xu, C.; Liu, Y.; Xing, J.; Wang, W.; Sun, M.; Dan, J.; et al. 2024. FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio. arXiv preprint arXiv:2403.01901

    2024 arXiv

  28. [36]

    Yamagishi, J.; Veaux, C.; MacDonald, K.; et al. 2019. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92)

    2019

  29. [37]

    Yang, D.; Liu, S.; Huang, R.; Weng, C.; and Meng, H. 2023. InstructTTS: Modelling expressive TTS in discrete latent space with natural language style prompt. arXiv preprint arXiv:2301.13662

    2023 arXiv

  30. [38]

    B.; Liang, P

    Zadeh, A. B.; Liang, P. P.; Poria, S.; Cambria; et al. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2018

  31. [39]

    J.; Jia, Y.; et al

    Zen, H.; Dang, V.; Clark, R.; Zhang, Y.; Weiss, R. J.; Jia, Y.; et al. 2019. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882

    2019 arXiv

  32. [40]

    Zhang, S.; Ding, Y.; Wei, Z.; and Guan, C. 2021. Continuous emotion recognition with audio-visual leader-follower attentive fusion. In Proc. of Int. Conf. on Computer Vision, 3567--3574

    2021

  33. [41]

    Zhang, Y.; Liu, G.; Lei, Y.; Chen, Y.; Yin, H.; Xie, L.; and Li, Z. 2023. Promptspeaker: Speaker Generation Based on Text Descriptions. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 1--7. IEEE

    2023

  34. [42]

    Zhang, Z.; Wu, B.; and Schuller, B. 2019. Attention-augmented end-to-end multi-task learning for emotion prediction from speech. In Proc. of IEEE Int. Conf. on Audio, Speech and Signal Processing , 6705--6709

    2019

  35. [43]

    Zheng, Y.; Yang, H.; Zhang, T.; Bao, J.; Chen, D.; et al. 2022. General facial representation learning in a visual-linguistic manner. In Proc. of Int. Conf. on Computer Vision and Pattern Recognition, 18697--18709

    2022

  36. [44]

    Zhou, H.; Du, J.; Zhang, Y.; Wang, Q.; Liu, Q.-F.; and Lee, C.-H. 2021. Information fusion in attention networks using adaptive and multi-level factorized bilinear pooling for audio-visual emotion recognition. IEEE/ACM Transactions on audio, speech, and language processing, 29...

    2021

  37. [45]

    Zhou, K.; Sisman, B.; Liu, R.; and Li, H. 2022. Emotional voice conversion: Theory, databases and ESD. Speech Communication, 137: 1--18

    2022

  38. [46]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.