Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DICE-Talk claims to generate emotionally expressive talking-head videos that preserve speaker identity by modeling emotion as an identity-agnostic Gaussian distribution and conditioning a diffusion model with a learned emotion bank.

desk verdict A capable emotional talking-head system whose central 'identity-agnostic' claim is contradicted by its own contrastive objective; worth a serious referee, but heavy revision is needed. read the letter →

arxiv 2504.18087 v2 pith:KDSXRPZV submitted 2025-04-25 cs.CV

Weipeng Tan , Chuming Lin , Chengming Xu , FeiFan Xu , Xiaobin Hu , Xiaozhong Ji , Junwei Zhu , Chengjie Wang
show 1 more author
Yanwei Fu
This is my paper · ORCID
classification cs.CV
keywords talkingheadgenerationemotioncontroldiffusionmodelidentitydisentanglementbankaudio-visualcontrastivelearningvectorquantization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Emotional talking-head generation currently faces three problems: audio's emotional cues are underused, emotion representations leak speaker identity, and emotions are learned in isolation. This paper claims that all three can be addressed by representing emotion as a distribution rather than a fixed code, and by letting emotion conditions interact with a learned bank of emotion prototypes inside a diffusion model. If the claim holds, users could drive a portrait's emotion from an example video or from a fixed emotion prompt, including for identities never seen in training, while keeping the speaker recognizable. The experiments on MEAD and HDTF report higher emotion accuracy than prior emotion-aware methods, with lip-sync and visual quality at competitive levels.

What carries the argument

The central object is the disentangled emotion embedder: a transformer that fuses visual features and Whisper audio features by cross-attention, then aggregates them into a mean $\mu_s$ and variance $\sigma_s^2$ of a Gaussian emotion prior, from which $s$ is sampled. This is the mechanism supposed to make emotion identity-agnostic, since sampling a different $s$ each time stops the generator from latching onto reference-specific facial details. Around it sit two supporting mechanisms: the correlation-enhanced emotion conditioning module, where a learnable emotion bank $C$ stores shared emotion prototypes via vector quantization and cross-attention lets the prior attend to several relevant prototypes at once, such as anger and disgust together for an aggressive expression, and the emotion discrimination objective, an MLP classifier on pooled U-Net latents that adds a cross-entropy loss to the denoising objective. At inference the bank also permits skipping retrieval and attending over the whole bank instead.

What would settle it

Run the learned embedder on held-out clips and measure the inter/intra-cluster ratio with emotion fixed and identity varied: if same-identity clips cluster more tightly than same-emotion clips across identities, the prior is carrying identity and the disentanglement claim is falsified; a simpler observation would be checking whether an audio-only emotion prior still keeps Emo-Score near the reported out-of-domain level of about 0.54 while F-SIM remains near 0.87.

Watch

Extended reading notes

Core claim

On the paper's own terms, DICE-Talk is an audio-driven, diffusion-based talking-head generator whose contribution is to separate who is speaking from how they are speaking. It encodes an emotion reference video, frames plus corresponding audio, into a Gaussian emotion prior $s = \mu_s + \sigma_s\epsilon$, so that each clip maps to a distribution rather than a single vector; the randomness is meant to absorb identity-specific expression habits and prevent them from being copied onto the target face. That prior is refined by an emotion bank $C\in\mathbb{R}^{K \times d_s}$ trained with a vector-quantization commitment loss, and the refined feature is injected into the Stable Video Diffusion (SVD) U-Net through cross-attention. An emotion discriminator classifies noisy intermediate latents, adding an affect-accuracy gradient to the denoising loss. The paper reports that this yields Emo-Scores of 0.5424 for a video-prompt variant and 0.5527 for a fixed-prompt variant on an out-of-domain test set, above the emotion-aware methods compared, and a facial-similarity F-SIM of 0.8704 in its ablation, which it reads as evidence that identity is retained while emotions change.

Load-bearing premise

The load-bearing premise is that the contrastive training signal, where positive pairs are clips with the same identity and same emotion and negatives are clips with different identities or different emotions, teaches the embedder to discard identity rather than encode it; if that premise is wrong, the disentanglement the method is named for is not achieved.

Editorial extensions

If this is right

  • Emotion control becomes a continuous interface: a user can supply either an emotional reference clip or a fixed emotion prompt, and linear interpolation between two emotion embeddings produces smooth transitions between expressions in the generated video.
  • Because the emotion bank stores shared prototypes, the generator can combine elements of related emotions, producing mixed or graded expressions rather than only discrete categorical labels.
  • The latent-space emotion discriminator adds affective pressure inside the denoising process, so emotion accuracy does not have to be traded against lip synchronization or frame quality.
  • The reported clustering of emotion priors by emotion and intensity indicates the embedding space is organized along affective dimensions, which could support emotion editing and intensity control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The contrastive pairs are defined so that a clip of a different person with the same emotion counts as negative; under that signal the embedding is pushed to separate identities as much as emotions, so the identity-agnostic claim would be stronger if pairs were restricted to same-emotion, different-identity comparisons.
  • The out-of-domain evaluation uses ten reference images, so the headline generalization numbers are preliminary; scaling the same pipeline to a larger identity and emotion set would be a natural stress test.
  • Gaussian sampling may be acting as a regularizer that suppresses copying of reference-video facial details, which suggests the same distributional-prior trick could transfer to other attribute-transfer settings, such as gaze direction or accent.
  • The emotion bank, being a discrete learnable codebook, offers a small inspectable space for emotion prompts; one could manually edit prototypes or probe which combinations produce perceptually distinct expressions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DICE-Talk, a diffusion-based audio-driven talking head generation framework that aims to disentangle speaker identity from emotional expression and to model correlations among emotions. The method has three components: a disentangled emotion embedder that fuses audio and visual features via cross-attention and represents the emotion prior as a Gaussian distribution; a correlation-enhanced emotion conditioning module with a learnable emotion bank using vector quantization and attention; and an emotion discrimination objective that classifies affect from intermediate diffusion latents. Experiments on MEAD, HDTF, and a small out-of-domain set compare against ten baselines, including emotion-aware methods EAMM, StyleTalk, and EAT, reporting FVD, FID, SyncNet, Emo-Score, F-SIM, and a user study.

Significance. If the identity-agnostic emotion representation were convincingly established, DICE-Talk would be a practical contribution to controllable portrait animation, with a sensible three-part architecture and unusually broad comparisons, including an out-of-domain generalization test and a user study. The paper clearly identifies known limitations of prior emotional THG work (audio underuse, identity leakage, isolated emotion learning) and proposes concrete mechanisms for each. The main weakness is that the training objective used for the core disentanglement claim appears to encode identity rather than discard it, and the ablation evidence for cross-identity invariance is indirect. The reported gains in Emo-Score over StyleTalk and EAT are promising, but without error bars their statistical reliability is unclear.

major comments (3)
  1. [Section 3.1, Eq. (8)] The contrastive objective is inconsistent with the identity-agnostic claim. The paper defines positive pairs as clips 'with the same identity and emotion' and negative pairs as clips 'with different identities or emotions.' Under this definition, a clip from speaker B with the same emotion as anchor A is a negative, so the loss actively pushes apart emotion representations across identities. This encourages the embedding to encode identity, contradicting the stated goal that 'codes with similar emotions should cluster together' and the contribution claim of disentangling speaker identity from emotional expressions. To support identity-agnostic emotion learning, positive pairs should include different identities sharing the same emotion, or negatives should be restricted to differ in emotion only. As written, the training signal may produce an identity-dependent emotion representation, which would undermine the method's core contribution and the generalization claims to unseen identities.
  2. [Section 4.5, Table 5] The clustering-strength experiment does not measure cross-identity invariance. The text states that clustering strength is computed 'with different emotions of the same identity as categories,' i.e., the categories are emotions within a single identity. A high ratio only shows that the embedder separates emotions for a fixed identity, which an identity-coded representation can also achieve. This table therefore cannot certify that the emotion priors are identity-agnostic, nor does it support the claim that audio-visual fusion reduces identity leakage. A meaningful test would compute clustering with emotions as categories across multiple identities (e.g., inter-identity same-emotion distance relative to intra-identity same-emotion distance).
  3. [Section 4.5, Table 4] The claim that the emotion bank contributes to identity preservation is not supported by the reported numbers. The row 'w/o emotion bank' gives F-SIM 0.8775, while 'Ours' gives 0.8704; since F-SIM is described as facial similarity with higher being better, the emotion bank slightly reduces identity similarity even though it improves Emo-Score. The sentence that the bank and the discrimination objective 'significantly contribute to emotional expression and identity preservation' is therefore misleading for the bank component. In addition, no error bars or statistical significance tests are reported for any table, so differences such as Emo-Score 0.5424 (Ours-V) vs 0.5527 (Ours-P) or the small F-SIM gap cannot be assessed for reliability.
minor comments (5)
  1. [Eq. (4)-(5)] The softmax in the attention-based aggregation should specify that it is taken over the time dimension; otherwise the weighting in Equations (4) and (5) is ambiguous.
  2. [Section 4.2] The Emo-Score description says 'we selected four easily recognizable emotions—neutral, happy, surprised, and angry,' but the metric from [26] may cover more categories; please clarify how the average is computed and whether the same four-category protocol is applied to the out-of-domain test set.
  3. [References] References [2] and [3] are the same technical report (Stable Video Diffusion) and should be consolidated to avoid duplicate citation entries.
  4. [Figure 4(c)] The t-SNE plot is said to use color intensity to indicate emotion intensity, but no colorbar or explicit intensity labels are provided, making the intensity trend difficult for the reader to verify.
  5. [Section 4.1] The out-of-domain test set is described as comprising '10 reference images and corresponding audio samples.' With only ten reference images, the generalization results in the right half of Table 1 are based on a very small sample; this should be acknowledged and ideally supplemented with a confidence interval or per-sample analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: reported emotion and identity metrics come from external models, and the method's derivation does not reduce to its training inputs by construction.

full rationale

DICE-Talk's reported results are not circular: the emotion accuracy metric (Emo-Score) comes from an external facial expression recognition model [26], lip-sync accuracy from SyncNet [8], and identity preservation from F-SIM, none of which are fitted parameters of DICE-Talk. The main generation loss in Eq. 15 combines denoising, classification, and vector-quantization terms; the classification term L_cls is a training objective, not an evaluation metric, so it cannot by itself manufacture the test-set Emo-Scores reported in Tables 1 and 4. The fixed prompt feature used for Ours-P is the mean of emotion priors over a large number of videos of the same emotion, which is a standard class-conditional conditioning choice rather than a prediction of the test labels. The contrastive objective in Eqs. 7-8 is a legitimate correctness concern: with same-emotion/different-identity clips treated as negatives, the learned embedding may encode identity rather than discard it, so the claimed identity-agnostic property is not guaranteed. However, this is a generalization/validity risk, not a circularity, because the paper's identity-disentanglement claim is not defined as the output of that loss and the reported evaluation does not reduce by construction to the training signal. The only self-citation, [13] Sonic used for an audio attention layer in the overview, is an architectural implementation choice and is not load-bearing. No equation in the paper is equivalent to its own input, and no fitted evaluation constant is renamed as a prediction.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central machinery rests on unstated hyperparameters and on the assumption that the contrastive objective and Gaussian sampling produce identity-agnostic emotion codes. None of these are directly verified. The contrastive pair definition, as written, is in tension with the identity-agnostic goal.

free parameters (5)
  • weight for emotion classification loss (lambda)
    In Eq. 15, L = L_denoising + λ L_cls + L_vq. The value of λ is not reported, but the final emotion accuracy and identity preservation depend on this balance.
  • commitment weight (beta)
    In Eq. 17, β weights the feature alignment loss versus the commitment loss in the VQ emotion bank. Not specified.
  • InfoNCE temperature (tau)
    Eq. 7 uses temperature τ in the contrastive similarity. Not specified.
  • emotion bank size K
    The number of prototypes in the emotion bank C (dimensions K × d_s) is not given. This controls the granularity of emotion correlation and generalization.
  • dropout rates for modality and condition dropping = 5% for diffusion training; embedder dropout rate unspecified
    Section 4.1 states 5% of data drops audio, 5% drops image, 5% drops emotion prior, and 5% drops all conditions. The embedder's random dropout trick is mentioned but the rate is not reported.
assumptions (3)
  • domain assumption MEAD emotion annotations are correct and sufficient as ground truth.
    All training and evaluation rely on MEAD's emotion labels; mislabeled or ambiguous clips propagate into emotion priors, the emotion bank, and Emo-Score evaluation.
  • ad hoc to paper A Gaussian distribution over emotion embeddings captures identity-specific volatility while keeping emotion content in the mean.
    Section 3.1 models emotion as (μ, σ) and claims this yields identity-agnostic representation. This is a modeling assumption with no proof or direct measurement.
  • domain assumption Audio (Whisper-Tiny features) contains sufficient prosodic emotion cues to assist emotion extraction.
    Motivates the cross-modal design; Table 5 shows audio-only clustering is weak (2.44 versus 6.25 for video), so this assumption is only partially supported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation." pith.science (2026). https://pith.science/paper/KDSXRPZV

@misc{pith2026250418087,
  author       = {Pith},
  title        = {Pith review of: Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDSXRPZV}},
  note         = {Machine review of arXiv:2504.18087}
}
read the original abstract

Recent advances in Talking Head Generation (THG) have achieved impressive lip synchronization and visual quality through diffusion models; yet existing methods struggle to generate emotionally expressive portraits while preserving speaker identity. We identify three critical limitations in current emotional talking head generation: insufficient utilization of audio's inherent emotional cues, identity leakage in emotion representations, and isolated learning of emotion correlations. To address these challenges, we propose a novel framework dubbed as DICE-Talk, following the idea of disentangling identity with emotion, and then cooperating emotions with similar characteristics. First, we develop a disentangled emotion embedder that jointly models audio-visual emotional cues through cross-modal attention, representing emotions as identity-agnostic Gaussian distributions. Second, we introduce a correlation-enhanced emotion conditioning module with learnable Emotion Banks that explicitly capture inter-emotion relationships through vector quantization and attention-based feature aggregation. Third, we design an emotion discrimination objective that enforces affective consistency during the diffusion process through latent-space classification. Extensive experiments on MEAD and HDTF datasets demonstrate our method's superiority, outperforming state-of-the-art approaches in emotion accuracy while maintaining competitive lip-sync performance. Qualitative results and user studies further confirm our method's ability to generate identity-preserving portraits with rich, correlated emotional expressions that naturally adapt to unseen identities.

Figures

Figures reproduced from arXiv: 2504.18087 by the authors.

Figure 1
Figure 1. Our DICE-Talk, as an emotional THG method based [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework of DICE-Talk. Our method comprises three key components: disentangled emotion embedder, correlation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visual comparisons with recent state-of-the-art taking head generation methods. Our method can obtain more [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visual results of user and ablation studies. (a): Visual comparisons with StyleTalk and EAT. (b): Comparison of [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Our visual results with different emotions and identities on out of domain dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 28 canonical work pages

  1. [1]

    Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In ACM SIGGRAPH

    work page 1999

  2. [3]

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. 2023. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 (2023)

    arXiv 2023

  3. [4]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In ICML

    work page 2020

  4. [5]

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. 2024. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136 (2024)

    arXiv 2024

  5. [6]

    Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. 2022. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In ACM SIGGRAPH Asia

    work page 2022

  6. [7]

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. 2018. VoxCeleb2: Deep Speaker Recognition. In Interspeech 2018. ISCA. doi:10.21437/interspeech.2018- 1929

    work page doi:10.21437/interspeech.2018- 2018

  7. [8]

    Joon Son Chung and Andrew Zisserman. 2017. Out of time: automated lip sync in the wild. In ACCV Workshops

    work page 2017

  8. [9]

    Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. 2024. Hallo2: Long-duration and high-resolution audio-driven portrait image animation. arXiv preprint arXiv:2410.07718 (2024)

    arXiv 2024

Show all 48 references
  1. [10]

    Yuan Gan, Zongxin Yang, Xihang Yue, Lingyun Sun, and Yi Yang. 2023. Efficient emotional adaptation for audio-driven talking-head generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 22634–22645

    2023

  2. [11]

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS

    2017

  3. [12]

    Siteng Huang, Biao Gong, Yutong Feng, Xi Chen, Yuqian Fu, Yu Liu, and Donglin Wang. 2024. Learning disentangled identifiers for action-customized text-to- image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7797–7806

    2024

  4. [13]

    Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. 2024. Sonic: Shifting Focus to Global Audio Perception in Portrait Animation.arXiv preprint arXiv:2411.16331 (2024)

    2024 arXiv

  5. [14]

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. 2022. Eamm: One-shot emotional talking face via audio-based emotion- aware motion model. In ACM SIGGRAPH

    2022

  6. [15]

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. 2021. Audio-driven emotional video portraits. In CVPR

    2021

  7. [16]

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. 2024. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.02634 (2024)

    2024 arXiv

  8. [17]

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. 2022. Elucidating the design space of diffusion-based generative models.Advances in neural information processing systems 35 (2022), 26565–26577

    2022

  9. [18]

    Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic opti- mization. In ICLR

    2015

  10. [19]

    Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. 2022. Expressive talking head generation with granular audio-visual control. In CVPR

    2022

  11. [20]

    Yifeng Ma, Suzhen Wang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding, Zhi- dong Deng, and Xin Yu. 2023. Styletalk: One-shot talking head generation with controllable speaking styles. In AAAI

    2023

  12. [21]

    Jiancheng Pan, Shiye Lei, Yuqian Fu, Jiahao Li, Yanxing Liu, Yuze Sun, Xiao He, Long Peng, Xiaomeng Huang, and Bo Zhao. 2025. EarthSynth: Generating Infor- mative Earth Observation with Diffusion Models. arXiv preprint arXiv:2505.12108 (2025)

    2025 arXiv

  13. [22]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. InNeurIPS

    2019

  14. [23]

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar

  15. [24]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In ICML

    2023

  16. [25]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In CVPR

    2022

  17. [26]

    Elena Ryumina, Denis Dresvyanskiy, and Alexey Karpov. 2022. In Search of a Robust Facial Expressions Recognition Model: A Large-Scale Visual Cross-Corpus Study. Neurocomputing (2022). doi:10.1016/j.neucom.2022.10.013

    2022 doi

  18. [27]

    Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. 2023. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In CVPR

    2023

  19. [28]

    Sanjana Sinha, Sandika Biswas, Ravindra Yadav, and Brojeshwar Bhowmick. 2022. Emotion-controllable generalized talking face generation. In IJCAI

    2022

  20. [29]

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. 2024. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. In ECCV

    2024

  21. [30]

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. 2019. FVD: A new metric for video genera- tion. In ICLR Workshops

    2019

  22. [31]

    Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems 30 (2017)

    2017

  23. [32]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research (2008)

    2008

  24. [33]

    Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. 2024. V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation.arXiv preprint arXiv:2406.02511 (2024)

    2024 arXiv

  25. [34]

    Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. 2023. Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert. In CVPR

    2023

  26. [35]

    Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. 2020. Mead: A large-scale audio- visual dataset for emotional talking-face generation. In ECCV

    2020

  27. [36]

    Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. 2021. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. In IJCAI

    2021

  28. [37]

    Suzhen Wang, Lincheng Li, Yu Ding, and Xin Yu. 2022. One-shot talking face generation from single-speaker audio-visual correlation learning. In AAAI

    2022

  29. [38]

    Huawei Wei, Zejun Yang, and Zhisheng Wang. 2024. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694 (2024)

    2024 arXiv

  30. [39]

    Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng

  31. [40]

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2023. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV

    2023

  32. [41]

    You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. 2024. X-portrait: Expressive portrait animation with hierarchical motion attention. In ACM SIGGRAPH

    2024

  33. [42]

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jing- dong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. 2024. Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation. arXiv preprint arXiv:2406.08801 (2024)

    2024 arXiv

  34. [43]

    Shurong Yang, Huadong Li, Juhao Wu, Minhao Jing, Linze Li, Renhe Ji, Jiajun Liang, and Haoqiang Fan. 2024. MegActor: Harness the Power of Raw Video for Vivid Portrait Animation. arXiv preprint arXiv:2405.20851 (2024)

    2024 arXiv

  35. [44]

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. 2023. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In CVPR

    2023

  36. [45]

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. 2021. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In CVPR

    2021

  37. [46]

    Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Zi- wei Liu. 2021. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In CVPR

    2021

  38. [47]

    Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. 2020. Makelttalk: speaker-aware talking-head animation. ACM TOG (2020)

    2020

  39. [2020]

    In ACM MM

    A lip sync expert is all you need for speech to lip generation in the wild. In ACM MM

  40. [2021]

    In ACM MM

    Imitating arbitrary talking style for realistic audio-driven talking face synthesis. In ACM MM

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.