REVIEW 6 cited by
TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Audio-driven talking head generation has drawn growing attention. To produce talking head videos with desired facial expressions, previous methods rely on extra reference videos to provide expression information, which may be difficult to find and hence limits their usage. In this work, we propose TalkCLIP, a framework that can generate talking heads where the expressions are specified by natural language, hence allowing for specifying expressions more conveniently. To model the mapping from text to expressions, we first construct a text-video paired talking head dataset where each video has diverse text descriptions that depict both coarse-grained emotions and fine-grained facial movements. Leveraging the proposed dataset, we introduce a CLIP-based style encoder that projects natural language-based descriptions to the representations of expressions. TalkCLIP can even infer expressions for descriptions unseen during training. TalkCLIP can also use text to modulate expression intensity and edit expressions. Extensive experiments demonstrate that TalkCLIP achieves the advanced capability of generating photo-realistic talking heads with vivid facial expressions guided by text descriptions.
Forward citations
Cited by 6 Pith papers
-
EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions
EmoteGPT regresses FLAME 3DMM expression parameters from explicit or implicit text using an MLLM with a dedicated <Expr> token, trained on the new Txt2Emote dataset plus image data, outperforming prior text-to-3D face...
-
Think2Sing: Orchestrating Structured Motion Subtitles for Singing-Driven 3D Head Animation
Think2Sing uses LLM-generated, time-aligned motion subtitles and a motion-intensity proxy to guide diffusion-based 3D head animation from singing audio and lyrics.
-
CEM-Net: Cross-Emotion Memory Network for Emotional Talking Face Generation
CEM-Net stores cross-emotion expression displacements in a memory bank so a generated talking face matches the emotion in the audio even when the reference image emotion conflicts.
-
Exploring Timeline Control for Facial Motion Generation
A diffusion model generates natural facial motions from user-specified multi-track timelines, using TICC-based frame-level action interval annotation for training and evaluation.
-
EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis
EDTalk++ disentangles talking-head video into four orthogonal motion banks (mouth, pose, eyes, expression) and drives them from either video or audio inputs.
-
MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding
A 3D facial animation framework that disentangles content and emotion and predicts frame-wise emotion intensity from audio plus text for dynamic expressions.
Discussion (0). Sign in to comment.