REVIEW 6 cited by
DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The art of communication beyond speech there are gestures. The automatic co-speech gesture generation draws much attention in computer animation. It is a challenging task due to the diversity of gestures and the difficulty of matching the rhythm and semantics of the gesture to the corresponding speech. To address these problems, we present DiffuseStyleGesture, a diffusion model based speech-driven gesture generation approach. It generates high-quality, speech-matched, stylized, and diverse co-speech gestures based on given speeches of arbitrary length. Specifically, we introduce cross-local attention and self-attention to the gesture diffusion pipeline to generate better speech matched and realistic gestures. We then train our model with classifier-free guidance to control the gesture style by interpolation or extrapolation. Additionally, we improve the diversity of generated gestures with different initial gestures and noise. Extensive experiments show that our method outperforms recent approaches on speech-driven gesture generation. Our code, pre-trained models, and demos are available at https://github.com/YoungSeng/DiffuseStyleGesture.
Forward citations
Cited by 6 Pith papers
-
M3G: Multi-Granular Gesture Generator for Audio-Driven Full-Body Human Motion Synthesis
A multi-granular VQ-VAE that encodes and reconstructs motion at several time scales, paired with an audio-to-token predictor, lowers FGD and beats EMAGE in perceptual A/B tests on BEAT2 full-body gesture generation.
-
SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis
SemTalk generates co-speech body motion by separating rhythm-based base gestures from semantically important sparse gestures and blending them with a learned frame-level semantic score.
-
VersatileMotion: A Unified Framework for Motion Synthesis and Comprehension
A unified multimodal motion LLM that handles nine generation and comprehension tasks across text, audio, and single/multi-agent motion, backed by a new dataset and tokenizer.
-
DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2
DiM-Gestor generates co-speech 3D gestures from Chinese speech using a Mamba-2 and AdaLN diffusion architecture, and releases the CCG dataset.
-
SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning
A two-stage VQ-VAE and crossmodal transformer with coherence and relevance losses produces semantically aware co-speech gestures, beating four baselines on BEAT and TED Expressive for FGD, diversity, and SRGR.
-
Inter-Diffusion Generation Model of Speakers and Listeners for Effective Communication
An inter-diffusion model generates speaker and listener full-body gestures from speech audio by coupling dual-branch denoisers with role-specific condition scaling.
Discussion (0). Continue with ORCID to comment.