Pith. sign in

REVIEW 5 major objections 6 minor 46 references

Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single diffusion transformer with shared weights and adapter modules generates both co-speech gestures and an expressive talking face, claiming state-of-the-art quality on the SHOW dataset with only 27.7M parameters.

desk verdict Genuinely new adapter-based joint face+body architecture, but the SOTA claim is undermined by the paper's own ablation numbers; the case rests on a small user study. read the letter →

arxiv 2412.14333 v1 pith:IMYZEOL6 submitted 2024-12-18 cs.CV

Steven Hogue , Chenxu Zhang , Yapeng Tian , Xiaohu Guo This is my paper · ORCID
classification cs.CV
keywords co-speechgesturegenerationtalkingheaddiffusionmodelsadaptermodulessharedlatentspacecross-modalattentionparameter-efficientSHOWdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that co-speech gestures and an expressive talking face can be generated together by a single diffusion transformer whose weights are shared between the two modalities, instead of by two separate networks or branches. Small adapter modules, each made of two cross-modal attention layers followed by a bottleneck, map face and body motion into a common latent space and let each modality influence the other. On the SHOW dataset the method reports the best FMD, FED, jaw L1, landmark L1, and landmark velocity distance among the compared baselines while using 27.7M parameters, about half of the 51.2M needed by the authors' own separate two-network ablation. A 10-participant user study rates the generated motions above all baselines and ablations and below ground truth alone. If correct, the result would make joint full-body avatars cheaper to train and deploy without sacrificing motion quality.

What carries the argument

The load-bearing object is the adapter-augmented shared transformer block. Each block contains four adapters per layer, two for the face stream and two for the body stream; each adapter begins with two cross-modal attention layers that exchange information between the face and body sequences, then passes through a bottleneck of downsampling, activation, hidden, and upsampling layers. The transformer's weights are shared by both modalities, with only lightweight per-modality projection layers at input and output, and the model is trained to predict the clean sample directly at each diffusion timestep plus a velocity-smoothing loss. At inference, long sequences are stitched from overlapping windows using the last $M$ frames of the previous segment as the seed for the next, with linear interpolation across the overlap.

What would settle it

Train the same architecture on a larger corpus such as BEAT with more speakers, and compare per-modality FMD and landmark L1 against two separate same-capacity models. If the shared-network model is meaningfully worse on both modalities, the shared-latent premise fails. Alternatively, remove the cross-modal attention heads from the adapters; if face and body metrics do not change, the claimed cross-modal influence is not doing the work attributed to it.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that adapter modules, originally designed to adapt pretrained large models, can instead be trained from scratch inside a single randomly initialized transformer to unite two weakly correlated tasks. The network receives face parameters (one jaw joint plus 100 expression blend shapes) and body parameters (43 joints) through separate projection layers, feeds them in parallel through shared transformer blocks, and uses adapter cross-attention so face and body denoising inform each other. Trained on the SHOW dataset with the standard 80/10/10 split, the model reports the lowest FMD (1758.13), FED (1260.01), jaw L1 (0.00161), landmark L1 (0.1532), and LVD (0.0276) among LS3DCG, DiffGesture, TalkSHOW, and DiffSHEG, with a parameter count of 27.7M, and it receives the highest user-study scores among all non-ground-truth methods. The paper concludes that sharing one transformer with small adapters captures the weak correlation between gesture and facial motion while avoiding the parameter duplication of separate networks.

Load-bearing premise

The load-bearing premise is that face motion and body motion can be mapped into one common latent space and modeled by a single shared transformer with only small adapters, without either modality degrading the other.

Editorial extensions

If this is right

  • A single network, rather than two separately trained models, can produce both body gestures and facial motion, cutting memory and training cost.
  • The face and body streams influence each other through cross-modal attention, so the weak correlation between gesture and expression is exploited rather than ignored.
  • Long motion sequences are generated from arbitrary audio by chaining overlapping windows, with a seed gesture from the previous segment and interpolation across overlaps.
  • The user-study results indicate that smooth, temporally consistent face motion is perceived as more realistic than high-variance jittery motion, even where distributional metrics disagree.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shared-latent assumption holds beyond SHOW's four speakers, the same adapter pattern could be extended to more than two output streams, such as hand or eye-gaze channels, without multiplying the core transformer's parameter count.
  • A testable extension is to train on a larger multi-speaker corpus and measure per-modality quality; the parameter-efficiency claim would remain meaningful only if the joint model keeps both body and face metrics within a small margin of dedicated separate models.
  • The paper's own discussion of jittery baselines scoring well on diversity and beat consistency suggests that current Fréchet-style motion metrics reward high variance even when humans judge the motion unnatural; a metric that penalizes velocity noise would sharpen comparisons.
  • Because direct sample prediction plus a velocity loss smooths the outputs, the approach may transfer to other weakly correlated motion pairs such as speech-driven eyebrow and hand motion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a joint co-speech gesture and expressive talking face generation method using a single diffusion transformer with adapter modules. Face and body motions are projected into a shared latent space and processed in parallel by shared transformer blocks, with cross-modal adapters letting the two modalities influence each other. The model is trained on the SHOW dataset with reconstruction and velocity losses, and evaluated against LS3DCG, DiffGesture, TalkSHOW, and DiffSHEG, plus ablations that remove the adapter or split the network. The authors claim state-of-the-art performance with a reduced parameter count, supported by a table of quantitative metrics and a ten-participant user study.

Significance. If the claims are validated, the adapter-based shared-transformer design would be a practically useful step toward parameter-efficient joint generation of weakly correlated body and face motion, and the cross-modal attention mechanism is a reasonable way to let the two tasks inform each other. The manuscript offers a clear architecture, a released code repository, an ablation study, and a user study, which are strengths for reproducibility. However, the quantitative evidence is currently inconsistent: the same metrics that are used to claim superiority in Table 1 are dismissed as misleading in the ablation section, and the only consistently favorable evidence for the adapter design is a small user study without statistical analysis. The parameter-reduction claim is also qualified by the existence of a combined baseline with fewer parameters, and the state-of-the-art claim is not supported on all reported face metrics. Overall, the central idea is promising but the evidence needs substantial strengthening before the paper can support its headline claims.

major comments (5)
  1. [Section 4.3 vs Section 4.4, Tables 1 and 2] The paper uses the same quantitative metrics inconsistently. Table 2 shows that Ours is worse than both Separate and Combined on FMD, FED, Div(All), Div(Face), and BC, and Section 4.4 explains this by arguing that these metrics are misleading because the alternative models produce jittery or dull motions. Yet Section 4.3 and Table 1 use FMD, FED, Div, and BC as evidence of state-of-the-art performance over the baselines. This is internally inconsistent unless the authors provide independent validation that these metrics track human perceptual quality in the direction claimed, for example by correlating metric values with user-study ratings. Without such validation, the paper should either avoid relying on FMD/FED/Div/BC as primary evidence or provide a principled reason why they are valid for comparing against baselines but invalid for comparing against ablations.
  2. [Section 4.5, Table 3] The user study is the only evidence that the adapter architecture improves perceived quality over the Separate and Combined ablations, but it is reported with only ten participants, no confidence intervals, no significance tests, and no description of randomization, blinding, stimulus ordering, or inter-rater agreement. This is too weak to carry the central perceptual claim. The authors should either report a larger, statistically analyzed study or substantially temper the claim that the user study validates higher performance.
  3. [Table 1 and Abstract] The abstract states that the framework 'maintains state-of-the-art co-speech gesture and talking head generation performance,' but Table 1 does not support this on several of the paper's own metrics: LS3DCG achieves better Jaw L1 (0.00147 vs 0.00161), Lmk L1 (0.1410 vs 0.1532), and LVD (0.0273 vs 0.0276), and DiffSHEG achieves better Div(All) (1924.78 vs 1845.15) and Div(Face) (1609.03 vs 1521.68). The paper should either qualify which metrics are being claimed as state-of-the-art or provide a principled weighting of these metrics.
  4. [Section 2.3 and Section 4.3] The related-work section cites EMAGE [21] as the most relevant joint holistic method, where face and body are trained jointly with cross-attention, yet EMAGE is not included in the quantitative or user-study comparisons. Since EMAGE is the closest prior work addressing the same joint task, its absence makes the 'state-of-the-art' claim for joint methods unsubstantiated. The authors should add an EMAGE comparison on the SHOW dataset or explicitly justify its omission.
  5. [Section 4.4 and Abstract] The parameter-reduction claim is overstated. In Table 2, the Combined ablation (25.6M parameters) has fewer parameters than Ours (27.7M), and LS3DCG in Table 1 has 18.7M parameters. The claim of 'significantly reduces the number of parameters' is only true when comparing to the Separate (51.2M) or Split (53.1M) two-network setups, not to all baselines. This limitation should be stated in the abstract and conclusion.
minor comments (6)
  1. [Section 3.2.1] There is a typo in the text: 'p(x0,' should be 'p(x0)' or similar, and the following sentence 'where x0 represents the motion sequence' is incomplete.
  2. [Section 4.1] The phrase 'that gives us (J + 1) × 3 + E = 232 total parameters for the network to predict' is misleading: 232 is the output dimensionality of the motion representation, not the number of trainable parameters. Please reword.
  3. [Section 4.3] The statement 'Diversity and quality are often correlated with a trade-off between the two' is confusing as written; it appears to say the opposite of the intended meaning. Please clarify the relationship between diversity and quality.
  4. [Section 4.3] When describing DiffGesture, the paper says the authors 'additionally train a separate model for the face,' but it does not describe the architecture or training details of that face model, which is important for assessing the fairness of the comparison.
  5. [Section 4.4] The text says jitteriness 'can result in higher diversity and higher beat consistency, as discussed in Sec. 4,' but the relevant discussion is in Section 4.3, not Section 4 in general. Please provide a specific reference.
  6. [Section 4.5 and Table 3] The user-study section should report how many videos were shown per method, whether all methods used the same 12 audio clips, how participants were recruited, and whether the order of videos was randomized across participants.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the method is evaluated empirically against external baselines; the Table 1/Table 2 metric inconsistency is a correctness concern, not circularity.

full rationale

The derivation chain is empirical and self-contained. The method trains a single diffusion transformer with adapters on the SHOW train split using standard reconstruction and velocity losses (Eqs. 4-5), and evaluates on held-out test sequences with FMD/FED, diversity, beat consistency, landmark errors, and a user study. The FMD/FED autoencoders are external metric components trained following prior work ([22] or [5]), not fitted parameters of the generative model that are then relabeled as predictions; they do not encode the paper's conclusions. There are no self-citations, no imported uniqueness theorems, and no ansatz smuggled in via author-cited prior work; the adapter design follows LA VisH [19], which is an independent architectural choice, not a forced result. The apparent tension between Table 1 (Ours best on FMD/FED) and Table 2 (Separate/Combined beat Ours on FMD/FED/Div/BC), with the paper calling those metrics 'misleading,' is a consistency/correctness issue about evidence quality, not a circular derivation. No load-bearing step reduces to its own input, so circularity score is 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its assumptions are standard machine learning choices plus the domain-specific premise that face and body share useful weak correlation in a common latent space. The main under-specified items are hyperparameters (loss weights, adapter sizes) that are necessary for exact reproduction.

free parameters (7)
  • Clip length N = 34
    Sliding window length for training and inference, chosen by hand; affects how much audio and motion context the model sees.
  • Seed overlap M = 4
    Frames of overlap used as seed pose; chosen by hand, affects segment stitching and reliance on the initial pose.
  • Denoising steps = 500
    500-step DDPM used at inference; chosen for a quality-speed trade-off and not shown to be swept or optimized.
  • Body joints J = 43
    Dataset-defined body joints (13 body plus 30 hands); input dimensionality inherited from SHOW, not a learned architectural choice.
  • Expression dimension E = 100
    SHOW dataset blend shape weights; input dimensionality inherited from the dataset and treated as ground truth.
  • Velocity loss weight = not reported
    Equation (5) adds a velocity smoothing term, but the scalar weight relative to the reconstruction loss is not stated, leaving the objective under-specified.
  • Adapter bottleneck dimensions = not reported
    Adapter down/up-sample and hidden sizes are not given, despite being central to parameter count and capacity.
assumptions (5)
  • standard math Denoising diffusion probabilistic model framework (Ho et al.)
    The paper uses the standard DDPM forward and reverse processes for the motion sequence; this is background theory, not derived in the paper.
  • domain assumption HuBERT features capture speech semantics relevant to both gesture and face
    A frozen HuBERT encoder with a trained linear projection supplies conditioning; the paper assumes these features are sufficient for both body and face generation.
  • domain assumption Face and body motion are weakly but usefully correlated
    The central premise that cross-modal attention between face and body improves generation; if the modalities are uncorrelated, the shared model would have no benefit over separate ones.
  • domain assumption SHOW dataset quality and representativeness
    Mocap and expression labels from four speakers are treated as ground truth and adequate for state-of-the-art claims; no cross-dataset validation is performed.
  • domain assumption A single shared latent space can represent both body joint rotations and facial expression weights
    The paper's design premise that the shared transformer can denoise both modalities without a capacity bottleneck; this is the weakest modeling assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters." pith.science (2026). https://pith.science/paper/IMYZEOL6

@misc{pith2026241214333,
  author       = {Pith},
  title        = {Pith review of: Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMYZEOL6}},
  note         = {Machine review of arXiv:2412.14333}
}
read the original abstract

Recent advances in co-speech gesture and talking head generation have been impressive, yet most methods focus on only one of the two tasks. Those that attempt to generate both often rely on separate models or network modules, increasing training complexity and ignoring the inherent relationship between face and body movements. To address the challenges, in this paper, we propose a novel model architecture that jointly generates face and body motions within a single network. This approach leverages shared weights between modalities, facilitated by adapters that enable adaptation to a common latent space. Our experiments demonstrate that the proposed framework not only maintains state-of-the-art co-speech gesture and talking head generation performance but also significantly reduces the number of parameters required.

Figures

Figures reproduced from arXiv: 2412.14333 by the authors.

Figure 1
Figure 1. Framework and Architecture Overview. The top left shows the general network architecture, with inputs of a face motion parameters and body parameters being fed into separate projection layers, combined with conditional information ct from the bottom left, and the noisy latents xt. The transformer network (in green) denoises the latents, which are fed into the network separately. The right is the architecture of our … view at source ↗
Figure 2
Figure 2. Qualitative Comparison. We compare sequences of motions for our method, TalkSHOW and DiffGesture. Our motions are more diverse and dynamic compared to the baselines. Methods FMD↓ Div (All)↑ BC↑ FED↓ Div (Face)↑ Jaw L1↓ Lmk L1↓ LVD↓ Params LS3DCG [10] 2185.23 1314.38 0.791 1618.83 1064.19 0.00147 0.1410 0.0273 18.7M DiffGesture [46] 5831.43 1405.14 0.758 5000.59 1176.17 0.00192 0.2088 0.0333 51.6M TalkSHOW [42] 3681.… view at source ↗
Figure 3
Figure 3. Qualitative Comparison for Ablation Study. Our model produces smooth dynamic motion while the alternative architectures generate jittery motions that move little from the mean position. Methods FMD↓ Div (All)↑ BC↑ FED↓ Div (Face)↑ Jaw L1↓ Lmk L1↓ LVD↓ Params Separate 1525.92 2043.56 0.764 1235.55 1698.91 0.00174 0.1671 0.0289 51.2M Combined 1560.36 1936.99 0.772 994.21 1580.54 0.00172 0.1641 0.0278 25.6M Split 9420.… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 37 canonical work pages

  1. [21]

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J. Black. Emage: Towards unified holis- tic co-speech gesture generation via expressive masked audio gesture modeling, 2024. 2

    work page 2024

  2. [1]

    Style-controllable speech-driven gesture synthesis using normalising flows

    Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow. Style-controllable speech-driven gesture synthesis using normalising flows. In Computer Graphics Forum, 2020. 2

    work page 2020

  3. [2]

    Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings.TOG,

    Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings.TOG,

    work page

  4. [3]

    Gesturediffuclip: Gesture diffusion model with clip latents

    Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents. TOG, 42(4), 2023. 2

    work page 2023

  5. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz...

    2020

  6. [5]

    Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and ges- ture generation

    Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and ges- ture generation. In CVPR, 2024. 1, 2, 5, 6, 8

    work page 2024

  7. [6]

    Hierarchical cross-modal talking face generation with dynamic pixel-wise loss

    Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In CVPR, 2019. 6

    work page 2019

  8. [7]

    Diffusion-based co-speech gesture genera- tion using joint text and audio representation

    Anna Deichler, Shivam Mehta, Simon Alexanderson, and Jonas Beskow. Diffusion-based co-speech gesture genera- tion using joint text and audio representation. In INTER- NATIONAL CONFERENCE ON MULTIMODAL INTERAC- TION, ICMI ’23. ACM, Oct. 2023. 2

    work page 2023

Show all 46 references
  1. [8]

    Faceformer: Speech-driven 3d facial anima- tion with transformers

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech-driven 3d facial anima- tion with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  2. [9]

    Learning individual styles of conversational gesture

    Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, An- drew Owens, and Jitendra Malik. Learning individual styles of conversational gesture. In CVPR, 2019. 1, 2

    2019

  3. [10]

    Learning speech-driven 3d conversational gestures from video

    Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed El- gharib, and Christian Theobalt. Learning speech-driven 3d conversational gestures from video. In Proceedings of the 21st ACM International Conference on Intelligent Virt...

    2021

  4. [11]

    Deber- tav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2021

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. Deber- tav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2021. 3

    2021

  5. [12]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, ed...

    2017

  6. [13]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. NeurIPS, 2020. 3, 4

    2020

  7. [14]

    Parameter-efficient transfer learning for nlp, 2019

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019. 1, 3

    2019

  8. [15]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units, 2021

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units, 2021. 4

    2021

  9. [16]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations , 2022. 1, 3

    2022

  10. [17]

    Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders

    Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders. In ICCV, 2021. 2

    2021

  11. [18]

    Speech2video synthesis with 3d skeleton regularization and expressive body poses

    Miao Liao, Sibo Zhang, Peng Wang, Hao Zhu, Xinxin Zuo, and Ruigang Yang. Speech2video synthesis with 3d skeleton regularization and expressive body poses. In ACCV, 2020. 2

    2020

  12. [19]

    Vision transformers are parameter-efficient audio- visual learners

    Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. Vision transformers are parameter-efficient audio- visual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 4

    2023

  13. [20]

    Ex- ploring versatile generative language model via parameter- efficient transfer learning

    Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Ex- ploring versatile generative language model via parameter- efficient transfer learning. arXiv preprint arXiv:2004.03829,

    2004 arXiv

  14. [22]

    Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis, 2022

    Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis, 2022. 7

    2022

  15. [23]

    Audio-driven co-speech gesture video generation

    Xian Liu, Qianyi Wu, Hang Zhou, Yuanqi Du, Wayne Wu, Dahua Lin, and Ziwei Liu. Audio-driven co-speech gesture video generation. NeurIPS, 2022. 2

    2022

  16. [24]

    Learning hierarchical cross-modal association for co- speech gesture generation

    Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co- speech gesture generation. In CVPR, 2022. 2

    2022

  17. [25]

    Roberta: A robustly optimized bert pretraining approach, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. 3

    2019

  18. [26]

    Repaint: Inpaint- ing using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andr´es Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpaint- ing using denoising diffusion probabilistic models. CoRR, abs/2201.09865, 2022. 8 9

    2022 arXiv

  19. [27]

    Lcm-lora: A universal stable-diffusion acceleration module, 2023

    Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolin´ario Passos, Longbo Huang, Jian Li, and Hang Zhao. Lcm-lora: A universal stable-diffusion acceleration module, 2023. 1, 3

    2023

  20. [28]

    Bodyformer: Semantics-guided 3d body gesture synthesis with transformer

    Kunkun Pang, Dafei Qin, Yingruo Fan, Julian Habekost, Takaaki Shiratori, Junichi Yamagishi, and Taku Komura. Bodyformer: Semantics-guided 3d body gesture synthesis with transformer. TOG, 2023. 2

    2023

  21. [29]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. 3

    2023

  22. [30]

    Speech drives templates: Co-speech gesture synthesis with learned templates

    Shenhan Qian, Zhi Tu, Yihao Zhi, Wen Liu, and Shenghua Gao. Speech drives templates: Co-speech gesture synthesis with learned templates. In ICCV, 2021. 2

    2021

  23. [31]

    Hierarchical text-conditional image gener- ation with clip latents, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents, 2022. 4

    2022

  24. [32]

    Bilen, and A

    S-A Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. InAdvances in Neural Information Processing Systems, 2017. 3

    2017

  25. [33]

    Difftalk: Crafting diffusion models for generalized audio-driven portraits animation

    Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In CVPR, 2023. 2

    2023

  26. [34]

    Co-speech gesture synthesis by reinforcement learning with contrastive pre- trained rewards

    Mingyang Sun, Mengchen Zhao, Yaqing Hou, Minglei Li, Huang Xu, Songcen Xu, and Jianye Hao. Co-speech gesture synthesis by reinforcement learning with contrastive pre- trained rewards. In CVPR, 2023. 2

    2023

  27. [35]

    Human motion diffu- sion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffu- sion model. In ICLR, 2023. 4

    2023

  28. [36]

    Imitator: Personalized speech-driven 3d facial animation

    Balamurugan Thambiraja, Ikhsanul Habibie, Sadegh Aliak- barian, Darren Cosker, Christian Theobalt, and Justus Thies. Imitator: Personalized speech-driven 3d facial animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 20621–20631, October

  29. [37]

    Edge: Editable dance generation from music

    Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In CVPR, 2023. 4

    2023

  30. [38]

    FVD: A new metric for video generation, 2019

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. FVD: A new metric for video generation, 2019. 5

    2019

  31. [39]

    Codetalker: Speech-driven 3d facial animation with discrete motion prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12780–12790, 2023. 2

    2023

  32. [40]

    Diffus- estylegesture: Stylized audio-driven co-speech gesture gen- eration with diffusion models

    Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao. Diffus- estylegesture: Stylized audio-driven co-speech gesture gen- eration with diffusion models. In Proceedings of the 32nd International Joint Conference on Artificial Int...

    2023

  33. [41]

    Audio-driven stylized gesture generation with flow-based model

    Sheng Ye, Yu-Hui Wen, Yanan Sun, Ying He, Ziyang Zhang, Yaoyuan Wang, Weihua He, and Yong-Jin Liu. Audio-driven stylized gesture generation with flow-based model. InECCV,

  34. [42]

    Generating holistic 3d human motion from speech

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. InCVPR,

  35. [43]

    Speech ges- ture generation from the trimodal context of text, audio, and speaker identity

    Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech ges- ture generation from the trimodal context of text, audio, and speaker identity. TOG, 2020. 2, 5

    2020

  36. [44]

    Robots learn social skills: End-to-end learning of co-speech gesture generation for hu- manoid robots

    Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jae- hong Kim, and Geehyuk Lee. Robots learn social skills: End-to-end learning of co-speech gesture generation for hu- manoid robots. In ICRA, 2019. 2

    2019

  37. [45]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. arXiv preprint arXiv:2211.12194, 2022. 2

    2022 arXiv

  38. [46]

    Taming diffusion models for audio-driven co-speech gesture generation

    Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for audio-driven co-speech gesture generation. In CVPR, 2023. 1, 2, 6, 7, 8 10

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.