Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that modeling body-region interactions with spatial then temporal attention and accelerating flow matching with latent shortcut learning yields state-of-the-art, real-time co-speech gesture generation on BEAT2.

desk verdict A well-ablated empirical system for fast co-speech gesture generation, but the shortcut equation that carries the speed/quality claim is written inconsistently, and without code the headline result is not yet reproducible. read the letter →

arxiv 2501.18898 v3 pith:IMUMMJFF submitted 2025-01-31 cs.CV cs.GR

Pinxin Liu , Luchuan Song , Junhua Huang , Haiyang Liu , Chenliang Xu This is my paper · ORCID
classification cs.CVcs.GR
keywords co-speechgesturegenerationflowmatchinglatentshortcutmodelspatial-temporalattentionresidualvectorquantizationreal-timeinferenceBEAT2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces GestureLSM, a method that generates full-body co-speech gestures from audio and text in real time while keeping the gestures natural and coordinated. It argues that prior systems fail on two fronts: they model body regions separately, losing the spatial interactions that make movements coherent, and they rely on slow iterative or autoregressive sampling. GestureLSM addresses both by tokenizing body regions with residual vector quantization, applying spatial attention across regions within a frame followed by temporal attention across time, and replacing slow sampling with a latent shortcut flow-matching model. The paper claims this combination beats existing methods on the BEAT2 benchmark, reaching an FGD of 4.088 with an average inference time of 0.039 seconds per sentence on one NVIDIA A100. A sympathetic reader would care because natural, real-time gesture generation is a core requirement for digital avatars and embodied agents.

What carries the argument

The central machinery is the latent shortcut model applied to an RVQ-quantized gesture manifold, paired with a spatial-then-temporal attention transformer. Residual vector quantization converts each body region into discrete tokens, giving a compact latent space in which the shortcut function operates. The shortcut model predicts future latent states conditioned on speech and on a step size $d$, and the self-consistency rule in Eq. (6) forces larger shortcuts to equal the average of two smaller shortcuts, enabling the model to generate quality motion in few sampling steps. The spatial and temporal attention modules handle body-region coherence and motion dynamics, respectively, and the beta timestep distribution in Eq. (7) concentrates training samples near the poorly predicted early denoising region.

What would settle it

A direct test is to measure how FGD changes as the number of sampling steps drops from 8 to 1 on a subset of BEAT2 sentences with rapid, nonlinear hand-torso coordination. If the shortcut rule is valid on the latent manifold, the quality drop should be small; a large drop, as already hinted by the reported 1-step FGD of 6.235 versus 8-step FGD of 4.088, would indicate the linearity assumption fails exactly in the fast-sampling regime the paper advertises.

Watch

Extended reading notes

Core claim

GestureLSM's central claim is that co-speech gesture generation can simultaneously achieve state-of-the-art quality and real-time speed when the generator explicitly models inter-region body interactions and uses a flow-matching model trained with latent shortcut learning and $\beta$-distributed timestep sampling. The model first quantizes body motion into discrete tokens for hands, upper body, lower body, and facial expressions, then applies spatial attention to coordinate body regions within each frame and temporal attention to model motion dynamics across time. For fast sampling, it defines a shortcut function $f_\theta(a, x_t, t, d)$ that predicts the displacement from a latent state $x_t$ to a later state $x'_{t+d}$, and enforces the self-consistency rule $f_\theta(a, x_t, t, 2d) = \tfrac{1}{2} f_\theta(a, x_t, t, d) + \tfrac{1}{2} f_\theta(a, x'_{t+d}, t, d)$, so that large jumps decompose into averaged smaller jumps. The paper also reports that in gesture generation the velocity prediction error is largest as timesteps approach $t=1$, and that a left-skewed $\beta$ timestep distribution during training substantially improves generation quality. With eight sampling steps the model achieves an FGD of 4.088 on BEAT2, a Beat Constancy of 0.714 close to the ground-truth value of 0.703, and an average inference time of 0.039 seconds per sentence, outperforming diffusion and autoregressive baselines on both quality and speed.

Load-bearing premise

The load-bearing premise is that the self-consistency rule in Eq. (6), which treats a shortcut of size $2d$ as exactly the average of two shortcuts of size $d$, is a valid approximation on the learned RVQ gesture latent manifold for every timestep and step size.

Editorial extensions

If this is right

  • At 0.039 seconds per sentence, GestureLSM exceeds real-time rates for avatar animation on a single A100, making live speech-driven gesture generation feasible for interactive applications.
  • The two-step variant, with FGD 4.988 and inference time 0.018 seconds, already surpasses the previous best reported FGD of 4.687 from SynTalker, showing that few-step shortcut sampling is practically usable.
  • Explicit spatial attention over tokenized body regions yields coordinated whole-body gestures, such as both hands extending to emphasize "very important," rather than the disjointed region-wise motions the paper attributes to separate decoding.
  • The model's Beat Constancy of 0.714 is closer to the ground-truth 0.703 than any baseline, supporting the paper's argument that rhythm metrics should be evaluated relative to real human motion rather than maximized.
  • The reported ablation shows that combining latent shortcut learning with ReFlow2 enables one- or two-step sampling with only a slight performance drop, indicating a route to even faster inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's finding that flow-matching loss peaks near $t=1$ for gesture generation, and that left-skewed beta sampling fixes it, may transfer to other flow-matching tasks where the initial denoising step is the hardest, not just speech-driven motion.
  • The spatial-then-temporal attention design could be tested on other multi-part motion generation tasks, such as dance or sign language, where separate body-part decoding is also a known source of incoherence.
  • Since the paper shows that raw Beat Constancy can be misleading, future gesture-generation benchmarks should report rhythm metrics as a distance to ground truth rather than optimizing for higher values.
  • The linearity assumption in the shortcut rule could be stress-tested by measuring the curvature of the learned ODE trajectories in the RVQ latent space; high curvature regions would indicate where the shortcut approximation is weakest.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GestureLSM, a flow-matching-based co-speech gesture generator that operates on RVQ-quantized tokens for four body regions and combines spatial attention across regions with temporal attention across frames. The method introduces a latent shortcut model with a self-consistency objective and a beta-distributed timestep sampling schedule to accelerate inference. The authors report state-of-the-art results on BEAT2 (FGD 4.088, inference time 0.039 s per sentence on an A100), supported by ablations over modules, architectures, model types, speed-up baselines, sampling steps, and hyperparameters, plus a user study.

Significance. If the speed/quality trade-off holds, GestureLSM is a practically useful contribution to real-time co-speech gesture generation. The ablation coverage is unusually broad for this area: the paper compares spatial-temporal attention against several architectural alternatives, compares shortcut learning against LCM, ReFlow, and distillation baselines, and reports training-time costs. The spatial-temporal attention design is clean and the empirical case for it is reasonably strong. The main weaknesses are that the central shortcut equation is not well-defined as written, the headline numbers are selected using test-set FGD, and the reported metrics lack uncertainty quantification; these issues currently prevent the SOTA claim from being fully verifiable.

major comments (4)
  1. [Section 3.3, Eq. (5) and Eq. (6)] Equation (6) is not a valid self-consistency condition under the definition in Eq. (5). Eq. (5) defines f_theta(a, x_t, t, d) as a displacement, since x'_{t+d} = x_t + f_theta(...). For two successive d-step shortcuts, consistency requires f_theta(a, x_t, t, 2d) = f_theta(a, x_t, t, d) + f_theta(a, x'_{t+d}, t+d, d), because the second sub-shortcut starts at time t+d. Eq. (6) instead averages the two sub-shortcuts with weight 1/2 and evaluates the second term at time t. If f_theta is instead intended as a per-unit-time velocity, Eq. (5) is missing the factor d and the second time argument is still wrong. Since this self-consistency loss is the mechanism behind the 8-step FGD of 4.088 in Table 2f, the training objective as written cannot reproduce the headline result. Please correct the equation or the definition and clarify which convention is used; the absence of released code or weights makes this impossible to resolve from the manuscript alone.
  2. [Section 4.4 and Appendix D] The hyperparameters of the beta schedule (alpha, beta), the classifier-free guidance scale, and the number of sampling steps are selected using the same BEAT2 test-set FGD reported in Tables 2c, 2f, 3b, and 3e. No separate validation split or cross-validation procedure is described. The headline FGD of 4.088 is therefore a best-selected test value rather than an out-of-sample prediction. This should be fixed by introducing a validation split, or by reporting the selection procedure and confirming that the final test evaluation is performed only after all hyperparameters are fixed.
  3. [Tables 1-3 and Figure 7] All main quantitative results are reported as point estimates without error bars, confidence intervals, or significance tests. The SOTA claims, such as FGD 4.088 versus 4.687 for SynTalker in Table 1, are based on single runs. The user study has 20 participants and 320 samples, but Figure 7 reports only mean MOS values without variance or statistical testing. Please add multiple-seed runs and appropriate statistical reporting for the central comparisons.
  4. [Section 4.2 and Table 1] The inference-speed metric is inconsistently defined: the caption of Table 1 says 'Average Inference Time per Sentence (AIST)', the text in Section 4.2 reports '0.039 seconds per frame', and the abstract says '0.039 seconds per sentence'. Because real-time inference is one of the two central claims, the paper must state the exact evaluation protocol, including sequence length, batch size, and whether the time is per frame or per sentence, and make all mentions consistent.
minor comments (5)
  1. [Abstract and Introduction] There are several copyediting issues: 'remains challenges' should be 'remains challenging', 'cohernet' should be 'coherent', and the contribution list would benefit from proofreading for grammar and punctuation.
  2. [Figure 3 caption] The caption reads 'GesutureLSM' instead of 'GestureLSM'.
  3. [Section 4.2, Beat Constancy] The text says 'the higher the better' for Beat Constancy but then argues that matching the ground-truth value is the correct target. Please clarify the intended interpretation of BC in the main text and specify how the 'closeness to ground truth' comparison is computed.
  4. [Appendix D, Table 3a] The BC values for the concatenation and addition variants (5.479 and 6.234) are far outside the plausible range of the BC metric and are not discussed. If these are real, please explain; if they are artifacts of a different metric definition, please correct or remove them.
  5. [Section 3.3, Eq. (7)] Please define the beta function B(alpha, beta) and clarify how the sampled t values are mapped to the flow-matching schedule, including any discretization used during training.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GestureLSM's central claim is an empirical benchmark result, and its training objectives are external or self-contained rather than derived from the metric.

full rationale

The paper's central claim is an empirical evaluation result, not a derivation whose output is equivalent to its input. The self-consistency rule in Eq. (6) is a training objective adopted and adapted from Shortcut Models [13]; it constrains the model's shortcuts rather than defining the evaluation metric, so it is not self-definitional. The beta time-stamp schedule is selected by comparing FGD values in ablations, but the paper explicitly adopts the EMAGE train-validation-test split, and no text states that the final reported FGD is computed on the same split used for hyperparameter selection; without such a statement, treating the SOTA number as a fitted parameter would be speculation. Self-citations ([20, 32, 33, 49–51]) are contextual or concern implementation details such as codebook resets, and they do not carry the central speed/quality claim. The possible algebraic mismatch between Eq. (5) and Eq. (6) is a correctness and reproducibility concern, not a circularity of the kind this pass flags. I therefore find no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are postulated; the method introduces algorithmic components only. The free-parameter ledger captures the numbers fitted during model selection on the evaluation metric.

free parameters (3)
  • Beta distribution shape parameters (alpha, beta) = alpha=2.0, beta=1.2
    Selected using FGD on the BEAT2 test set across schedules (Table 2c, Table 3e); the same test FGD is then reported as the headline SOTA result.
  • Classifier-free guidance scale = 2.0
    Ablated over 1.0 to 3.0 in Table 3b and chosen by best FGD.
  • Number of sampling steps = 8
    Chosen from Table 2f as the quality and speed operating point; the real-time claim depends on this choice on an A100.
assumptions (4)
  • domain assumption Rectified flow ODE and shortcut self-consistency objective transfer from image latent space to quantized gesture latent space.
    Equations (4) to (6) in Section 3.3 are applied to RVQ gesture tokens without a formal argument that the linear interpolation and self-consistency rule hold for human motion dynamics.
  • domain assumption BEAT2 and the EMAGE train, validation, and test split are representative for evaluating co-speech gesture quality.
    Section 4.1 uses the split protocol from EMAGE and treats this single dataset as the basis for the state-of-the-art claim.
  • domain assumption FGD, Diversity, BC, and MSE are sufficient and correctly oriented metrics for gesture realism and synchronization.
    Section 4.2 and Appendix C define the metrics; the paper itself changes the BC convention from higher is better to closer to ground truth, which is an additional assumption.
  • ad hoc to paper Spatial attention followed by temporal attention is an adequate factorization of full-body spatial-temporal dependencies.
    This design is the paper's central architectural proposal; its validity is supported only by ablations on a single dataset and not by a theoretical argument.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling." pith.science (2026). https://pith.science/paper/IMUMMJFF

@misc{pith2026250118898,
  author       = {Pith},
  title        = {Pith review of: GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMUMMJFF}},
  note         = {Machine review of arXiv:2501.18898}
}
read the original abstract

Generating full-body human gestures based on speech signals remains challenges on quality and speed. Existing approaches model different body regions such as body, legs and hands separately, which fail to capture the spatial interactions between them and result in unnatural and disjointed movements. Additionally, their autoregressive/diffusion-based pipelines show slow generation speed due to dozens of inference steps. To address these two challenges, we propose GestureLSM, a flow-matching-based approach for Co-Speech Gesture Generation with spatial-temporal modeling. Our method i) explicitly model the interaction of tokenized body regions through spatial and temporal attention, for generating coherent full-body gestures. ii) introduce the flow matching to enable more efficient sampling by explicitly modeling the latent velocity space. To overcome the suboptimal performance of flow matching baseline, we propose latent shortcut learning and beta distribution time stamp sampling during training to enhance gesture synthesis quality and accelerate inference. Combining the spatial-temporal modeling and improved flow matching-based framework, GestureLSM achieves state-of-the-art performance on BEAT2 while significantly reducing inference time compared to existing methods, highlighting its potential for enhancing digital humans and embodied agents in real-world applications. Project Page: https://andypinxinliu.github.io/GestureLSM

Figures

Figures reproduced from arXiv: 2501.18898 by the authors.

Figure 1
Figure 1. GestureLSM. We present Gesture Latent Shortcut Model, a method that generates full-body human gestures from speech and text scripts with high quality and real-time speed. It explicitly models the body regions interactions, e.g., the interactions between body and hands, to achieve coherent gesture motions. Besides, it is also capable of real-time generation based on shortcut sampling. Abstract Generating full-body hu… view at source ↗
Figure 2
Figure 2. Our GestureLSM achieves significant generation quality [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of GesutureLSM. (1) Our GestureLSM generate full-body gestures from speech and text scripts. The concatenated [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The details of Latent Shortcut Model. GestureLSM en [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Time Sampling Comparison. For various time sam￾pling schedules, beta schedule performs the best, i.e., lowest train￾ing loss, with skewed pattern (left) to counteract the ineffective￾ness of model prediction when t approaches 1 (right). To sample from the flow model, a…
Figure 6
Figure 6. Figure 6: The subjective comparisons results. Compared with other methods, GestureLSM presents more natural gesture motions and local [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: User Study. Our GestureLSM have higher user ratings with a clear margin on Realness, Synchrony, and Smoothness. 4.3. Qualitative Comparisons Evaluation Results As depicted in [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Avatar Video Generation Application. After generat￾ing a sequence of gesture motions conditioned on speech audio by GestureLSM, we project the 3D keypoints to a 2D plane, serving as keypoint guidance for avatar video generation based on 2D an￾imation methods like Anima…
Figure 9
Figure 9. Figure 9: Training dynamics of key evaluation metrics. (a) Beat constancy decreases, indicating a shift from overly rigid beat-following [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A semantically aligned latent space plus multi-token cross-attention conditioning sets a new state of the art in text-to-human-motion generation on HumanML3D.

Reference graph

Works this paper leans on

66 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, 2023. 2

    work page 2023

  2. [2]

    Nonver- bal Behaviors, Persuasion, and Credibility

    Judee K Burgoon, Thomas Birk, and Michael Pfau. Nonver- bal Behaviors, Persuasion, and Credibility. Human commu- nication research, 17(1):140–169, 1990. 1

    work page 1990

  3. [3]

    Everybody Dance Now

    Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody Dance Now. In ICCV, 2019. 2

    work page 2019

  4. [4]

    Enabling synergistic full-body control in prompt-based co-speech motion generation

    Bohong Chen, Yumeng Li, Yao-Xiang Ding, Tianjia Shao, and Kun Zhou. Enabling synergistic full-body control in prompt-based co-speech motion generation. In Proceedings of the 32nd ACM International Conference on Multimedia , page 10, New York, NY , USA, 2024. ACM. 2, 3, 7

    work page 2024

  5. [5]

    DiffSHEG: A Diffusion-Based Ap- proach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation, 2024

    Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. DiffSHEG: A Diffusion-Based Ap- proach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation, 2024. 2, 7

    2024

  6. [6]

    WavLM: Large-Scale Self- Supervised Pre-Training for Full Stack Speech Processing

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-Scale Self- Supervised Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing , 16 (6):1505–1518, 2022. 12

    work page 2022

  7. [7]

    Motionlcm: Real-time control- lable motion generation via latent consistency model

    Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlcm: Real-time control- lable motion generation via latent consistency model. arXiv preprint arXiv:2404.19759, 2024. 8

    arXiv 2024

  8. [8]

    The Interplay Between Gesture and Speech in the Production of Referring Expressions: Investigating the Tradeoff Hypothe- sis

    Jan P De Ruiter, Adrian Bangerter, and Paula Dings. The Interplay Between Gesture and Speech in the Production of Referring Expressions: Investigating the Tradeoff Hypothe- sis. Topics in cognitive science, 4(2):232–248, 2012. 1

    work page 2012

Show all 66 references
  1. [9]

    Diffusion-based co-speech gesture genera- tion using joint text and audio representation

    Anna Deichler, Shivam Mehta, Simon Alexanderson, and Jonas Beskow. Diffusion-based co-speech gesture genera- tion using joint text and audio representation. In INTER- NATIONAL CONFERENCE ON MULTIMODAL INTERAC- TION. ACM, 2023. 2

    2023

  2. [10]

    BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805, 2018. 3

    2018 arXiv

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  4. [12]

    Scaling rectified flow trans- formers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yan- nik Marek, and Robin Rombach. Scaling rectified flow tr...

    2024

  5. [13]

    One step diffusion via shortcut models, 2024

    Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models, 2024. 3, 5

    2024

  6. [14]

    Eraseanything: Enabling concept erasure in rectified flow transformers

    Daiheng Gao, Shilin Lu, Wenbo Zhou, Jiaming Chu, Jie Zhang, Mengxi Jia, Bang Zhang, Zhaoxin Fan, and Weiming Zhang. Eraseanything: Enabling concept erasure in rectified flow transformers. In Forty-second International Conference on Machine Learning, 2025. 3

    2025

  7. [15]

    Ginosar, A

    S. Ginosar, A. Bar, G. Kohavi, C. Chan, A. Owens, and J. Malik. Learning Individual Styles of Conversational Ges- ture. In CVPR. IEEE, 2019. 2

    2019

  8. [16]

    Momask: Generative masked model- ing of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked model- ing of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1900–1910, 2024. 7

    1900

  9. [17]

    Learning speech-driven 3d conversational gestures from video

    Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed El- gharib, and Christian Theobalt. Learning speech-driven 3d conversational gestures from video. arXiv preprint arXiv:2102.06837, 2021. 1

    2021 arXiv

  10. [18]

    Denoising diffu- sion probabilistic models, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models, 2020. 2, 7

    2020

  11. [19]

    Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation. arXiv preprint arXiv:2311.17117, 2023. 8

    2023 arXiv

  12. [20]

    Modeling and driving human body soundfields through acoustic primitives, 2024

    Chao Huang, Dejan Markovic, Chenliang Xu, and Alexan- der Richard. Modeling and driving human body soundfields through acoustic primitives, 2024. 1

    2024

  13. [21]

    Autoregressive Image Generation Using Residual Quantization, 2022

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive Image Generation Using Residual Quantization, 2022. 3

    2022

  14. [22]

    Improving the training of rectified flows, 2024

    Sangyun Lee, Zinan Lin, and Giulia Fanti. Improving the training of rectified flows, 2024. 5

    2024

  15. [23]

    Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders

    Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 112...

    2021

  16. [24]

    Set you straight: Auto-steering denoising trajectories to sidestep unwanted concepts, 2025

    Leyang Li, Shilin Lu, Yan Ren, and Adams Wai-Kin Kong. Set you straight: Auto-steering denoising trajectories to sidestep unwanted concepts, 2025. 2

    2025

  17. [25]

    AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. AI Choreographer: Music Conditioned 3D Dance Generation with AIST++. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13401–13412, 2021. 5, 12

    2021

  18. [26]

    Flow Matching for Generative Modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matt Le. Flow Matching for Generative Modeling. arXiv preprint arXiv:2210.02747, 2022. 4

    2022 arXiv

  19. [27]

    DisCo: Disentan- gled Implicit Content and Rhythm Learning for Diverse Co- Speech Gestures Synthesis

    Haiyang Liu, Naoya Iwamoto, Zihao Zhu, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. DisCo: Disentan- gled Implicit Content and Rhythm Learning for Diverse Co- Speech Gestures Synthesis. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3764–3773,

  20. [28]

    9 BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis

    Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. 9 BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis. arXiv preprint arXiv:2203.05297, 2022. 2, 7

    2022 arXiv

  21. [29]

    EMAGE: Towards Unified Holistic Co- Speech Gesture Generation via Masked Audio Gesture Mod- eling

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Naoya Iwamoto, Bo Zheng, and Michael J Black. EMAGE: Towards Unified Holistic Co- Speech Gesture Generation via Masked Audio Gesture Mod- eling. arXiv preprint arXiv:2401.00374, 2023. 1, 2, 3, 5, 7

    2023 arXiv

  22. [30]

    Tango: Co-speech gesture video reenactment with hi- erarchical audio motion embedding and diffusion interpola- tion

    Haiyang Liu, Xingchao Yang, Tomoya Akiyama, Yuantian Huang, Qiaoge Li, Shigeru Kuriyama, and Takafumi Take- tomi. Tango: Co-speech gesture video reenactment with hi- erarchical audio motion embedding and diffusion interpola- tion. arXiv preprint arXiv:2410.04221, 2024. 2

    arXiv

  23. [31]

    Semges: Semantics-aware co-speech gesture genera- tion using semantic coherence and relevance learning, 2025

    Lanmiao Liu, Esam Ghaleb, Aslı ¨Ozy¨urek, and Zerrin Yu- mak. Semges: Semantics-aware co-speech gesture genera- tion using semantic coherence and relevance learning, 2025

    2025

  24. [32]

    Intentional gesture: Deliver your intentions with gestures for speech, 2025

    Pinxin Liu, Haiyang Liu, Luchuan Song, and Chenliang Xu. Intentional gesture: Deliver your intentions with gestures for speech, 2025

    2025

  25. [33]

    Contextual gesture: Co- speech gesture video generation through context-aware ges- ture representation, 2025

    Pinxin Liu, Pengfei Zhang, Hyeongwoo Kim, Pablo Garrido, Ari Sharpio, and Kyle Olszewski. Contextual gesture: Co- speech gesture video generation through context-aware ges- ture representation, 2025. 1, 2, 12

    2025

  26. [34]

    Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022. 3, 4, 5, 7, 8

    2022

  27. [35]

    Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation

    Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation. In CVPR, pages 10462– 10472, 2022. 2, 7

    2022

  28. [36]

    Instaflow: One step is enough for high-quality diffusion- based text-to-image generation

    Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion- based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023. 3

    2023

  29. [37]

    Towards variable and coordinated holistic co-speech motion generation

    Yifei Liu, Qiong Cao, Yandong Wen, Huaiguang Jiang, and Changxing Ding. Towards variable and coordinated holistic co-speech motion generation. arXiv preprint arXiv:2404.00368, 2024. 7, 12

    2024 arXiv

  30. [38]

    Tf-icon: Diffusion-based training-free cross-domain image composi- tion

    Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. Tf-icon: Diffusion-based training-free cross-domain image composi- tion. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV) , pages 2294–2305, 2023. 2

    2023

  31. [39]

    Mace: Mass concept erasure in diffusion models

    Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6430–6440, 2024

    2024

  32. [40]

    Robust watermarking using generative pri- ors against image editing: From benchmarking to advances

    Shilin Lu, Zihan Zhou, Jiayou Lu, Yuanzhi Zhu, and Adams Wai-Kin Kong. Robust watermarking using generative pri- ors against image editing: From benchmarking to advances. arXiv preprint arXiv:2410.18775, 2024. 2

    2024 arXiv

  33. [41]

    Latent consistency models: Synthesizing high- resolution images with few-step inference

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high- resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023. 3

    2023 arXiv

  34. [42]

    Dctdiff: Intriguing properties of im- age generative modeling in the dct space

    Mang Ning, Mingxiao Li, Jianlin Su, Haozhe Jia, Lanmiao Liu, Martin Bene ˇs, Wenshuo Chen, Albert Ali Salah, and Itir Onal Ertugrul. Dctdiff: Intriguing properties of im- age generative modeling in the dct space. arXiv preprint arXiv:2412.15032, 2024. 2

    2024 arXiv

  35. [43]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive Body Capture: 3D Hands, Face, and Body from a Single Image. In CVPR, 2019. 2, 8

    2019

  36. [44]

    MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

    Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation. In CVPR, 2023. 2

    2023

  37. [45]

    Fast high- resolution image synthesis with latent adversarial diffusion distillation

    Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high- resolution image synthesis with latent adversarial diffusion distillation. In SIGGRAPH Asia 2024 Conference Papers , pages 1–11, 2024. 3

    2024

  38. [46]

    Talking face video generation with editable expression

    Luchuan Song, Bin Liu, and Nenghai Yu. Talking face video generation with editable expression. In Image and Graphics: 11th International Conference, ICIG 2021, Haikou, China, August 6–8, 2021, Proceedings, Part III 11, pages 753–764. Springer, 2021. 1

    2021

  39. [47]

    Fsft-net: face transfer video generation with few-shot views

    Luchuan Song, Guojun Yin, Bin Liu, Yuhui Zhang, and Nenghai Yu. Fsft-net: face transfer video generation with few-shot views. In 2021 IEEE international conference on image processing (ICIP), pages 3582–3586. IEEE, 2021. 2

    2021

  40. [48]

    Emotional listener portrait: Neural lis- tener head generation with emotion

    Luchuan Song, Guojun Yin, Zhenchao Jin, Xiaoyi Dong, and Chenliang Xu. Emotional listener portrait: Neural lis- tener head generation with emotion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 20839–20849, 2023. 1

    2023

  41. [49]

    Texttoon: Real-time text toonify head avatar from single video

    Luchuan Song, Lele Chen, Celong Liu, Pinxin Liu, and Chenliang Xu. Texttoon: Real-time text toonify head avatar from single video. In SIGGRAPH Asia 2024 Conference Pa- pers, pages 1–11, 2024. 1

    2024

  42. [50]

    Tri 2-plane: Thinking head avatar via fea- ture pyramid

    Luchuan Song, Pinxin Liu, Lele Chen, Guojun Yin, and Chenliang Xu. Tri 2-plane: Thinking head avatar via fea- ture pyramid. In European Conference on Computer Vision, pages 1–20. Springer, 2024

    2024

  43. [51]

    Adaptive super resolution for one-shot talking-head genera- tion

    Luchuan Song, Pinxin Liu, Guojun Yin, and Chenliang Xu. Adaptive super resolution for one-shot talking-head genera- tion. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4115–4119, 2024. 1

    2024

  44. [52]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 3

    2023 arXiv

  45. [53]

    Generative ai for cel- animation: A survey

    Yunlong Tang, Junjia Guo, Pinxin Liu, Zhiyuan Wang, Hang Hua, Jia-Xing Zhong, Yunzhong Xiao, Chao Huang, Luchuan Song, Susan Liang, et al. Generative ai for cel- animation: A survey. arXiv preprint arXiv:2501.06250 ,

  46. [54]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. 4, 7 10

    2023

  47. [55]

    Rectified diffusion: Straightness is not your need in rectified flow, 2024

    Fu-Yun Wang, Ling Yang, Zhaoyang Huang, Mengdi Wang, and Hongsheng Li. Rectified diffusion: Straightness is not your need in rectified flow, 2024. 3

    2024

  48. [56]

    High-Resolution Im- age Synthesis and Semantic Manipulation with Conditional GANs

    Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-Resolution Im- age Synthesis and Semantic Manipulation with Conditional GANs. In CVPR, 2018. 2

    2018

  49. [57]

    Codetalker: Speech-driven 3d facial animation with discrete motion prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12780–12790, 2023. 5

    2023

  50. [58]

    Chain of generation: Multi-modal gesture synthesis via cascaded conditional control, 2023

    Zunnan Xu, Yachao Zhang, Sicheng Yang, Ronghui Li, and Xiu Li. Chain of generation: Multi-modal gesture synthesis via cascaded conditional control, 2023. 2

    2023

  51. [59]

    Mambatalk: Ef- ficient holistic gesture synthesis with selective state space models, 2024

    Zunnan Xu, Yukang Lin, Haonan Han, Sicheng Yang, Ronghui Li, Yachao Zhang, and Xiu Li. Mambatalk: Ef- ficient holistic gesture synthesis with selective state space models, 2024. 1, 2, 7

    2024

  52. [60]

    Generating Holistic 3D Human Motion from Speech

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating Holistic 3D Human Motion from Speech. In CVPR, 2023. 1, 2, 3, 7

    2023

  53. [61]

    One-step diffusion with distribution matching distillation

    Tianwei Yin, Micha ¨el Gharbi, Richard Zhang, Eli Shecht- man, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 6613–6623...

    2024

  54. [62]

    Speech Ges- ture Generation from the Trimodal Context of Text, Audio, and Speaker Identity

    Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech Ges- ture Generation from the Trimodal Context of Text, Audio, and Speaker Identity. ACM TOG, 39(6), 2020. 5, 12

    2020

  55. [63]

    Kinmo: Kinematic-aware human motion understanding and generation, 2024

    Pengfei Zhang, Pinxin Liu, Hyeongwoo Kim, Pablo Garrido, and Bindita Chaudhuri. Kinmo: Kinematic-aware human motion understanding and generation, 2024. 2

    2024

  56. [64]

    Semantic gestic- ulator: Semantics-aware co-speech gesture synthesis, 2024

    Zeyi Zhang, Tenglong Ao, Yuyao Zhang, Qingzhe Gao, Chuan Lin, Baoquan Chen, and Libin Liu. Semantic gestic- ulator: Semantics-aware co-speech gesture synthesis, 2024. 2

    2024

  57. [65]

    Slimflow: Training smaller one-step diffusion models with rectified flow, 2024

    Yuanzhi Zhu, Xingchao Liu, and Qiang Liu. Slimflow: Training smaller one-step diffusion models with rectified flow, 2024. 3

    2024

  58. [66]

    Oftsr: One-step flow for image super- resolution with tunable fidelity-realism trade-offs, 2024

    Yuanzhi Zhu, Ruiqing Wang, Shilin Lu, Junnan Li, Hanshu Yan, and Kai Zhang. Oftsr: One-step flow for image super- resolution with tunable fidelity-realism trade-offs, 2024. 3 11 A. Overview The supplementary document contains implementation de- tails, metric details, additiona...

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.