Pith. sign in

REVIEW 4 major objections 6 minor 59 references

SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SemGes claims that aligning gesture and text embeddings at two semantic levels produces more realistic and more diverse co-speech gestures than existing methods.

desk verdict Solid two-stage gesture generation with real gains, but Eq. 6 doesn't deliver the claimed semantic coherence and SRGR is built on the same annotations as the loss. read the letter →

arxiv 2507.19359 v1 pith:CMRFJPRL submitted 2025-07-25 cs.CV

Lanmiao Liu , Esam Ghaleb , Asl{i} Özyürek , Zerrin Yumak This is my paper · ORCID
classification cs.CV
keywords co-speechgesturegenerationsemanticcoherencerelevanceVQ-VAEcross-modaltransformermultimodalsynthesisspeakeridentitygestures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that co-speech gesture generation can be made semantically grounded by combining two complementary signals: a global alignment between text and motion latents and a local emphasis on moments annotated as carrying semantic meaning. It argues that current models mostly produce rhythmic beat gestures and miss representational gestures such as iconic or metaphoric ones. SemGes builds a two-stage system, a VQ-VAE motion prior for hands and body followed by a cross-modal transformer that fuses audio, text semantics, and speaker identity, and reports that the semantic coherence and relevance losses improve realism, diversity, and semantic recall. If the result holds, avatars could gesture with meaning rather than only rhythm.

What carries the argument

The load-bearing mechanism is a two-stage latent-space design with two semantics-specific losses. The VQ-VAE stage discretizes body and hand motion into separate codebooks, giving a compositional motion prior. The semantic coherence loss then pulls a trainable text encoder into the frozen motion latent space via cosine similarity, aligning discourse-level meaning with movement. The semantic relevance loss, a piecewise quadratic-to-linear penalty scaled by an annotation factor, makes the decoder pay extra attention to temporally local moments labeled as semantically meaningful. Together with the multimodal quantization consistency loss, these objectives ensure that the final generated gestures match both word-level and global semantics.

What would settle it

Have a fresh annotator mark semantic gestures on the BEAT test split and recompute SRGR, or rerun the user study, for SemGes with and without the semantic-relevance loss; if the advantage disappears under fresh annotations, the loss is fitting annotation patterns rather than generalizable semantics.

Watch

Extended reading notes

Core claim

The central claim is that semantic gestures are best generated by keeping global and local semantics in one framework. In Stage 1, separate VQ-VAEs learn discrete codebooks for hand and body motion. In Stage 2, a cross-modal transformer takes HuBERT audio features, FastText word embeddings, and speaker identity; a semantic coherence loss maximizes cosine similarity between text encoder outputs and frozen motion encodings, while a semantic relevance loss up-weights reconstruction error at annotated semantic-gesture moments with a Huber-style piecewise penalty. A multimodal quantization consistency loss aligns the fused representation with the learned motion codebooks before decoding. On BEAT, SemGes reports an FGD of 4.467 versus 6.623 for the best baseline, an SRGR of 0.256, and a Diversity of 305.706; on TED Expressive it reports an FGD of 7.263 and a Diversity of 302.772. The paper also reports a lower Beat Consistency score and attributes this to the semantic focus rather than to rhythmic alignment.

Load-bearing premise

The load-bearing premise is that BEAT's semantic-relevance annotations are complete and accurate enough that up-weighting reconstruction error at annotated moments teaches generalizable semantic gestures rather than merely teaching the model where annotators placed labels.

Editorial extensions

If this is right

  • Avatar systems can move beyond beat-synchronized gestures to produce iconic and metaphoric gestures tied to the words being spoken.
  • The overlap-and-combine inference routine lets a model trained on short clips generate long, smoothly continuous gesture sequences without additional training.
  • Freezing the motion encoder and training only the text-side alignment is sufficient to improve semantic recall, so the pipeline needs no extra motion supervision for semantics.
  • Because hands and body have separate codebooks, the framework can coordinate full-body and fine hand gestures from a single multimodal latent code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test is to relabel a held-out portion of BEAT with fresh annotations: if SRGR gains depend on matching the original annotator's label positions rather than on word meaning, the advantage should shrink.
  • The same coherence-plus-relevance recipe could transfer to other sparse-annotation generation tasks, such as templated explainer agents or sign-language animation, where only a few moments carry meaning.
  • Because the text encoder is trained against a frozen motion encoder, rare or abstract words with weak FastText embeddings may yield weaker iconic gestures; testing on low-frequency semantic words would reveal this.
  • A stronger test of generalization would be training on one set of topics and probing gestures for unseen topic-specific words, which the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SemGes, a two-stage framework for co-speech gesture generation. Stage 1 trains separate VQ-VAEs for hand and body motions; Stage 2 uses a cross-modal Transformer conditioned on HuBERT audio features, FastText-based transcript embeddings, and speaker identity. Training combines a semantic coherence loss (Eq. 6), a multimodal quantization consistency loss (Eq. 7), and a semantic relevance loss (Eq. 9), with a clip-overlap long-sequence inference strategy. On BEAT and TED Expressive, the authors report improved FGD, Diversity, and SRGR over four baselines, plus user-study ratings, and provide ablations of the two semantic losses.

Significance. If the results hold, SemGes would be a useful practical advance: it is a complete two-stage system with code/data links, ablations, a user study, and state-of-the-art objective numbers on two benchmarks. The strongest assets are the attempt to combine global text-motion alignment with local relevance weighting and the evaluation on two datasets. However, the evidence for the central semantic-coherence claim is currently weak: Eq. (6) is not contrastive as claimed, the SRGR metric is tied to the training annotations, and no direct semantic alignment test is provided. These issues are fixable in revision, so the work is conditionally positioned rather than rejected.

major comments (4)
  1. [Section 3.2.1, Eq. (6)] The text states that the loss 'maximize[s] the similarity of correct (gesture, transcript) pairs and minimize[s] it for mismatched pairs,' but Eq. (6) contains only positive terms: (1 - cos(Z_h, Z_s)) + (1 - cos(Z_b, Z_s)). There are no negative samples, no margin, and no temperature-scaled softmax over negatives. Minimizing Eq. (6) can be achieved by mapping all transcripts to a single point or by any update that raises positive cosine similarity; nothing in the objective penalizes a mismatched transcript being similar to a gesture. Consequently, the claimed 'discourse-level context' and 'semantic coherence' are not formal consequences of the stated objective. I recommend adding a true contrastive term (e.g., in-batch negatives with a margin or temperature) and, more importantly, evaluating the module with a cross-modal retrieval or alignment test rather than only with global FGD and Diversity.
  2. [Section 5 and Eq. (9)] The Semantic-Relevant Gesture Recall (SRGR) metric is computed from the same BEAT semantic-relevance annotations that weight the semantic relevance loss in Eq. (9). The SRGR gains in Table 1 may therefore reflect learning where annotated moments occur rather than generalizable semantic alignment; the paper does not report a direct semantic-alignment test that is independent of the training labels. In addition, the annotation relevance factor lambda in Eq. (9) is never specified, so the strength of the up-weighting and its effect on SRGR cannot be assessed. Please report a decoupled evaluation (e.g., human judgments of iconicity or metaphoricity on held-out unannotated moments, or a retrieval test on novel text-motion pairs) and specify the value or schedule of lambda.
  3. [Table 1, Section 5] The reported gains over baselines are presented without error bars, confidence intervals, or significance tests for any objective metric. Given that the central SOTA claim rests on FGD 4.467 versus 6.623 for DiffSheg, a single seed without variance is difficult to assess. Moreover, the Beat Consistency of SemGes (0.453) is far below all baselines (0.797–0.922), and the paper's explanation that this is 'expected' is supported only by a separate beat-dominant evaluation reported in prose (0.689), not in the main table. Please provide results over at least three seeds with variance and a significance test, and include the beat-dominant experiment in the tables.
  4. [Table 2, Section 5] The ablation for the semantic coherence module reports only FGD, Diversity, and SRGR, none of which measures whether matched text-motion pairs are closer than mismatched pairs. Thus the claimed contribution of the coherence loss is not independently verified. A simple retrieval precision@1 score or a contrastive alignment metric on held-out clips would directly test the module; without such a test, the 'w/o Semantic Coherence' row cannot distinguish the effect of semantic coherence from the effect of simply adding another text encoder to the model.
minor comments (6)
  1. [Section 5 heading] The heading 'Quantitive Objective Evaluations' should read 'Quantitative Objective Evaluations.'
  2. [Eq. (4)] The stop-gradient notation in Eq. (4) is written as 'sgr' and is never defined; please define it explicitly or replace it with the standard 'sg' notation with a one-line explanation.
  3. [Algorithm 1] Algorithm 1 overwrites the first four frames of each clip with the last four frames of the previous clip, but the pseudocode does not describe any blending or smoothing at the seam, so it is unclear how 'smooth continuity' is achieved.
  4. [Supplementary material] The main text repeatedly refers to supplementary sections for implementation details, additional evaluations, and limitations, but the supplementary material is not included with the arXiv submission; it should be made available for review.
  5. [Section 4.2 and Table 1] The baseline name is inconsistent: the method is called 'DiffGesture' in Table 1 but 'DiffGes' in Section 4.2; please use one consistent name throughout.
  6. [Table 2] The TED-Expressive column for 'w/o Semantic Relevance Module' is marked with dashes, but the text does not explicitly state in the table caption or notes that this is because the semantic relevance loss was not used for TED-Expressive; please add that explanation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the training losses are standard supervised objectives and the semantic-relevance train/eval label overlap is not a fitted prediction.

full rationale

SemGes's derivation chain is self-contained. Stage 1 trains a VQ-VAE with a reconstruction and commitment objective (Eq. 4) on ground-truth motion. Stage 2 optimizes three differentiable losses: cosine alignment between frozen motion encodings and a trainable text encoder (Eq. 6), quantization consistency between the fused multimodal latent and quantized ground-truth motion codes (Eq. 7), and a reconstruction error weighted by BEAT's semantic-relevance annotations (Eqs. 9-10). None of these equations defines its target quantity in terms of the quantity it later claims to predict; the model is trained to reconstruct motion and align representations, then evaluated on held-out data with standard metrics (FGD, BC, Diversity, SRGR). The semantic relevance loss and SRGR both use BEAT's semantic-relevance annotations: this is a train/evaluation label overlap, not a fitted parameter renamed as a prediction, and it does not make the central claim true by construction. The prose around Eq. 6 states that mismatched (gesture, transcript) pairs are minimized, but the equation contains only positive-pair cosine terms; this is a support gap for the discourse-level coherence claim and a correctness concern, not circularity. No load-bearing self-citation or imported uniqueness theorem appears, and the main FGD, Diversity, and user-study results are independent of the semantic annotation loop.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three assumptions: the text embedding space is semantically relevant to gestures, cosine similarity in the learned latent space reflects coherence, and the BEAT semantic-relevance labels are trustworthy. The only explicit hand-chosen constants are alpha=0.01 and the unspecified lambda weighting factor; the model does not introduce new physical or conceptual entities.

free parameters (2)
  • alpha = 0.01
    A hand-chosen threshold in the piecewise semantic relevance loss (Eq. 10).
  • lambda = not specified
    The annotation relevance factor in Eq. 9 is never given a numeric value in the paper; it controls how strongly annotated semantic moments are up-weighted in training.
assumptions (3)
  • domain assumption Pre-trained FastText word embeddings capture word semantics relevant to gestures.
    The semantic encoder is initialized from a pre-trained FastText model and kept fixed; the paper assumes these word vectors carry the semantics needed for gestures (Section 3.2.1).
  • ad hoc to paper Cosine similarity in a learned shared embedding space is a valid measure of semantic coherence between text and motion.
    The semantic coherence loss directly penalizes 1 - cos(Z_h, Z_s); the paper provides no intrinsic validation of this embedding space (Eq. 6).
  • domain assumption BEAT semantic-relevance annotations are reliable labels for meaningful gestures.
    Both the relevance loss (Eq. 9) and the SRGR evaluation rely on these annotations; if they are noisy or incomplete, the semantic grounding claim weakens (Sections 4.1, 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning." pith.science (2026). https://pith.science/paper/CMRFJPRL

@misc{pith2026250719359,
  author       = {Pith},
  title        = {Pith review of: SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMRFJPRL}},
  note         = {Machine review of arXiv:2507.19359}
}
read the original abstract

Creating a virtual avatar with semantically coherent gestures that are aligned with speech is a challenging task. Existing gesture generation research mainly focused on generating rhythmic beat gestures, neglecting the semantic context of the gestures. In this paper, we propose a novel approach for semantic grounding in co-speech gesture generation that integrates semantic information at both fine-grained and global levels. Our approach starts with learning the motion prior through a vector-quantized variational autoencoder. Built on this model, a second-stage module is applied to automatically generate gestures from speech, text-based semantics and speaker identity that ensures consistency between the semantic relevance of generated gestures and co-occurring speech semantics through semantic coherence and relevance modules. Experimental results demonstrate that our approach enhances the realism and coherence of semantic gestures. Extensive experiments and user studies show that our method outperforms state-of-the-art approaches across two benchmarks in co-speech gesture generation in both objective and subjective metrics. The qualitative results of our model, code, dataset and pre-trained models can be viewed at https://semgesture.github.io/.

Figures

Figures reproduced from arXiv: 2507.19359 by the authors.

Figure 1
Figure 1. SemGes integrates audio, text-based semantics, and speaker identity to produce both contextually relevant (discourse-level) and fine-grained (local) gestures. A seman￾tic coherence module aligns text and motion embeddings. The multimodal consistency loss synchronizes the quantized mul￾timodal representations to match the quantized learned motion features for final speech-driven semantics-aware gesture gen￾eration. T… view at source ↗
Figure 2
Figure 2. We pre-train two VQ-VAEs by reconstructing body [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. SemGes employs three training pathways: (1) Global semantic coherence, which minimizes latent disparities between [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Semantic Coherence Embedding Learning. 3.2.1. Semantic Coherence Embedding Learning To align text-based semantics with motion embeddings, we introduce a shared embedding space for both mo￾tion priors and speech transcripts. Specifically, we em￾bed word tokens using a p…
Figure 5
Figure 5. Figure 5: Comparisons with baselines and ground truth gestures. Compared to the baseline method, our approach generates [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Average ratings of users for ground truth ges [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 51 canonical work pages

  1. [1]

    Low-resource adaptation for personalized co-speech gesture generation

    Chaitanya Ahuja, Dong Won Lee, and Louis-Philippe Morency. Low-resource adaptation for personalized co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20566–20576, 2022. 6

    work page 2022

  2. [2]

    Continual learning for personalized co-speech gesture generation

    Chaitanya Ahuja, Pratik Joshi, Ryo Ishii, and Louis- Philippe Morency. Continual learning for personalized co-speech gesture generation. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 20893–20903, 2023. 6

    work page 2023

  3. [3]

    Style-controllable speech-driven gesture synthesis using normalising flows

    Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow. Style-controllable speech-driven gesture synthesis using normalising flows. In Computer Graphics Forum , pages 487–496. Wiley Online Library, 2020. 2

    work page 2020

  4. [4]

    Listen, denoise, action! audio- driven motion synthesis with diffusion models

    Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio- driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–20, 2023. 6

    work page 2023

  5. [5]

    Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural em- beddings

    Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural em- beddings. ACM Transactions on Graphics (TOG), 41(6): 1–19, 2022. 2, 3

    work page 2022

  6. [6]

    Gesturedif- fuclip: Gesture diffusion model with clip latents

    Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturedif- fuclip: Gesture diffusion model with clip latents. ACM Transactions on Graphics (TOG), 42(4):1–18, 2023. 2, 3, 6

    work page 2023

  7. [7]

    Probabilistic fasttext for multi-sense word embeddings

    Ben Athiwaratkun, Andrew Gordon Wilson, and Anima Anandkumar. Probabilistic fasttext for multi-sense word embeddings. arXiv preprint arXiv:1806.02901, 2018. 7

    arXiv 2018

  8. [8]

    Enriching word vectors with subword information

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the association for compu- tational linguistics, 5:135–146, 2017. 4

    work page 2017

Show all 59 references
  1. [9]

    Realtime multi-person 2d pose estimation using part affinity fields

    Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291– 7299, 2017. 5

    2017

  2. [10]

    Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expres- sion and gesture generation

    Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expres- sion and gesture generation. In CVPR, 2024. 2, 6

    2024

  3. [11]

    Emotional speech-driven 3d body animation via disen- tangled latent diffusion

    Kiran Chhatre, Nikos Athanasiou, Giorgio Becherini, Christopher Peters, Michael J Black, Timo Bolkart, et al. Emotional speech-driven 3d body animation via disen- tangled latent diffusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, p...

    1942

  4. [12]

    Ad- versarial gesture generation with realistic gesture phas- ing

    Ylva Ferstl, Michael Neff, and Rachel McDonnell. Ad- versarial gesture generation with realistic gesture phas- ing. Computers & Graphics, 89:117–130, 2020. 2

    2020

  5. [13]

    Learning co-speech gesture representations in dialogue through contrastive learning: An intrinsic evaluation

    Esam Ghaleb, Bulat Khaertdinov, Wim Pouw, Marlou Rasenberg, Judith Holler, Asli Ozyurek, and Raquel Fern´andez. Learning co-speech gesture representations in dialogue through contrastive learning: An intrinsic evaluation. In Proceedings of the 26th International Conference on ...

  6. [14]

    I see what you mean: Co-speech ges- tures for reference resolution in multimodal dialogue

    Esam Ghaleb, Bulat Khaertdinov, Aslı ¨Ozy¨urek, and Raquel Fern´andez. I see what you mean: Co-speech ges- tures for reference resolution in multimodal dialogue. In Proceedings of the 63rd Conference of the Association for Computational Linguistics (ACL Findings), 2025. To appear. 1

    2025

  7. [15]

    Generating diverse and natu- ral 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natu- ral 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152–5161, 2022. 2

    2022

  8. [16]

    Tm2t: Stochastic and tokenized modeling for the recip- rocal generation of 3d human motions and texts

    Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the recip- rocal generation of 3d human motions and texts. In Eu- ropean Conference on Computer Vision, pages 580–597. Springer, 2022. 2

    2022

  9. [17]

    A motion matching-based framework for controllable gesture synthesis from speech

    Ikhsanul Habibie, Mohamed Elgharib, Kripasindhu Sarkar, Ahsan Abdullah, Simbarashe Nyatsanga, Michael Neff, and Christian Theobalt. A motion matching-based framework for controllable gesture synthesis from speech. In ACM SIGGRAPH 2022 conference proceedings, pages 1–9, 2022. 6

    2022

  10. [18]

    Moglow: Probabilistic and controllable motion synthesis using normalising flows

    Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Transactions on Graphics (TOG), 39(6):1–14, 2020. 2

    2020

  11. [19]

    Multimodal lan- guage processing in human communication

    Judith Holler and Stephen C Levinson. Multimodal lan- guage processing in human communication. Trends in Cognitive Sciences, 23(8):639–652, 2019. 1

    2019

  12. [20]

    Hubert: Self-supervised speech repre- sentation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrah- man Mohamed. Hubert: Self-supervised speech repre- sentation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing,...

    2021

  13. [21]

    Gesture units, gesture phrases and speech

    Adam Kendon. Gesture units, gesture phrases and speech. In Gesture: Visible Action as Utterance , chap- ter 7, page 108–126. Cambridge University Press, 2004. 1

    2004

  14. [22]

    Gesticulator: A framework for semantically-aware speech-driven gesture generation

    Taras Kucherenko, Patrik Jonell, Sanne Van Waveren, Gustav Eje Henter, Simon Alexandersson, Iolanda Leite, and Hedvig Kjellstr ¨om. Gesticulator: A framework for semantically-aware speech-driven gesture generation. In Proceedings of the 2020 international conference on multimo...

    2020

  15. [23]

    Danceformer: Music conditioned 3d dance generation with parametric motion transformer

    Buyu Li, Yongchi Zhao, Shi Zhelun, and Lu Sheng. Danceformer: Music conditioned 3d dance generation with parametric motion transformer. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 1272–1279, 2022. 2

    2022

  16. [24]

    Audio2gestures: Gen- erating diverse gestures from speech audio with condi- tional variational autoencoders

    Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Audio2gestures: Gen- erating diverse gestures from speech audio with condi- tional variational autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 11...

    2021

  17. [25]

    Ai choreographer: Music conditioned 3d dance generation with aist++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 13401–13412, 2021. 6

    2021

  18. [26]

    Seeg: Semantic energized co-speech gesture generation

    Yuanzhi Liang, Qianyu Feng, Linchao Zhu, Li Hu, Pan Pan, and Yi Yang. Seeg: Semantic energized co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 10473–10482, 2022. 2, 6

    2022

  19. [27]

    Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis

    Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. In Euro- pean Conference on Computer Vision , pages 612–630. Springer, ...

    2022

  20. [28]

    Emage: To- wards unified holistic co-speech gesture generation via expressive masked audio gesture modeling

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J Black. Emage: To- wards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. In Proceed- ings of the IEEE/CV...

    2024

  21. [29]

    A survey on deep multi-modal learn- ing for body language recognition and generation

    Li Liu, Lufei Gao, Wentao Lei, Fengji Ma, Xiaotian Lin, and Jinting Wang. A survey on deep multi-modal learn- ing for body language recognition and generation. arXiv preprint arXiv:2308.08849, 2023. 1

    2023 arXiv

  22. [30]

    Human gesture recognition with a flow- based model for human robot interaction

    Lanmiao Liu, Chuang Yu, Siyang Song, Zhidong Su, and Adriana Tapus. Human gesture recognition with a flow- based model for human robot interaction. In Compan- ion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, pages 548–551, 2023. 2

    2023

  23. [31]

    Gesturelsm: Latent shortcut based co-speech gesture generation with spatial-temporal mod- eling

    Pinxin Liu, Luchuan Song, Junhua Huang, Haiyang Liu, and Chenliang Xu. Gesturelsm: Latent shortcut based co-speech gesture generation with spatial-temporal mod- eling. arXiv preprint arXiv:2501.18898, 2025. 6

    2025 arXiv

  24. [32]

    Audio-driven co-speech gesture video generation

    Xian Liu, Qianyi Wu, Hang Zhou, Yuanqi Du, Wayne Wu, Dahua Lin, and Ziwei Liu. Audio-driven co-speech gesture video generation. Advances in Neural Informa- tion Processing Systems, 35:21386–21399, 2022. 2

    2022

  25. [33]

    Learning hierarchical cross-modal associa- tion for co-speech gesture generation

    Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal associa- tion for co-speech gesture generation. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recogni...

    2022

  26. [34]

    Towards variable and coordinated holistic co-speech motion generation

    Yifei Liu, Qiong Cao, Yandong Wen, Huaiguang Jiang, and Changxing Ding. Towards variable and coordinated holistic co-speech motion generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 1566–1576, 2024. 6

    2024

  27. [35]

    Hand and mind

    David McNeill. Hand and mind. Advances in Visual Semiotics, 351, 1992. 1

    1992

  28. [36]

    Convofusion: Multi-modal conversational diffusion for co-speech gesture synthesis

    Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, and Chris- tian Theobalt. Convofusion: Multi-modal conversational diffusion for co-speech gesture synthesis. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  29. [37]

    Retrieving se- mantics from the deep: an rag solution for gesture syn- thesis

    M Hamza Mughal, Rishabh Dabral, Merel CJ Scholman, Vera Demberg, and Christian Theobalt. Retrieving se- mantics from the deep: an rag solution for gesture syn- thesis. arXiv preprint arXiv:2412.06786, 2024. 6

    2024 arXiv

  30. [38]

    From audio to photoreal embodiment: Synthe- sizing humans in conversations

    Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthe- sizing humans in conversations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2024

  31. [39]

    Dctdiff: Intriguing proper- ties of image generative modeling in the dct space.arXiv preprint arXiv:2412.15032, 2024

    Mang Ning, Mingxiao Li, Jianlin Su, Haozhe Jia, Lan- miao Liu, Martin Bene ˇs, Wenshuo Chen, Albert Ali Salah, and Itir Onal Ertugrul. Dctdiff: Intriguing proper- ties of image generative modeling in the dct space.arXiv preprint arXiv:2412.15032, 2024. 2

    2024 arXiv

  32. [40]

    A com- prehensive review of data-driven co-speech gesture gen- eration

    Simbarashe Nyatsanga, Taras Kucherenko, Chaitanya Ahuja, Gustav Eje Henter, and Michael Neff. A com- prehensive review of data-driven co-speech gesture gen- eration. In Computer Graphics Forum, pages 569–596. Wiley Online Library, 2023. 1, 2

    2023

  33. [41]

    Hearing and seeing meaning in speech and gesture: Insights from brain and behaviour

    Aslı ¨Ozy¨urek. Hearing and seeing meaning in speech and gesture: Insights from brain and behaviour. Philo- sophical Transactions of the Royal Society B: Biological Sciences, 369(1651):20130296, 2014. 1

    2014

  34. [42]

    Bodyformer: Semantics-guided 3d body gesture synthe- sis with transformer

    Kunkun Pang, Dafei Qin, Yingruo Fan, Julian Habekost, Takaaki Shiratori, Junichi Yamagishi, and Taku Komura. Bodyformer: Semantics-guided 3d body gesture synthe- sis with transformer. ACM Transactions on Graphics (TOG), 42(4):1–12, 2023. 6

    2023

  35. [43]

    Expressive body capture: 3d hands, face, and body from a single image

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognit...

    2019

  36. [44]

    Weakly-supervised emotion transition learning for diverse 3d co-speech gesture gen- eration

    Xingqun Qi, Jiahao Pan, Peng Li, Ruibin Yuan, Xiaowei Chi, Mengfei Li, Wenhan Luo, Wei Xue, Shanghang Zhang, Qifeng Liu, et al. Weakly-supervised emotion transition learning for diverse 3d co-speech gesture gen- eration. In Proceedings of the IEEE/CVF Conference on Computer Vi...

    2024

  37. [45]

    Co- speech gesture synthesis by reinforcement learning with contrastive pre-trained rewards

    Mingyang Sun, Mengchen Zhao, Yaqing Hou, Minglei Li, Huang Xu, Songcen Xu, and Jianye Hao. Co- speech gesture synthesis by reinforcement learning with contrastive pre-trained rewards. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2...

    2023

  38. [46]

    Motionclip: Exposing human mo- tion generation to clip space

    Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human mo- tion generation to clip space. InEuropean Conference on Computer Vision, pages 358–374. Springer, 2022. 3

    2022

  39. [47]

    Human mo- tion diffusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human mo- tion diffusion model. In The Eleventh International Con- ference on Learning Representations, 2022. 2

    2022

  40. [48]

    Neural dis- crete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural dis- crete representation learning. Advances in neural infor- mation processing systems, 30, 2017. 3

    2017

  41. [49]

    Augmented co-speech gesture generation: Including form and meaning features to guide learning-based gesture synthesis

    Hendric V oß and Stefan Kopp. Augmented co-speech gesture generation: Including form and meaning features to guide learning-based gesture synthesis. In Proceed- ings of the 23rd ACM International Conference on Intel- ligent Virtual Agents, pages 1–8, 2023. 2, 6

    2023

  42. [50]

    Diffusestylegesture: Stylized audio-driven co- speech gesture generation with diffusion models

    Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao. Diffusestylegesture: Stylized audio-driven co- speech gesture generation with diffusion models. arXiv preprint arXiv:2305.04919, 2023. 2

    2023 arXiv

  43. [51]

    Qpgesture: Quantization-based and phase-guided mo- tion matching for natural speech-driven gesture gener- ation

    Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, and Haolin Zhuang. Qpgesture: Quantization-based and phase-guided mo- tion matching for natural speech-driven gesture gener- ation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2023

  44. [52]

    Audio-driven stylized gesture generation with flow- based model

    Sheng Ye, Yu-Hui Wen, Yanan Sun, Ying He, Ziyang Zhang, Yaoyuan Wang, Weihua He, and Yong-Jin Liu. Audio-driven stylized gesture generation with flow- based model. In European Conference on Computer Vi- sion, pages 712–728. Springer, 2022. 6

    2022

  45. [53]

    Generating holistic 3d human motion from speech

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yan- dong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. In CVPR, 2023. 3

    2023

  46. [54]

    Generating holistic 3d human motion from speech

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yan- dong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 469– 480, 2023. 6

    2023

  47. [55]

    Speech gesture generation from the trimodal context of text, au- dio, and speaker identity.ACM Transactions on Graphics (TOG), 39(6):1–16, 2020

    Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech gesture generation from the trimodal context of text, au- dio, and speaker identity.ACM Transactions on Graphics (TOG), 39(6):1–16, 2020. 6

    2020

  48. [56]

    Kinmo: Kinematic-aware human motion understanding and generation

    Pengfei Zhang, Pinxin Liu, Hyeongwoo Kim, Pablo Gar- rido, and Bindita Chaudhuri. Kinmo: Kinematic-aware human motion understanding and generation. arXiv preprint arXiv:2411.15472, 2024. 3

    2024 arXiv

  49. [57]

    Semantic gesticulator: Semantics-aware co-speech gesture synthe- sis

    Zeyi Zhang, Tenglong Ao, Yuyao Zhang, Qingzhe Gao, Chuan Lin, Baoquan Chen, and Libin Liu. Semantic gesticulator: Semantics-aware co-speech gesture synthe- sis. ACM Transactions on Graphics (TOG), 43(4):1–17,

  50. [58]

    Livelyspeaker: Towards semantic-aware co-speech gesture generation

    Yihao Zhi, Xiaodong Cun, Xuelin Chen, Xi Shen, Wen Guo, Shaoli Huang, and Shenghua Gao. Livelyspeaker: Towards semantic-aware co-speech gesture generation. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 20807–20817, 2023. 2, 3, 6

    2023

  51. [59]

    Taming diffusion models for audio- driven co-speech gesture generation

    Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for audio- driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10544–10553, 2023. 2, 3, 6

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.