Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

It Takes Two: Real-time Co-Speech Two-person's Interaction Generation via Reactive Auto-regressive Diffusion Model

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims the first online system that synthesizes interactive two-person full-body motion from speech in real time.

desk verdict Genuinely new task and a solid system plus dataset, but the 'real-time online' claim does not survive contact with the model's own future-speech condition. read the letter →

arxiv 2412.02419 v1 pith:XQZNLJSQ submitted 2024-12-03 cs.SD cs.CVcs.GRcs.MMeess.AS

Mingyi Shi , Dafei Qin , Leo Ho , Zhouyingcheng Liao , Yinghao Huang , Junichi Yamagishi , Taku Komura This is my paper · ORCID
classification cs.SDcs.CVcs.GRcs.MMeess.AS
keywords co-speechgesturesynthesistwo-personinteractiongenerationautoregressivediffusionmodelreal-timemotionspeech-drivenanimationtrajectorycontrolconversationaldatasetfull-body
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to make conversational avatars move the way people actually move in dialogue: each person's body responds to their own speech, the partner's speech, and the partner's current motion, while also walking, turning, hugging, handing objects, and performing other dynamic acts. It claims to be the first system that can do this online, in real time, rather than offline after the full conversation is known. The generator is an autoregressive diffusion model that consumes the recent motion of both characters, future speech features from both, and a predicted 2D root trajectory, producing each 45-frame window of two-person motion in roughly 8 ms (over 100 fps). To make the task learnable, the authors extend an existing two-person conversational dataset with 402 new clips covering hugging, handshaking, waving, grabbing, patting, and high-fives. If the claims hold, live virtual agents, games, and VR characters could hold natural and spatially active conversations instead of standing still and gesturing.

What carries the argument

The engine is a conditional denoising diffusion transformer with separated condition tokens. For each sliding window it takes past self motion, the partner's past motion, future speech features (semantic tokens from a pretrained speech tokenizer, Mel-spectrogram acoustics, and rhythm), and a future 2D root trajectory (body positions plus facing directions) for both characters, and denoises the next 45 frames of two-person motion in one pass. Training uses a random mask on the partner's motion so the model can also learn from single-person co-speech data, and inference applies a classifier-free-guidance-style interpolation between conditional and unconditional predictions to tune how strongly a character follows the partner. Alternating which character is root-normalized preserves relative position between the two bodies. Long sequences are stabilized by blending the input trajectory with a self-predicted trajectory and by cross-fading overlapping clip boundaries.

What would settle it

Measure the trajectory predictor's root-position and facing-direction error against held-out ground-truth motion for the same two-speaker audio: if the average endpoint error is large relative to the distance needed for a handshake or object handoff, then the interactive realism cannot be attributed to the planned trajectory. A second check is to inspect whether contact actions such as handshakes actually produce touching hands in generated clips, since the authors state that no explicit joint-position constraint is used.

Watch

Extended reading notes

Core claim

The central claim is that two-person interactive full-body motion can be generated autoregressively from two speech streams in real time, with each character reacting to the other's audio and motion while following a planned root trajectory. The authors report that the full model outperforms single-person co-speech baselines and two-person interaction baselines on a benchmark built from the extended dataset, and that the reactive autoregressive design is what makes the difference: removing the reference-person condition degrades interaction quality sharply. They also introduce InterAct++, an enriched dataset of dynamic conversational interactions, and show that their system can be trained on single-person data through a random masking scheme that behaves like classifier-free guidance. The stated novelty is the combination: real-time speed, autoregressive reactivity, two-character full-body output, and trajectory controllability in one system, which no prior work achieves together.

Load-bearing premise

The load-bearing premise is that the trajectory predictor can reliably anticipate each speaker's future root path and facing direction from speech and context alone, because the full-body diffusion generator is conditioned on those predicted trajectories even though trajectory accuracy is not measured in the main evaluation.

Editorial extensions

If this is right

  • Because each 45-frame window is generated in about 8 ms, a deployed system can animate a conversation as it happens, at over 100 fps.
  • Because generation is conditioned on a planned 2D trajectory, an animator or game engine can steer where characters walk and face while speech drives their gestures.
  • The random masking of the partner's motion lets the same model train on single-person co-speech datasets, so the approach can absorb large existing corpora beyond the collected two-person data.
  • The new InterAct++ clips provide test material for dynamic actions such as hugging, handshaking, waving, grabbing, patting, and high-fives that earlier two-person conversation datasets lacked.
  • Online applications such as VR, games, and customer-service avatars stand to gain two characters that converse and move together in real time rather than standing and gesturing in place.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because trajectory accuracy is never reported as a standalone number, an immediate next evaluation would be to measure endpoint and heading error against ground truth; I would expect this to be the main factor separating success from failure on unseen conversations.
  • The condition channels are generic, so the dual-stream design could in principle be extended to three or more characters by adding more partner tokens, but the paper does not test this.
  • The random-mask guidance weight already interpolates between fully reactive and fully autonomous behavior, which suggests a user-facing slider for how strongly a character follows its partner; the paper does not explore this interaction-control dimension.
  • The authors' own limitation about handshaking implies that contact-critical motions need an explicit geometric constraint; adding a contact-aware loss or post-contact correction is a natural testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper introduces an audio-driven, autoregressive diffusion framework for generating full-body motions of two interacting characters from speech, together with a new dataset (InterAct++) that extends the existing InterAct dataset with more dynamic interactions. The method predicts root trajectories from speech and then generates two-person motion conditioned on past and future speech, past motion, and trajectories. The authors report state-of-the-art results on single-person co-speech generation (BEAT), a two-person speech-to-motion benchmark, and an interaction generation task, and they claim real-time online performance with 8 ms per motion clip. The manuscript includes ablations, a user study, and a plan to release code and data.

Significance. If the real-time online claim were correct, this would be a significant advance for interactive character control and conversation animation, and the InterAct++ dataset is a useful addition. The two-stage trajectory-to-full-body design and the separate token conditioning are sensible, and the experiments cover standard metrics and ablations. However, the paper's headline contribution is undermined by the use of ground-truth future speech, which makes the pipeline non-causal; the trajectory module is also not evaluated directly. These issues prevent the claims as stated from being accepted.

major comments (3)
  1. [Sec. 3.2.1, Eq. (2), Sec. 3.3] The claimed online/real-time capability is not supported by the model definition. Equation (2) conditions the generator on future speech features s and s_refer for the entire 35-frame future window (N=45 frames with 10 past frames, Sec. 3.3). At 30 FPS this requires roughly 1.17 seconds of audio to be buffered before a motion clip can be produced, so the system is non-causal. The acknowledgment in Sec. 3.2.1 that the future information is not predicted but used as ground truth ('We will leave a model that predicts such information for future work') confirms that this is an architectural requirement, not an implementation detail. The 8 ms per-clip inference time reported in Sec. 3.3 omits this mandatory lookahead as well as the costs of SpeechTokenizer and trajectory prediction. Consequently, the abstract's 'online manner', the title's 'Real-time', and the 'reactive' characterization are not justified. I recommend either removing these claims and repositioning the work as an offline future-aware generation system, or introducing and evaluating a causal variant that uses only past speech.
  2. [Sec. 3.1 / D.1 / Table 5] The trajectory prediction module G_traj is a load-bearing component: the full-body diffusion model is conditioned on predicted trajectories (Sec. 3.1 and D.1), and the ablation 'w/o traj.' (Table 5) shows FPD degrading from 47.74 to 103.49 and FDD from 117.88 to 181.95 when trajectory conditioning is removed. However, the main experiments never report a direct evaluation of trajectory prediction accuracy (e.g., root position error, facing-direction error, path divergence) on held-out conversations, nor do they analyze how trajectory errors accumulate over autoregressive windows. Without such an evaluation, the claimed interactive quality and the practical usability of the fully automatic pipeline are not fully established.
  3. [Sec. 3.3] The 'Real-time Performance' paragraph reports only the motion generation module's inference time (8 ms per clip). A real-time claim requires an end-to-end latency measurement that includes audio tokenization (SpeechTokenizer), trajectory prediction, the autoregressive context update, and the blending operations described in Sec. 3.3 and D.4. Please provide a complete latency breakdown on the evaluation hardware, and specify the total latency from audio input to rendered motion output, including the buffering requirement of the future-speech conditioning.
minor comments (4)
  1. [Sec. 5.2 / Fig. 5] Audio2Photoreal is cited as [41] in Sec. 5.2 and the caption of Fig. 5, but reference [41] is 'Audio2Gestures' (Li et al.); the correct citation is [52] (Ng et al.). Please fix.
  2. [Table 5] The abbreviation 'SCT' is used without definition; please define 'Separated Conditional Tokens' when first used in Sec. 3.2.1 or in the table caption.
  3. [Table 3] The row label 'IntetAct' should be 'InterAct'.
  4. [Sec. 3.2.1, Eq. (2)] Equation (2) uses G_m while the preceding text and Eq. (1) use G; please make the notation consistent and clarify whether the subscript denotes the masked/modal variant.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the main claims rest on held-out evaluations and independent modules, though the work leans on several self-citations for architecture and dataset.

full rationale

The paper is an empirical machine-learning system rather than a derivation from first principles. Its central claims—autoregressive two-person co-speech motion generation, trajectory-guided synthesis, and improved quality over baselines—are supported by held-out test splits (Sec. 5.1: "The dataset is split into training and test sets with a ratio of 8:2 with their original split") and compared against external baselines such as LDA, EMAGE, AMUSE, Audio2Photoreal, ReMoS, and InterAct. The trajectory predictor G_traj is a separately trained module (Sec. 3.1 and D.1), and the main motion generator is conditioned on it rather than fitted to reproduce the reported metrics. The reported FPD/FDD improvements in Tables 2-5 are measured under a common protocol, not defined into existence. There are self-citations with author overlap: the 8-step diffusion setup and HFTE trajectory extension come from the authors' own [11], the InterAct dataset [29] is by overlapping authors and also serves as the closest baseline, and the enriched InterAct++ dataset is collected by the authors. However, these are used as engineering choices and comparison data, not as unverified premises that force the central conclusion. The real-time claim is undermined by the acknowledged use of future speech in Eq. (2) ("Future speech features s"; "Using the future information may sound unrealistic for interactive applications... We will leave a model that predicts such information for future work"), and the 8ms figure omits the required lookahead and preprocessing. That is a correctness/feasibility problem, not circularity: the model's outputs are not equivalent to its inputs by construction, and no prediction is statistically forced from a fitted subset. Therefore the circularity score is low.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical; no new physical entities are postulated. The system depends on several hand-set hyperparameters and domain assumptions about the sufficiency of speech, trajectory, and partner-motion conditioning for generating interactive two-person motion.

free parameters (6)
  • CFG guidance scale gamma = not reported
    Equation (3) uses a hyperparameter gamma to balance conditional and unconditional generation; the main text does not state the value used in experiments.
  • Trajectory blending weight alpha = not reported
    Equation (6) blends the input trajectory with a self-predicted trajectory; alpha is scenario-dependent and its value is not fixed in the paper.
  • Loss weights lambda_pos and lambda_vel = 0.2 and 0.5
    Reported in Sec. 3.2.3, Equation (4); chosen by hand.
  • Reference masking probability = 0.15
    Probability of masking the partner's motion during training (Sec. 3.2.2); chosen without ablation.
  • Window size and overlap = N=45, 10 past frames
    Autoregressive window configuration in Sec. 3.3; not swept.
  • Diffusion steps = 8
    Number of denoising steps for real-time performance (Sec. 3.3); trade-off with quality not analyzed.
assumptions (5)
  • standard math The diffusion forward process and learned reverse process can represent the conditional distribution of two-person motion.
    Invoked in Sec. 3.2.1 following Ho et al. [24].
  • domain assumption Speech-derived semantic (discrete token), melodic, and rhythmic features, together with trajectory and partner motion, are a sufficient conditioning set for generating conversational interaction.
    The entire generator design in Sec. 3.2.1 assumes these inputs determine the output motion; no analysis of missing conditioning information is given.
  • domain assumption The InterAct++ dataset, after retargeting, is representative of the test scenarios and supports generalization.
    The paper uses an 8:2 split of a 1.7-hour dataset (Sec. 5.1) and claims generalization, but the limited scale is acknowledged in the Limitations.
  • domain assumption The first dimension of SpeechTokenizer tokens preserves the semantic content needed for gesture alignment.
    Sec. 3.1 and supplementary C justify this with a t-SNE figure but no quantitative evaluation of downstream gesture quality.
  • domain assumption Root trajectory on the ground plane is a sufficient control signal for full-body interaction.
    The two-stage design (trajectory then body) in Sec. 3.1 rests on this; the large drop when trajectory is ablated (Table 5) shows its importance but also that it is a load-bearing assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of It Takes Two: Real-time Co-Speech Two-person's Interaction Generation via Reactive Auto-regressive Diffusion Model." pith.science (2026). https://pith.science/paper/XQZNLJSQ

@misc{pith2026241202419,
  author       = {Pith},
  title        = {Pith review of: It Takes Two: Real-time Co-Speech Two-person's Interaction Generation via Reactive Auto-regressive Diffusion Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQZNLJSQ}},
  note         = {Machine review of arXiv:2412.02419}
}
read the original abstract

Conversational scenarios are very common in real-world settings, yet existing co-speech motion synthesis approaches often fall short in these contexts, where one person's audio and gestures will influence the other's responses. Additionally, most existing methods rely on offline sequence-to-sequence frameworks, which are unsuitable for online applications. In this work, we introduce an audio-driven, auto-regressive system designed to synthesize dynamic movements for two characters during a conversation. At the core of our approach is a diffusion-based full-body motion synthesis model, which is conditioned on the past states of both characters, speech audio, and a task-oriented motion trajectory input, allowing for flexible spatial control. To enhance the model's ability to learn diverse interactions, we have enriched existing two-person conversational motion datasets with more dynamic and interactive motions. We evaluate our system through multiple experiments to show it outperforms across a variety of tasks, including single and two-person co-speech motion generation, as well as interactive motion generation. To the best of our knowledge, this is the first system capable of generating interactive full-body motions for two characters from speech in an online manner.

Figures

Figures reproduced from arXiv: 2412.02419 by the authors.

Figure 1
Figure 1. Our system addresses a novel task, that takes the speech of two persons as input to generate dynamic full-body interactions [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Concept diagram. Our system obtains two persons’ speech as input to generate full-body motion. We employ Large￾Speech-Model (LSM) to extract the semantic token, which are then fed into our autoregressive motion generation module, to pro￾duce interactive motion with the guidance of predicted trajectory. speech motion generation system for two persons, capable of producing the full-body motion of two characters in syn… view at source ↗
Figure 3
Figure 3. The overview of our autoregressive motion generator. Through a dual streaming design, the motion of two persons are generated [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Our generator delivers realistic interaction between two [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The qualitative comparison among various co-speech methods. We use a consistent SMPL-X representation for the mesh render [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: A pair of actors wearing full capture outfits, recording [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 8
Figure 8. Figure 8: Left: The unified skeleton in our experiments. Right: [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 7
Figure 7. Figure 7: We extract the speech semantic features besed on [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 9
Figure 9. Figure 9: Our trajectory prediction system takes multiple condi [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: The training loss and validation loss of our model. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MotionPersona: Characteristics-aware Locomotion Control

    cs.GR 2025-05 conditional novelty 6.0 of 10

    A single diffusion-based controller generates real-time character locomotion conditioned on body shape, text-described traits, and user control, plus a few-shot personalization mode.

Reference graph

Works this paper leans on

78 extracted references · 68 canonical work pages · cited by 1 Pith paper

  1. [1]

    Style transfer for co-speech gesture animation: A multi-speaker conditional-mixture approach

    Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Philippe Morency. Style transfer for co-speech gesture animation: A multi-speaker conditional-mixture approach. In European Conference on Computer Vision , pages 248–

    work page

  2. [2]

    Style-controllable speech-driven gesture synthesis using normalising flows

    Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow. Style-controllable speech-driven gesture synthesis using normalising flows. In Computer Graphics Forum, pages 487–496. Wiley Online Library, 2020. 3

    work page 2020

  3. [3]

    Listen, denoise, action! audio-driven motion synthesis with diffusion models

    Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–20, 2023. 2, 3, 4, 7, 8, 1

    work page 2023

  4. [4]

    Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings

    Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings. ACM Transactions on Graphics (TOG), 41(6):1–19, 2022. 3

    work page 2022

  5. [5]

    Gesturediffuclip: Gesture diffusion model with clip latents.ACM Transactions on Graphics (TOG), 42(4):1–18, 2023

    Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents.ACM Transactions on Graphics (TOG), 42(4):1–18, 2023. 2, 3

    work page 2023

  6. [6]

    Gesturediffuclip: Gesture diffusion model with clip latents.ACM Transactions on Graphics (TOG), 42(4):1–18, 2023

    Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents.ACM Transactions on Graphics (TOG), 42(4):1–18, 2023. 2, 4

    work page 2023

  7. [7]

    Speech2affectivegestures: Synthesizing co-speech ges- tures with generative adversarial affective expression learning

    Uttaran Bhattacharya, Elizabeth Childs, Nicholas Rewkowski, and Dinesh Manocha. Speech2affectivegestures: Synthesizing co-speech ges- tures with generative adversarial affective expression learning. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2027–2036, 2021. 3

    work page 2027

  8. [8]

    Digital life project: Autonomous 3d characters with social intelligence

    Zhongang Cai, Jianping Jiang, Zhongfei Qing, Xinying Guo, Mingyuan Zhang, Zhengyu Lin, Haiyi Mei, Chen Wei, Ruisi Wang, Wanqi Yin, et al. Digital life project: Autonomous 3d characters with social intelligence. 2024. 3

    work page 2024

Show all 78 references
  1. [9]

    Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information

    Justine Cassell, David McNeill, and Karl-Erik McCullough. Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information. Pragmatics & cognition, 7(1):1–34, 1999. 3

    1999

  2. [10]

    Beat: the behavior expression animation toolkit

    Justine Cassell, Hannes H ¨ogni Vilhj´almsson, and Timothy Bickmore. Beat: the behavior expression animation toolkit. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques , pages 477–486, 2001. 2, 3

    2001

  3. [11]

    Taming diffusion probabilistic mod- els for character control

    Rui Chen, Mingyi Shi, Shaoli Huang, Ping Tan, Taku Ko- mura, and Xuelin Chen. Taming diffusion probabilistic mod- els for character control. In SIGGRAPH, 2024. 4, 6

    2024

  4. [12]

    Executing your commands via motion diffusion in latent space

    Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18000–18010, 2023. 3

    2023

  5. [13]

    Black, and Timo Bolkart

    Kiran Chhatre, Radek Dan??ek, Nikos Athanasiou, Giorgio Becherini, Christopher Peters, Michael J. Black, and Timo Bolkart. Emotional speech-driven 3d body animation via dis- entangled latent diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    1942

  6. [14]

    The interplay between gesture and speech in the production of referring expressions: Investigating the tradeoff hypothesis

    Jan P De Ruiter, Adrian Bangerter, and Paula Dings. The interplay between gesture and speech in the production of referring expressions: Investigating the tradeoff hypothesis. Topics in cognitive science, 4(2):232–248, 2012. 3

    2012

  7. [15]

    Diffusion-based co-speech gesture genera- tion using joint text and audio representation

    Anna Deichler, Shivam Mehta, Simon Alexanderson, and Jonas Beskow. Diffusion-based co-speech gesture genera- tion using joint text and audio representation. InProceedings of the 25th International Conference on Multimodal Interac- tion, pages 755–762, 2023. 3

    2023

  8. [16]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 4

    2018 arXiv

  9. [17]

    Freemotion: A unified framework for number- free text-to-motion synthesis

    Ke Fan, Junshu Tang, Weijian Cao, Ran Yi, Moran Li, Jingyu Gong, Jiangning Zhang, Yabiao Wang, Chengjie Wang, and Lizhuang Ma. Freemotion: A unified framework for number- free text-to-motion synthesis. In Computer Vision – ECCV 2024, pages 93–109, Cham, 2025. Springer Nature S...

    2024

  10. [18]

    Troje, and Marc-Andr ´e Carbonneau

    Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F. Troje, and Marc-Andr ´e Carbonneau. Zeroeggs: Zero-shot example-based gesture generation from speech. Computer Graphics Forum, 42(1):206–216, 2023. 5

    2023

  11. [19]

    Remos: 3d motion- conditioned reaction synthesis for two-person interactions

    Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Chris- tian Theobalt, and Philipp Slusallek. Remos: 3d motion- conditioned reaction synthesis for two-person interactions. In European Conference on Computer Vision (ECCV), 2024. 3, 6, 8, 1

    2024

  12. [20]

    Interaction mix and match: Synthesizing close interaction using condi- tional hierarchical gan with multi-hot class embedding

    Aman Goel, Qianhui Men, and Edmond SL Ho. Interaction mix and match: Synthesizing close interaction using condi- tional hierarchical gan with multi-hot class embedding. In Computer Graphics Forum , pages 327–338. Wiley Online Library, 2022. 3

    2022

  13. [21]

    Learning speech-driven 3d conversational gestures from video

    Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed El- gharib, and Christian Theobalt. Learning speech-driven 3d conversational gestures from video. In Proceedings of the 21st ACM International Conference on Intelligent Virt...

    2021

  14. [22]

    Evaluation of speech-to-gesture generation using bi-directional lstm network

    Dai Hasegawa, Naoshi Kaneko, Shinichi Shirakawa, Hiroshi Sakuta, and Kazuhiko Sumi. Evaluation of speech-to-gesture generation using bi-directional lstm network. InProceedings of the 18th International Conference on Intelligent Virtual Agents, pages 79–86, 2018. 3

    2018

  15. [23]

    Moglow: Probabilistic and controllable motion synthesis using normalising flows

    Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Transactions on Graphics (TOG), 39(6):1–14, 2020. 2

    2020

  16. [24]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In NeurIPS, pages 6840–6851. Curran Associates, Inc., 2020. 4

    2020

  17. [25]

    Dead blending, 2023

    Daniel Holden. Dead blending, 2023. 6, 4

    2023

  18. [26]

    Phase- functioned neural networks for character control

    Daniel Holden, Taku Komura, and Jun Saito. Phase- functioned neural networks for character control. 36(4):1– 13, 2017. 4

    2017

  19. [27]

    Example- based control of human motion

    Eugene Hsu, Sommer Gentry, and Jovan Popovi ´c. Example- based control of human motion. In Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Com- puter animation, pages 69–77, 2004. 3

    2004

  20. [28]

    Robot behavior toolkit: generating effective social behaviors for robots

    Chien-Ming Huang and Bilge Mutlu. Robot behavior toolkit: generating effective social behaviors for robots. In Proceed- ings of the seventh annual ACM/IEEE international confer- ence on Human-Robot Interaction, pages 25–32, 2012. 3

    2012

  21. [29]

    Interact: Capture and modelling of realistic, ex- pressive and interactive activities between two persons in daily scenarios

    Yinghao Huang, Leo Ho, Dafei Qin, Mingyi Shi, and Taku Komura. Interact: Capture and modelling of realistic, ex- pressive and interactive activities between two persons in daily scenarios. arXiv preprint arXiv:, 2024. 2, 3, 6, 7

    2024

  22. [30]

    Intermask: 3d human interaction generation via collaborative masked modelling, 2024

    Muhammad Gohar Javed, Chuan Guo, Li Cheng, and Xingyu Li. Intermask: 3d human interaction generation via collaborative masked modelling, 2024. 3

    2024

  23. [31]

    Synchronized multi-character motion editing

    Manmyung Kim, Kyunglyul Hyun, Jongmin Kim, and Jehee Lee. Synchronized multi-character motion editing. ACM transactions on graphics (TOG), 28(3):1–9, 2009. 3

    2009

  24. [32]

    Tiling motion patches

    Manmyung Kim, Youngseok Hwang, Kyunglyul Hyun, and Jehee Lee. Tiling motion patches. InProceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Anima- tion, pages 117–126, 2012. 3

    2012

  25. [33]

    Towards a common framework for multimodal generation: The behavior markup language

    Stefan Kopp, Brigitte Krenn, Stacy Marsella, Andrew N Marshall, Catherine Pelachaud, Hannes Pirker, Kristinn R Th´orisson, and Hannes Vilhj ´almsson. Towards a common framework for multimodal generation: The behavior markup language. In Intelligent Virtual Agents: 6th Internat...

    2006

  26. [34]

    Gesticulator: A framework for semantically-aware speech-driven gesture generation

    Taras Kucherenko, Patrik Jonell, Sanne Van Waveren, Gustav Eje Henter, Simon Alexandersson, Iolanda Leite, and Hedvig Kjellstr ¨om. Gesticulator: A framework for semantically-aware speech-driven gesture generation. In Proceedings of the 2020 international conference on mul- ti...

    2020

  27. [35]

    The genea challenge 2023: A large- scale evaluation of gesture generation models in monadic and dyadic settings

    Taras Kucherenko, Rajmund Nagy, Youngwoo Yoon, Jieyeon Woo, Teodor Nikolov, Mihail Tsakov, and Gus- tav Eje Henter. The genea challenge 2023: A large- scale evaluation of gesture generation models in monadic and dyadic settings. In Proceedings of the 25th Interna- tional Confe...

    2023

  28. [36]

    Evaluating gesture generation in a large-scale open challenge: The genea challenge 2022

    Taras Kucherenko*, Pieter Wolfert*, Youngwoo Yoon*, Carla Viegas, Teodor Nikolov, Mihail Tsakov, and Gus- tav Eje Henter. Evaluating gesture generation in a large-scale open challenge: The genea challenge 2022. ACM Transac- tions on Graphics, 43(3):1–28, 2024. 3

    2022

  29. [37]

    Cross-conditioned recurrent networks for long- term synthesis of inter-person human motion interactions

    Jogendra Nath Kundu, Himanshu Buckchash, Priyanka Mandikal, Anirudh Jamkhandi, Venkatesh Babu Radhakrish- nan, et al. Cross-conditioned recurrent networks for long- term synthesis of inter-person human motion interactions. In Proceedings of the IEEE/CVF winter conference on ap...

    2020

  30. [38]

    Two-character motion analysis and synthesis

    Taesoo Kwon, Young-Sang Cho, Sang I Park, and Sung Yong Shin. Two-character motion analysis and synthesis. IEEE transactions on visualization and computer graphics, 14(3): 707–720, 2008. 3

    2008

  31. [39]

    Motion patches: building blocks for virtual environments annotated with motion data

    Kang Hoon Lee, Myung Geol Choi, and Jehee Lee. Motion patches: building blocks for virtual environments annotated with motion data. In ACM SIGGRAPH 2006 Papers, pages 898–906. 2006. 3

    2006

  32. [40]

    Gesture controllers

    Sergey Levine, Philipp Kr ¨ahenb¨uhl, Sebastian Thrun, and Vladlen Koltun. Gesture controllers. ACM Trans. Graph., 29(4):124:1–124:11, 2010. 3

    2010

  33. [41]

    Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders

    Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 112...

    2021

  34. [42]

    Ai choreographer: Music conditioned 3d dance generation with aist++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 13401– 13412, 2021. 7

    2021

  35. [43]

    Intergen: Diffusion-based multi-human motion genera- tion under complex interactions

    Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi-human motion genera- tion under complex interactions. International Journal of Computer Vision, pages 1–21, 2024. 3, 6

    2024

  36. [44]

    Com- position of complex optimal multi-character motions

    C Karen Liu, Aaron Hertzmann, and Zoran Popovi ´c. Com- position of complex optimal multi-character motions. InPro- ceedings of the 2006 ACM SIGGRAPH/Eurographics sym- posium on Computer animation, pages 215–222, 2006. 3

    2006

  37. [45]

    Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis

    Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. arXiv preprint arXiv:2203.05297, 2022. 5, 6, 7, 2

    2022 arXiv

  38. [46]

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J. Black. Emage: Towards unified holis- tic co-speech gesture generation via expressive masked audio gesture modeling. In 2024 IEEE/CVF Conference on...

    2024

  39. [47]

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J. Black. Emage: Towards unified holis- tic co-speech gesture generation via expressive masked audio gesture modeling. In 2024 IEEE/CVF Conference on...

    2024

  40. [48]

    Learning hierarchical cross-modal association for co- speech gesture generation

    Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co- speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2022

  41. [49]

    librosa: Audio and music signal analysis in python

    Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In SciPy, pages 18–24, 2015. 4

    2015

  42. [50]

    Gan-based reactive motion synthesis with class-aware discriminators for human–human interaction

    Qianhui Men, Hubert PH Shum, Edmond SL Ho, and Howard Leung. Gan-based reactive motion synthesis with class-aware discriminators for human–human interaction. Computers & Graphics, 102:634–645, 2022. 3

    2022

  43. [51]

    Gesture modeling and animation based on a proba- bilistic re-creation of speaker style

    Michael Neff, Michael Kipp, Irene Albrecht, and Hans-Peter Seidel. Gesture modeling and animation based on a proba- bilistic re-creation of speaker style. ACM Transactions On Graphics (TOG), 27(1):1–24, 2008. 3

    2008

  44. [52]

    From audio to photoreal embodiment: Synthesizing humans in conversations

    Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthesizing humans in conversations. In IEEE Conference on Computer Vision and Pattern Recognition, 2024. 2, 3, 7, 1

    2024

  45. [53]

    Glide: Towards photorealis- tic image generation and editing with text-guided diffusion models

    Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealis- tic image generation and editing with text-guided diffusion models. pages 16784–16804. PMLR, 2022. 4

    2022

  46. [54]

    A comprehensive re- view of data-driven co-speech gesture generation

    Simbarashe Nyatsanga, Taras Kucherenko, Chaitanya Ahuja, Gustav Eje Henter, and Michael Neff. A comprehensive re- view of data-driven co-speech gesture generation. In Com- puter Graphics Forum , pages 569–596. Wiley Online Li- brary, 2023. 3

    2023

  47. [55]

    Librispeech: An asr corpus based on public do- main audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public do- main audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, 2015. 2

    2015

  48. [56]

    Bodyformer: Semantics-guided 3d body gesture synthesis with transformer

    Kunkun Pang, Dafei Qin, Yingruo Fan, Julian Habekost, Takaaki Shiratori, Junichi Yamagishi, and Taku Komura. Bodyformer: Semantics-guided 3d body gesture synthesis with transformer. ACM Transactions on Graphics (TOG) , 42(4):1–12, 2023. 2, 3, 4

    2023

  49. [57]

    Do people use lan- guage production to make predictions during comprehen- sion? Trends in cognitive sciences , 11(3):105–110, 2007

    Martin J Pickering and Simon Garrod. Do people use lan- guage production to make predictions during comprehen- sion? Trends in cognitive sciences , 11(3):105–110, 2007. 4

    2007

  50. [58]

    Hierarchical text-conditional image gener- ation with clip latents, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents, 2022. 4

    2022

  51. [59]

    Human motion diffusion as a generative prior

    Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. In ICLR, 2024. 3

    2024

  52. [60]

    Interaction patches for multi-character animation

    Hubert PH Shum, Taku Komura, Masashi Shiraishi, and Shuntaro Yamazaki. Interaction patches for multi-character animation. ACM transactions on graphics (TOG), 27(5):1–8,

  53. [61]

    Duolando: Follower gpt with off-policy reinforcement learn- ing for dance accompaniment

    Li Siyao, Tianpei Gu, Zhitao Yang, Zhengyu Lin, Ziwei Liu, Henghui Ding, Lei Yang, and Chen Change Loy. Duolando: Follower gpt with off-policy reinforcement learn- ing for dance accompaniment. In ICLR, 2024. 3

    2024

  54. [62]

    Local motion phases for learning multi-contact charac- ter movements

    Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Za- man. Local motion phases for learning multi-contact charac- ter movements. 39(4):54–1, 2020. 4

    2020

  55. [63]

    Local motion phases for learning multi-contact charac- ter movements

    Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Za- man. Local motion phases for learning multi-contact charac- ter movements. ACM Transactions on Graphics (TOG), 39 (4):54–1, 2020. 3

    2020

  56. [64]

    Human motion diffu- sion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffu- sion model. In The Eleventh International Conference on Learning Representations, 2023. 3

    2023

  57. [65]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neu- ral Information Processing Systems. Curran Associates, Inc.,

  58. [66]

    Gesture and speech in interaction: An overview, 2014

    Petra Wagner, Zofia Malisz, and Stefan Kopp. Gesture and speech in interaction: An overview, 2014. 3

    2014

  59. [67]

    Generating and ranking diverse multi-character interactions

    Jungdam Won, Kyungho Lee, Carol O’Sullivan, Jessica K Hodgins, and Jehee Lee. Generating and ranking diverse multi-character interactions. ACM Transactions on Graph- ics (TOG), 33(6):1–12, 2014. 3

    2014

  60. [68]

    Eggesture: Entropy-guided vector quantized variational autoencoder for co-speech gesture generation

    Yiyong Xiao, Kai Shu, Haoyi Zhang, Baohua Yin, Wai Seng Cheang, Haoyang Wang, and Jiechao Gao. Eggesture: Entropy-guided vector quantized variational autoencoder for co-speech gesture generation. In Proceedings of the 32nd ACM International Conference on Multimedia , page 6113...

    2024

  61. [69]

    Speech ges- ture generation from the trimodal context of text, audio, and speaker identity

    Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech ges- ture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39 (6):1–16, 2020. 3

    2020

  62. [70]

    Motiondif- fuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022. 3

    2022 arXiv

  63. [71]

    Speechtokenizer: Unified speech tokenizer for speech language models, 2023

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech language models, 2023. 4, 2

    2023

  64. [72]

    Emdm: Efficient mo- tion diffusion model for fast, high-quality motion generation

    Wenyang Zhou, Zhiyang Dou, Zeyu Cao, Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient mo- tion diffusion model for fast, high-quality motion generation. arXiv preprint arXiv:2312.02256, 2023. 3 It Takes Two: Re...

    2023 arXiv

  65. [73]

    Two-person’s co-speech motion generation results,

  66. [74]

    Demonstration for the dataset,

  67. [75]

    Comparisons with AMUSE [13] and EMAGE [46] on single-person’s co-speech motion generation,

  68. [76]

    Comparisons with LDA [3] and Audio2Photoreal [52] on two-person conversational generation,

  69. [77]

    Comparisons with [19] on two-person interaction gener- ation,

  70. [78]

    Ablation comparisons of the two-person audio condi- tions and the trajectory conditions. B. Data Collection Content In InterAct++, 8 actors, consisting of 4 males and 4 females, were recruited for the capture. The actors performed the interactions in male-male or female-female...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.