Pith. sign in

REVIEW 4 major objections 5 minor 147 references

CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes coordinated diffusion noise optimization of three decoupled diffusion models to generate whole-body manipulation of articulated objects, with better motion quality and physical plausibility than adapted baselines on…

desk verdict First whole-body articulated-object manipulation generator with a plausible decoupled-diffusion recipe, but the coordination claim is not directly measured and the experimental evidence lacks error bars. read the letter →

arxiv 2505.21437 v1 pith:5ARANCGZ submitted 2025-05-27 cs.GR cs.CVcs.RO

Huaijin Pi , Zhi Cen , Zhiyang Dou , Taku Komura This is my paper
classification cs.GRcs.CVcs.RO
keywords whole-bodymotiongenerationhand-objectinteractionarticulatedobjectmanipulationdiffusionmodelsnoise-spaceoptimizationbasispointsetsSMPL-X
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that whole-body manipulation of articulated objects—body, both hands, and the moving object—can be generated as a single coordinated sequence by optimizing the input noise of three independently trained diffusion models, one for the body, one for the left hand, and one for the right hand. Coordination emerges from gradient flow along the human kinematic chain: hand objectives pull the torso and limbs into the right posture, while body objectives keep the hands consistent with the object. The claim matters because it decouples data requirements: the body model is trained on large general motion corpora and the hand models on hand-specific manipulation datasets, so no large paired whole-body manipulation corpus is needed to generate such interactions. The paper reports that the framework, called CoDA, outperforms adapted baselines on motion quality, foot skating, and hand-object contact metrics on both the ARCTIC and GRAB benchmarks, and it enables object pose control, walking while manipulating, and whole-body generation from hand-only trajectories.

What carries the argument

The load-bearing mechanism is coordinated diffusion noise optimization over decoupled motion priors. Three diffusion models—body $M_b$, left hand $M_{lh}$, right hand $M_{rh}$—are trained on separate marginal datasets, and inference solves $z^* = \arg\min_z L(f(z))$, where $f(z)$ maps the concatenated noise through the diffusion models and forward kinematics to global joint positions; the loss $L = \lambda_{ee} L_{ee} + \lambda_{pen} L_{pen} + \lambda_{reg} L_{reg}$ enforces end-effector tracking, penetration avoidance, and foot contact. The second supporting object is the shared basis point set (BPS) representation, which encodes both object geometry and the 12 end-effectors (wrists and fingertips) as distances to the same basis points, so the trajectory targets live in the same reference frame as the object. Coordinated body adaptation is attributed to gradients flowing from hand objectives through the kinematic chain during optimization.

What would settle it

A decisive test is to train the body model only on locomotion data and the hand models only on grasping data from a different object set, then run the full pipeline; if end-effector tracking accuracy and contact ratio remain high, the optimization alone creates coordination, and if they collapse, the coordinated manifold must already exist in the training data.

Watch

Extended reading notes

Core claim

CoDA's central claim is that joint body-hand-object motion for articulated manipulation can be synthesized by first generating an object trajectory and end-effector trajectories, then solving for the diffusion noise vectors of three decoupled models so that the denoised poses track those trajectories. The end-effector stage uses a unified basis point set (BPS) representation: fingertip and wrist positions are encoded as distances to the same basis points that encode the object geometry, which puts the interaction targets in the object's own reference frame and improves precision. The whole-body stage optimizes $z=\{z_b, z_{lh}, z_{rh}\}$ through a loss combining end-effector tracking, hand-object penetration, and foot-contact regularization, with DDIM sampling used so gradients can flow back through the solver. On ARCTIC, CoDA reports FID 2.283 versus 3.710 for the best adapted baseline, interpenetration volume 5.25, and contact ratio 0.086, close to the real-motion value of 0.085; user-study participants preferred it for realism and physical plausibility at rates around 88 percent. The authors state this is the first work to jointly generate body, hand, and articulated object motion for whole-body manipulation.

Load-bearing premise

The load-bearing premise is that separately trained body and hand priors, when pushed together by noise-space optimization, can land on plausible joint whole-body manipulation rather than on merely individually plausible parts.

Editorial extensions

If this is right

  • If the method holds up, whole-body manipulation can be synthesized without a large paired dataset: the body prior comes from general motion capture and the hand priors from hand-only interaction data, so novel combinations of body and finger motion become possible.
  • Users can control the object's pose at keyframes, shift the root target to make the character walk while manipulating, and feed wrist and fingertip trajectories from hand-only recordings to obtain full-body motion.
  • The generated sequences are accurate enough that the framework can output reference motion for physics-based humanoids, which then physically manipulate the object in simulation.
  • On the ARCTIC and GRAB benchmarks, the approach improves FID, foot skating, interpenetration volume and depth, and contact ratio over the adapted baselines considered.
  • The authors state this is the first method to jointly generate body, hand, and articulated object motion for whole-body manipulation tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because coordination is produced by optimization through the kinematic chain rather than by joint training, the same scheme could plausibly extend beyond three modules, for example adding a separate object-motion diffusion model or a face and expression module, provided the objectives remain differentiable.
  • A testable consequence is that the benefit of the decoupled design should grow as the body and hand training distributions diverge; an ablation that varies the amount of paired whole-body data during training would reveal how much coordination the optimization actually buys.
  • The 16.9-minute optimization per 300-frame sequence suggests the practical path to real-time use is distilling the optimized-noise behavior into a feed-forward conditional model, with the current pipeline serving as the teacher.
  • Because the end-effector targets are distances to a shared basis point set, the trajectory stage may transfer across object geometries and categories more easily than the motion stage; evaluating the same trajectory models on unseen object categories would test this directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CoDA, a three-stage framework for synthesizing whole-body manipulation of articulated objects. Given an initial human/object pose and text, it first generates an articulated object trajectory, then predicts end-effector trajectories using a unified basis-point-set (BPS) representation, and finally produces whole-body motion by optimizing the input noise of three independently trained diffusion models for the body, left hand, and right hand. The optimization losses track end-effector positions, penalize hand-object penetration, and regularize foot contact. Experiments on ARCTIC and GRAB report state-of-the-art FID, physical-plausibility, and user-study results, with ablations and demonstrations of object-pose control, walking while manipulating, and whole-body generation from hand-only data.

Significance. If validated, the paper makes a significant contribution: it is the first to jointly generate body, hand, and articulated-object motion for whole-body manipulation, and the decoupled diffusion design is a practical way to leverage large body-only and hand-only datasets. The BPS-based end-effector representation is a clean idea that ties trajectory prediction to object geometry, and the noise-space optimization over multiple models is a meaningful extension of DNO. However, the central empirical claims currently rest on aggregate metrics without error bars, an under-specified baseline extension protocol, a small user study, and no direct test of the joint plausibility of the assembled body-plus-hand poses.

major comments (4)
  1. [§4.3 (Eqs. 7-12) and §5.4 (Table 4)] The load-bearing assumption—that optimizing over the product of three independently trained marginal diffusion priors, using only the end-effector, penetration, and regularization losses, lands on the joint whole-body manipulation manifold—is not directly tested. None of the terms in Eq. (10), Eq. (11), or Eq. (12) evaluates the joint plausibility of the assembled body-plus-hand pose, and the body and hand priors share no training signal at the wrist interface. Ablation (e) shows that a single whole-body model performs worse (FID 2.699 vs 2.283, IV 12.45 vs 5.25), but this compares two architectures under the same optimization and does not verify that the product-of-marginals prior is faithful. I request a direct interface-consistency evaluation (e.g., wrist/palm orientation consistency with finger contacts, or distance of assembled poses to the real whole-body manifold) so that the 'coordination naturally emerges' claim is supported by a metric rather than by aggregate FID and a 16-participant study.
  2. [§5.3 and Tables 1 and 3] The state-of-the-art comparison is not fully reproducible because the protocol for extending IMoS, MDM, OMOMO, Text2HOI, and CHOIS to the whole-body articulated-object task is not described. It is not specified whether the same object-motion and end-effector generation models are used for all baselines, how the hand components are added, or which training data are used. In addition, no error bars, standard deviations, or significance tests are reported for any metric; since the sample sizes are not stated, the reader cannot assess the stability of the reported differences (e.g., FID 2.283 vs 3.710 for OMOMO on ARCTIC). Please report variances over multiple seeds or bootstrap intervals and describe the extension protocol in detail.
  3. [Table 2 and §5.2] The user study has 16 participants but no protocol details are given: the number of sequences evaluated, the number of pairwise comparisons per participant, the presentation order, the expertise of the participants, and the statistical test used are all missing. Since the claim that the method 'significantly outperforms' in perceptual quality rests on this table, please report the full protocol and a significance measure (e.g., a p-value or a confidence interval for the preference rates).
  4. [§D.2] The optimization schedule is hand-tuned with staged thresholds (300/500/800 steps) and loss weights λee=1, λpen=5.0, λreg=1.0, yet no sensitivity analysis is provided. The reported improvements in Table 4 depend on these choices; a small study varying the schedule and weights would establish that the method is not overfit to these specific settings.
minor comments (5)
  1. [Eq. (12)] The notation ∥min(Jy) − 0.02∥1 is ambiguous; please specify whether the minimum is over all joints, over each frame, or over the foot joints, and define the norm explicitly.
  2. [§2] There are minor punctuation errors, e.g., 'DiffH2O [13] applies ... interactions..' contains a double period.
  3. [§5.2] The R-Precision metric is not defined precisely; please state the retrieval procedure and the threshold used for Top-3 accuracy.
  4. [§4.2] The text says 'As shown in figure 3' but the figure is referenced elsewhere as 'Figure 3'; please use consistent capitalization.
  5. [§D.1] The phrase 'All diffusion models use 1000 sampling steps during training' conflates the number of diffusion steps in training with inference-time sampling; please rephrase to refer to the forward diffusion noise schedule.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline is trained on independent data and optimized against task losses, with self-citations confined to related work and implementation details.

full rationale

The derivation chain is self-contained. Object-trajectory generation (Sec. 4.1), end-effector BPS trajectory generation (Sec. 4.2), and whole-body noise-space optimization (Sec. 4.3) are each trained on external data (ARCTIC, GRAB, AMASS) under standard held-out evaluation protocols, and the inference-time loss L = lambda_ee*Lee + lambda_pen*Lpen + lambda_reg*Lreg is a genuine task objective: it tracks predicted end-effector positions, penalizes hand-object penetration, and regularizes foot contact. These losses are not re-fits of the evaluation metrics, and the FID, R-Precision, and user-study results are measured against held-out sequences. The DNO noise-optimization mechanism and the BPS representation are adopted from external prior work [40, 81, 132], not from a self-citation chain, and the ablations in Table 4 test each stage against reduced variants. The claim that coordination 'naturally emerges' is a design rationale: because the optimization back-propagates through differentiable forward kinematics, hand-tracking losses can update body noise; this is asserted rather than statistically proven, but that is a correctness-robustness concern, not circularity. Author self-citations appear in related-work lists and in the RoPE attention-window implementation detail (Sec. C, following [85]), but none is load-bearing for the central claim. No step was found where an output quantity equals a fitted input by construction, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The method relies on a set of hand-chosen hyperparameters (loss weights, thresholds, optimization schedule, basis point count, attention window) that are not all disclosed or ablated. The key domain assumptions concern the validity of combining independently trained diffusion priors and the recoverability of 3D positions from BPS distances. No new physical or conceptual entities are introduced.

free parameters (7)
  • loss weights λ_ee, λ_pen, λ_reg = 1, 5, 1 (with staged activation)
    Chosen by hand (Section 5.1 and D.2); affect the balance between trajectory tracking, penetration avoidance, and regularization; no sensitivity analysis provided.
  • penetration margin = 0.01 m
    Assumed finger thickness; Eq (11) penalizes fingertips closer than 1 cm to the object surface.
  • foot contact height threshold = 0.02 m
    Used in Eq (12) to determine binary foot-ground contact.
  • basis point set size K = not specified; visualization uses 64
    The number of basis points affects the precision of BPS descriptors and is not disclosed.
  • optimization schedule thresholds = 300 and 500 steps out of 800
    Staged activation of loss terms is hand-crafted (Section D.2) and not ablated.
  • DDIM steps for noise optimization = T=10
    Chosen following DNO; affects optimization quality and speed.
  • attention window = 120 frames
    Restricted to 120 neighboring frames (Section C); value chosen without ablations.
assumptions (5)
  • domain assumption SMPL-X is a differentiable, accurate parametric human body model (pose parameters produce a rigged mesh).
    Used throughout as the motion representation; Section 3 (Motion representations).
  • domain assumption Denoising diffusion training on motion capture data yields a valid generative prior for natural body and hand motion.
    Section 4.3; the entire optimization relies on the pretrained diffusion models staying on the natural motion manifold.
  • domain assumption Basis point set distances are an adequate and recoverable encoding of end-effector positions; minimizing Eq (2) yields accurate 3D trajectories.
    Section 4.2; no uniqueness or error bound is given for the non-convex distance reconstruction.
  • domain assumption Separately training body and hand diffusion models on different datasets and combining them via noise optimization covers the joint distribution of whole-body manipulation.
    Section 4.3; this is the main modeling bet and is not directly validated.
  • domain assumption CLIP text embeddings contain sufficient action semantics to condition object, end-effector, and whole-body generation.
    Section 4.1; all diffusion conditionings use CLIP features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects." pith.science (2026). https://pith.science/paper/5ARANCGZ

@misc{pith2026250521437,
  author       = {Pith},
  title        = {Pith review of: CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ARANCGZ}},
  note         = {Machine review of arXiv:2505.21437}
}
read the original abstract

Synthesizing whole-body manipulation of articulated objects, including body motion, hand motion, and object motion, is a critical yet challenging task with broad applications in virtual humans and robotics. The core challenges are twofold. First, achieving realistic whole-body motion requires tight coordination between the hands and the rest of the body, as their movements are interdependent during manipulation. Second, articulated object manipulation typically involves high degrees of freedom and demands higher precision, often requiring the fingers to be placed at specific regions to actuate movable parts. To address these challenges, we propose a novel coordinated diffusion noise optimization framework. Specifically, we perform noise-space optimization over three specialized diffusion models for the body, left hand, and right hand, each trained on its own motion dataset to improve generalization. Coordination naturally emerges through gradient flow along the human kinematic chain, allowing the global body posture to adapt in response to hand motion objectives with high fidelity. To further enhance precision in hand-object interaction, we adopt a unified representation based on basis point sets (BPS), where end-effector positions are encoded as distances to the same BPS used for object geometry. This unified representation captures fine-grained spatial relationships between the hand and articulated object parts, and the resulting trajectories serve as targets to guide the optimization of diffusion noise, producing highly accurate interaction motion. We conduct extensive experiments demonstrating that our method outperforms existing approaches in motion quality and physical plausibility, and enables various capabilities such as object pose control, simultaneous walking and manipulation, and whole-body generation from hand-only data.

Figures

Figures reproduced from arXiv: 2505.21437 by the authors.

Figure 1
Figure 1. Our approach enables: (a) generating whole-body manipulation of articulated objects from text input (e.g., “a person uses the mixer”); (b) manipulating the object to a target pose and articulation (the blue object is the target pose); (c) synthesizing whole-body motion guided by trajectories from hand-only data; (d) generating motions involving simultaneous walking and object manipulation (e.g., opening a box while … view at source ↗
Figure 2
Figure 2. Pipeline overview. (a) Given the initial human pose, object pose, and text, we first generate the articulated object trajectory and the corresponding end-effector trajectories via two conditional diffusion models. (b) We then optimize the latent noise inputs of three decoupled diffusion models by propagating gradients through the kinematic chain, guided by end-effector tracking, penetration, and regularization losse… view at source ↗
Figure 3
Figure 3. The illustration of the end￾effector BPS. (a) is the object BPS [132]. (b) is the proposed end-effector BPS represen￾tation. Gray points denote the basis points; pink/yellow are two object parts; blue indi￾cates a fingertip. Only one end-effector and 64 basis points are visualized for simplicity. Given the generated end-effector BPS sequence, we recover the end-effector trajectories by solving a sim￾ple optimization… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison. Given the text “A person uses the ketchup.”, our method generates the whole-body motion with better hand-object contact compared to baselines [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Generalization to different object geometry. We train the object motion and end-effector trajectory models on hand-only data [138] with different object geometries. These models are integrated into our framework to provide optimization targets, enabling realistic whole…
Figure 6
Figure 6. Figure 6: Object motion control. Our method could generate coherent whole-body motion with the object motion keyframe. The blue object indicates the object motion keyframe. a) Walk forward b) Walk backward c) Walk to the right d) Walk to the left [PITH_FULL_IMAGE:figures/full_f…
Figure 7
Figure 7. Figure 7: Simultaneous locomotion and manipulation. Our method enables the human to manipu￾late objects while simultaneously a) walking forward, b) walking backward, c) walking to the right, and d) walking to the left. The transparency of the meshes indicates time progression, w…
Figure 8
Figure 8. Figure 8: Deployment on simulated humanoids. We apply existing motion tracking techniques to deploy the generated motion to a simulated humanoid. The articulated object is physically manipulated by the humanoid within the physics simulator [64] [PITH_FULL_IMAGE:figures/full_fig…
Figure 9
Figure 9. Figure 9: Generating whole-body motion from hand-only dataset. We use the fingertip and object trajectories from the dataset and assign them as the optimization targets. After the optimization, we could get the whole-body motion. E.6 Inference speed [PITH_FULL_IMAGE:figures/ful…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

147 extracted references · 58 canonical work pages

  1. [1]

    Circle: Capture in rich contextual environments

    Joao Pedro Araújo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Jiajun Wu, Deepak Gopinath, Alexander William Clegg, and Karen Liu. Circle: Capture in rich contextual environments. In CVPR, 2023. 3

    2023

  2. [2]

    HOT3D: Hand and object tracking in 3D from egocentric multi-view videos

    Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. CVPR, 2025. 3, 8

    2025

  3. [3]

    Physi- cally plausible full-body hand-object interaction synthesis

    Jona Braun, Sammy Christen, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges. Physi- cally plausible full-body hand-object interaction synthesis. In International Conference on 3D Vision (3DV), 2024. 3

    2024

  4. [4]

    Generating human motion in 3d scenes from text descriptions

    Zhi Cen, Huaijin Pi, Sida Peng, Zehong Shen, Minghui Yang, Zhu Shuai, Hujun Bao, and Xiaowei Zhou. Generating human motion in 3d scenes from text descriptions. In CVPR, 2024. 2, 3

    2024

  5. [5]

    Ready-to-react: Online reaction policy for two-character interaction generation

    Zhi Cen, Huaijin Pi, Sida Peng, Qing Shuai, Yujun Shen, Hujun Bao, Xiaowei Zhou, and Ruizhen Hu. Ready-to-react: Online reaction policy for two-character interaction generation. In ICLR, 2025. 19

    2025

  6. [6]

    Text2hoi: Text-guided 3d motion generation for hand-object interaction

    Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1577–1585, 2024. 2, 3, 4, 7, 8, 24

    2024

  7. [7]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022. 19

    2022

  8. [8]

    The language of motion: Unifying verbal and non-verbal language of 3d human motion

    Changan Chen, Juze Zhang, Shrinidhi Kowshika Lakshmikanth, Yusu Fang, Ruizhi Shao, Gordon Wetzstein, Li Fei-Fei, and Ehsan Adeli. The language of motion: Unifying verbal and non-verbal language of 3d human motion. In arXiv, 2024. 19

    2024

Show all 147 references
  1. [9]

    Pay attention and move better: Harnessing attention for interactive motion generation and training-free editing

    Ling-Hao Chen, Shunlin Lu, Wenxun Dai, Zhiyang Dou, Xuan Ju, Jingbo Wang, Taku Komura, and Lei Zhang. Pay attention and move better: Harnessing attention for interactive motion generation and training-free editing. arXiv preprint arXiv:2410.18977, 2024. 19

    arXiv

  2. [10]

    Taming diffusion probabilistic models for character control

    Rui Chen, Mingyi Shi, Shaoli Huang, Ping Tan, Taku Komura, and Xuelin Chen. Taming diffusion probabilistic models for character control. In ACM SIGGRAPH 2024 Conference Papers, pages 1–10, 2024. 19

    2024

  3. [11]

    Executing your Commands via Motion Diffusion in Latent Space

    Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. Executing your Commands via Motion Diffusion in Latent Space. arXiv e-prints, art. arXiv:2212.04048, December 2022. doi: 10.48550/arXiv.2212.04048. 19

    2022 doi arXiv

  4. [12]

    D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions

    Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20577–2058...

    2022

  5. [13]

    Diffh2o: Diffusion-based synthesis of hand-object interactions from textual descriptions

    Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bugra Tekin. Diffh2o: Diffusion-based synthesis of hand-object interactions from textual descriptions. In SIGGRAPH Asia 2024 Conference Papers, 2024. 2, 3, 7

    2024

  6. [14]

    Laserhuman: Language-guided scene-aware human motion generation in free environment

    Peishan Cong, Ziyi Wang, Zhiyang Dou, Yiming Ren, Wei Yin, Kai Cheng, Yujing Sun, Xiaoxiao Long, Xinge Zhu, and Yuexin Ma. Laserhuman: Language-guided scene-aware human motion generation in free environment. arXiv preprint arXiv:2403.13307, 2024. 3

    2024 arXiv

  7. [15]

    Semgeomo: Dynamic con- textual human motion generation with semantic and geometric guidance

    Peishan Cong, Ziyi Wang, Yuexin Ma, and Xiangyu Yue. Semgeomo: Dynamic con- textual human motion generation with semantic and geometric guidance. arXiv preprint arXiv:2503.01291, 2025. 3 10

    2025 arXiv

  8. [16]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021. 3, 19

    2021

  9. [17]

    Cg-hoi: Contact-guided 3d human-object interaction genera- tion

    Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction genera- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19888–19901, 2024. 3

    2024

  10. [18]

    C· ase: Learning conditional adversarial skill embeddings for physics-based characters

    Zhiyang Dou, Xuelin Chen, Qingnan Fan, Taku Komura, and Wenping Wang. C· ase: Learning conditional adversarial skill embeddings for physics-based characters. In SIGGRAPH Asia 2023 Conference Papers, pages 1–11, 2023. 19

    2023

  11. [19]

    Black, and Otmar Hilliges

    Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2, 3,...

    2023

  12. [20]

    Imos: Intent-driven full-body motion synthesis for human-object interactions

    Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Imos: Intent-driven full-body motion synthesis for human-object interactions. In Computer Graphics Forum, 2023. 3, 4, 7, 8

    2023

  13. [21]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commun. ACM,

  14. [22]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 19

    2023 arXiv

  15. [23]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In CVPR, 2022. 7, 19

    2022

  16. [24]

    Momask: Gen- erative masked modeling of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Gen- erative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900–1910, 2024. 19

    1900

  17. [25]

    Robust motion in-betweening

    Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in-betweening. ACM Trans. Graph., 2020. 19

    2020

  18. [26]

    Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J. Black. Stochastic scene-aware motion prediction. In ICCV, 2021. 2, 3, 19

    2021

  19. [27]

    Synthesizing physical character-scene interactions

    Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scene interactions. In ACM SIGGRAPH 2023 Confer- ence Proceedings, pages 1–9, 2023. 3

    2023

  20. [28]

    Nemf: Neural motion fields for kinematic animation

    Chengan He, Jun Saito, James Zachary, Holly Rushmeier, and Yi Zhou. Nemf: Neural motion fields for kinematic animation. In NeurIPS, 2022. 3

    2022

  21. [29]

    Moglow: Probabilistic and controllable motion synthesis using normalising flows

    Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Trans. Graph., 2020. 19

    2020

  22. [30]

    Classifier-Free Diffusion Guidance

    Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. arXiv e-prints, art. arXiv:2207.12598, July 2022. 3, 19

    2022 arXiv

  23. [31]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 7, 19, 20

    2020

  24. [32]

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video Diffusion Models. arXiv e-prints, art. arXiv:2204.03458, April 2022. 19

    2022 arXiv

  25. [33]

    Phase-functioned neural networks for character control

    Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neural networks for character control. ACM Trans. Graph., 2017. 19 11

    2017

  26. [34]

    Hoigpt: Learning long sequence hand-object interaction with language models

    Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, Kris Kitani, Matt Feiszli, and Hao Tang. Hoigpt: Learning long sequence hand-object interaction with language models. In IEEE Conference on Comput...

    2025

  27. [35]

    Diffusion-based generation, optimization, and planning in 3d scenes

    Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chun Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. In CVPR, 2023. 3

    2023

  28. [36]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991. doi: 10.1162/neco.1991.3. 1.79. 19

    1991 doi

  29. [37]

    Motiongpt: Human motion as a foreign language

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067–20079,

  30. [38]

    Autonomous character-scene interaction synthesis from text instruction, 2024

    Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang, and Yixin Zhu. Autonomous character-scene interaction synthesis from text instruction, 2024. URL https: //arxiv.org/abs/2410.03187. 3

    2024 arXiv

  31. [39]

    Scaling up dynamic human-scene interaction modeling

    Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1737–1747, 2024. 3

    2024

  32. [40]

    Optimizing diffusion noise can serve as universal motion priors

    Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwa- janakorn, and Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1334–1345, ...

    2024

  33. [41]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, 2014. 20

    2014

  34. [42]

    Auto-Encoding Variational Bayes

    Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints, 2013. 19

    2013

  35. [43]

    Glow: Generative flow with invertible 1x1 convolutions

    Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. NeurIPS, 2018. 19

    2018

  36. [44]

    Eschernet: A generative model for scalable view synthesis

    Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew J Davison. Eschernet: A generative model for scalable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9503–9513, 2024. 5, 19

    2024

  37. [45]

    Nifty: Neural object interaction fields for guided human motion synthesis

    Nilesh Kulkarni, Davis Rempe, Kyle Genova, Abhijit Kundu, Justin Johnson, David Fouhey, and Leonidas Guibas. Nifty: Neural object interaction fields for guided human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9...

    2024

  38. [46]

    Object motion guided human motion synthesis

    Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG), 42(6):1–11, 2023. 2, 3, 5, 6, 7, 8

    2023

  39. [47]

    Controllable human-object interaction synthesis

    Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Controllable human-object interaction synthesis. In European Conference on Computer Vision, pages 54–72. Springer, 2024. 2, 3, 7, 8, 23

    2024

  40. [48]

    Ganima- tor: Neural motion synthesis from a single sequence

    Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung. Ganima- tor: Neural motion synthesis from a single sequence. ACM Trans. Graph., 2022. 19

    2022

  41. [49]

    Task-oriented human-object interactions generation with implicit neural representations

    Quanzhou Li, Jingbo Wang, Chen Change Loy, and Bo Dai. Task-oriented human-object interactions generation with implicit neural representations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3035–3044, 2024. 3 12

    2024

  42. [50]

    Ai choreographer: Music conditioned 3d dance generation with aist++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13401–13412, 2021. 19

    2021

  43. [51]

    Learning physics- based full-body human reaching and grasping from brief walking references

    Yitang Li, Mingxian Lin, Zhuo Lin, Yipeng Deng, Yue Cao, and Li Yi. Learning physics- based full-body human reaching and grasping from brief walking references. arXiv preprint arXiv:2503.07481, 2025. 3

    2025 arXiv

  44. [52]

    Motion-x: A large-scale 3d expressive whole-body human motion dataset

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 36, 2024. 19

    2024

  45. [53]

    Character controllers using motion vaes

    Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. Character controllers using motion vaes. ACM Trans. Graph., 2020. 7

    2020

  46. [54]

    Programmable motion generation for open-set motion control tasks

    Hanchao Liu, Xiaohang Zhan, Shaoli Huang, Tai-Jiang Mu, and Ying Shan. Programmable motion generation for open-set motion control tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1399–1408, 2024. 19

    2024

  47. [55]

    Geneoh diffusion: Towards generalizable hand-object interaction denoising via denoising diffusion

    Xueyi Liu and Li Yi. Geneoh diffusion: Towards generalizable hand-object interaction denoising via denoising diffusion. In The Twelfth International Conference on Learning Representations, 2024. 3

    2024

  48. [56]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7. 20

    2019

  49. [57]

    Choice: Coordinated human-object interaction in cluttered environments for pick-and-place actions

    Jintao Lu, He Zhang, Yuting Ye, Takaaki Shiratori, Sebastian Starke, and Taku Komura. Choice: Coordinated human-object interaction in cluttered environments for pick-and-place actions. arXiv preprint arXiv:2412.06702, 2024. 3

    2024

  50. [58]

    Scamo: Exploring the scaling law in autoregressive motion generation model

    Shunlin Lu, Jingbo Wang, Zeyu Lu, Ling-Hao Chen, Wenxun Dai, Junting Dong, Zhiyang Dou, Bo Dai, and Ruimao Zhang. Scamo: Exploring the scaling law in autoregressive motion generation model. arXiv preprint arXiv:2412.14559, 2024. 19

    2024 arXiv

  51. [59]

    Perpetual humanoid control for real-time simulated avatars

    Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895–10904, 2023. 9, 19, 22

    2023

  52. [60]

    Kitani, and Weipeng Xu

    Zhengyi Luo, Jinkun Cao, Sammy Christen, Alexander Winkler, Kris M. Kitani, and Weipeng Xu. Omnigrasp: Simulated humanoid grasping on diverse objects. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview. net/forum?id=Gl...

    2024

  53. [61]

    Kitani, and Weipeng Xu

    Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris M. Kitani, and Weipeng Xu. Universal humanoid motion representations for physics-based control. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum...

    2024

  54. [62]

    Himo: A new benchmark for full- body human interacting with multiple objects, 2024

    Xintao Lv, Liang Xu, Yichao Yan, Xin Jin, Congsheng Xu, Shuwen Wu, Yifan Liu, Lincheng Li, Mengxiao Bi, Wenjun Zeng, and Xiaokang Yang. Himo: A new benchmark for full- body human interacting with multiple objects, 2024. URL https://arxiv.org/abs/2407. 12371. 3

    2024

  55. [63]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In ICCV, 2019. 2, 6, 7, 8, 19, 21

    2019

  56. [64]

    Isaac gym: High performance gpu-based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021. 9, 19, 23

    2021 arXiv

  57. [65]

    Black, and Javier Romero

    Julieta Martinez, Michael J. Black, and Javier Romero. On human motion prediction using recurrent neural networks. In CVPR, 2017. 19 13

    2017

  58. [66]

    SDEdit: Guided image synthesis and editing with stochastic differential equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=a...

    2022

  59. [67]

    Generating continual human motion in diverse 3d scenes

    Aymen Mir, Xavier Puig, Angjoo Kanazawa, and Gerard Pons-Moll. Generating continual human motion in diverse 3d scenes. In 2024 International Conference on 3D Vision (3DV), pages 903–913. IEEE, 2024. 3

    2024

  60. [68]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021. 20

    2021

  61. [69]

    Ditto: Diffusion inference-time t-optimization for music generation.arXiv preprint arXiv:2401.12179,

    Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrick, and Nicholas J Bryan. Ditto: Diffusion inference-time t-optimization for music generation.arXiv preprint arXiv:2401.12179,

    arXiv

  62. [70]

    Openai: Introducing chatgpt

    OpenAI. Openai: Introducing chatgpt. https://openai.com/blog/chatgpt, 2022. 3, 19

    2022

  63. [71]

    Synthesizing physically plausible human motions in 3d scenes

    Liang Pan, Jingbo Wang, Buzhen Huang, Junyu Zhang, Haofan Wang, Xu Tang, and Yangang Wang. Synthesizing physically plausible human motions in 3d scenes. In 2024 International Conference on 3D Vision (3DV), pages 1498–1507. IEEE, 2024. 3

    2024

  64. [72]

    Tokenhsi: Unified synthesis of physical human-scene interactions through task tokenization

    Liang Pan, Zeshi Yang, Zhiyang Dou, Wenjia Wang, Buzhen Huang, Bo Dai, Taku Komura, and Jingbo Wang. Tokenhsi: Unified synthesis of physical human-scene interactions through task tokenization. In CVPR, 2025. 3

    2025

  65. [73]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019. 4

    2019

  66. [74]

    Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models

    Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06553, 2023. 3

    2023 arXiv

  67. [75]

    Deepmimic: Example- guided deep reinforcement learning of physics-based character skills

    Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1–14, 2018. 9, 19, 22

    2018

  68. [76]

    Amp: Adversarial motion priors for stylized physics-based character control

    Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG), 40(4):1–20, 2021. 19

    2021

  69. [77]

    Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters

    Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 41(4):1–17, 2022. 19

    2022

  70. [78]

    Black, and Gül Varol

    Mathis Petrovich, Michael J. Black, and Gül Varol. Action-conditioned 3D human motion synthesis with transformer V AE. InICCV, 2021. 19

    2021

  71. [79]

    Hierarchical generation of human-object interactions with diffusion probabilistic models

    Huaijin Pi, Sida Peng, Minghui Yang, Xiaowei Zhou, and Hujun Bao. Hierarchical generation of human-object interactions with diffusion probabilistic models. In ICCV, 2023. 2, 3, 7

    2023

  72. [80]

    The kit motion-language dataset

    Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 2016. 19

    2016

  73. [81]

    Efficient learning on point clouds with basis point sets

    Sergey Prokudin, Christoph Lassner, and Javier Romero. Efficient learning on point clouds with basis point sets. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4332–4341, 2019. 2, 5

    2019

  74. [82]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 5

    2021

  75. [83]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurIPS, 2019. 19 14

    2019

  76. [84]

    Trace and pace: Controllable pedestrian animation via guided trajectory diffusion

    Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 19

    2023

  77. [85]

    World-grounded human motion recovery via gravity-view coordinates

    Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng, Zechen Hu, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. World-grounded human motion recovery via gravity-view coordinates. In SIGGRAPH Asia Conference Proceedings, 2024. 20

    2024

  78. [86]

    Learning structured output representation using deep conditional generative models

    Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NeurIPS, 2015. 7

    2015

  79. [87]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL https://openreview. net/forum?id=St1giarCHLP. 7, 20

    2021

  80. [88]

    Hoianimator: Generating text-prompt human-object animations using novel perceptive diffusion models

    Wenfeng Song, Xinyu Zhang, Shuai Li, Yang Gao, Aimin Hao, Xia Hou, Chenglizhao Chen, Ning Li, and Hong Qin. Hoianimator: Generating text-prompt human-object animations using novel perceptive diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  81. [89]

    Neural state machine for character- scene interactions

    Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character- scene interactions. ACM Trans. Graph., 2019. 2, 3, 19

    2019

  82. [90]

    Local motion phases for learning multi-contact character movements

    Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. Local motion phases for learning multi-contact character movements. ACM Trans. Graph., 2020

    2020

  83. [91]

    Deepphase: Periodic autoencoders for learning motion phase manifolds

    Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Trans. Graph., 2022. 19

    2022

  84. [92]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. 5

    2024

  85. [93]

    Grab: A dataset of whole-body human grasping of objects

    Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16 , pages 581–600. Springer, 2020. 2, 3, ...

    2020

  86. [94]

    Black, and Dimitrios Tzionas

    Omid Taheri, Vasileios Choutas, Michael J. Black, and Dimitrios Tzionas. Goal: Generating 4d whole-body motion for hand-object grasping. In CVPR, 2022. 3

    2022

  87. [95]

    Omid Taheri, Yi Zhou, Dimitrios Tzionas, Yang Zhou, Duygu Ceylan, Soren Pirk, and Michael J. Black. GRIP: Generating interaction poses using latent consistency and spatial cues. In International Conference on 3D Vision (3DV), 2024. URL https://grip.is.tue. mpg.de. 3

    2024

  88. [96]

    Maskedmimic: Unified physics-based character control through masked motion inpainting

    Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Unified physics-based character control through masked motion inpainting. ACM Transactions on Graphics (TOG), 43(6):1–21, 2024. 3

    2024

  89. [97]

    Human motion diffusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In ICLR, 2023. 5, 7, 8, 9, 19, 20, 23

    2023

  90. [98]

    Edge: Editable dance generation from music

    Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In CVPR, 2023. 19

    2023

  91. [99]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 19, 20

    2017

  92. [100]

    End-to-end diffusion latent optimization improves classifier guidance

    Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7280–7290, 2023. 19

    2023

  93. [101]

    Edict: Exact diffusion inversion via coupled transformations

    Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22532–22541, 2023. 19 15

    2023

  94. [102]

    Tlcontrol: Trajectory and language control for human motion synthesis

    Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. Tlcontrol: Trajectory and language control for human motion synthesis. In ECCV 2024, pages 37–54. Springer Nature Switzerland, 2024. 19

    2024

  95. [103]

    Synthesizing long-term 3d human motion and interaction in 3d scenes

    Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction in 3d scenes. In CVPR, 2021. 3

    2021

  96. [104]

    Strategy and skill learning for physics- based table tennis animation

    Jiashun Wang, Jessica Hodgins, and Jungdam Won. Strategy and skill learning for physics- based table tennis animation. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 19

    2024

  97. [105]

    Towards diverse and natural scene-aware 3d human motion synthesis

    Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. Towards diverse and natural scene-aware 3d human motion synthesis. In CVPR, 2022. 3

    2022

  98. [106]

    Sims: Simulating human-scene interactions with real world script planning

    Wenjia Wang, Liang Pan, Zhiyang Dou, Zhouyingcheng Liao, Yuke Lou, Lei Yang, Jingbo Wang, and Taku Komura. Sims: Simulating human-scene interactions with real world script planning. arXiv preprint arXiv:2411.19921, 2024. 3

    2024 arXiv

  99. [107]

    Physhoi: Physics-based imitation of dynamic human-object interaction

    Yinhuai Wang, Jing Lin, Ailing Zeng, Zhengyi Luo, Jian Zhang, and Lei Zhang. Physhoi: Physics-based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.04393, 2023. 3, 9, 22

    2023 arXiv

  100. [108]

    Skillmimic: Learning reusable basketball skills from demonstrations

    Yinhuai Wang, Qihan Zhao, Runyi Yu, Ailing Zeng, Jing Lin, Zhengyi Luo, Hok Wai Tsui, Jiwen Yu, Xiu Li, Qifeng Chen, et al. Skillmimic: Learning reusable basketball skills from demonstrations. arXiv preprint arXiv:2408.15270, 2024. 3

    2024 arXiv

  101. [109]

    Physics-based character controllers using conditional vaes

    Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Physics-based character controllers using conditional vaes. ACM Trans. Graph. , 41(4), July 2022. ISSN 0730-0301. doi: 10.1145/3528223.3530067. URL https://doi.org/10.1145/3528223.3530067. 19

    2022

  102. [110]

    Saga: Stochastic whole-body grasping with contact

    Yan Wu, Jiahao Wang, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, and Siyu Tang. Saga: Stochastic whole-body grasping with contact. In ECCV, 2022. 3

    2022

  103. [111]

    Human-object interaction from human-level instructions

    Zhen Wu, Jiaman Li, Pei Xu, and C Karen Liu. Human-object interaction from human-level instructions. arXiv preprint arXiv:2406.17840, 2024. 3

    2024 arXiv

  104. [112]

    Motionstreamer: Streaming motion generation via diffusion-based autoregressive model in causal latent space

    Lixing Xiao, Shunlin Lu, Huaijin Pi, Ke Fan, Liang Pan, Yueer Zhou, Ziyong Feng, Xiaowei Zhou, Sida Peng, and Jingbo Wang. Motionstreamer: Streaming motion generation via diffusion-based autoregressive model in causal latent space. arXiv preprint arXiv:2503.15451,

    arXiv

  105. [113]

    Unified human-scene interaction via prompted chain-of-contacts

    Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-contacts. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=1...

    2024

  106. [114]

    Omnicontrol: Control any joint at any time for human motion generation

    Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. In The Twelfth International Conference on Learning Representations, 2024. 19

    2024

  107. [115]

    Interdiff: Generating 3d human-object interactions with physics-informed diffusion

    Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In ICCV, 2023. 2, 3

    2023

  108. [116]

    Interdreamer: Zero-shot text to 3d dynamic human-object interaction

    Sirui Xu, Yu-Xiong Wang, Liangyan Gui, et al. Interdreamer: Zero-shot text to 3d dynamic human-object interaction. Advances in Neural Information Processing Systems, 37:52858– 52890, 2024. 3

    2024

  109. [117]

    Guiding human- object interactions with rich geometry and relations

    Mengqing Xue, Yifei Liu, Ling Guo, Shaoli Huang, and Changxing Ding. Guiding human- object interactions with rich geometry and relations. arXiv preprint arXiv:2503.20172, 2025. 3

    2025 arXiv

  110. [118]

    OakInk: A large-scale knowledge repository for understanding hand-object interaction

    Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. OakInk: A large-scale knowledge repository for understanding hand-object interaction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2 16

    2022

  111. [119]

    Controlvae: Model-based learning of generative controllers for physics-based characters

    Heyuan Yao, Zhenhua Song, Baoquan Chen, and Libin Liu. Controlvae: Model-based learning of generative controllers for physics-based characters. ACM Transactions on Graphics (TOG), 41(6):1–16, 2022. 19

    2022

  112. [120]

    Moconvq: Unified physics-based motion control via scalable discrete representations

    Heyuan Yao, Zhenhua Song, Yuyang Zhou, Tenglong Ao, Baoquan Chen, and Libin Liu. Moconvq: Unified physics-based motion control via scalable discrete representations. ACM Transactions on Graphics (TOG), 43(4):1–21, 2024. 19

    2024

  113. [121]

    Generating human interaction motions in scenes with text control

    Hongwei Yi, Justus Thies, Michael J Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. In European Conference on Computer Vision, pages 246–263. Springer, 2025. 2, 3

    2025

  114. [122]

    Skillmimic-v2: Learning robust and generalizable interaction skills from sparse and noisy demonstrations

    Runyi Yu, Yinhuai Wang, Qihan Zhao, Hok Wai Tsui, Jingbo Wang, Ping Tan, and Qifeng Chen. Skillmimic-v2: Learning robust and generalizable interaction skills from sparse and noisy demonstrations. arXiv preprint arXiv:2505.02094, 2025. 3

    2025 arXiv

  115. [123]

    Oakink2: A dataset of bimanual hands-object manipulation in complex task completion

    Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page...

    2024

  116. [124]

    Learning physically simulated tennis skills from broadcast videos

    Haotian Zhang, Ye Yuan, Viktor Makoviychuk, Yunrong Guo, Sanja Fidler, Xue Bin Peng, and Kayvon Fatahalian. Learning physically simulated tennis skills from broadcast videos. ACM Trans. Graph., 2023. 19

    2023

  117. [125]

    Mode-adaptive neural networks for quadruped motion control

    He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion control. ACM Trans. Graph., 2018. 19

    2018

  118. [126]

    Manipnet: Neural manipulation synthesis with a hand-object spatial representation

    He Zhang, Yuting Ye, Takaaki Shiratori, and Taku Komura. Manipnet: Neural manipulation synthesis with a hand-object spatial representation. ACM Trans. Graph., 2021. 3

    2021

  119. [127]

    Artigrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation

    Hui Zhang, Sammy Christen, Zicong Fan, Luocheng Zheng, Jemin Hwangbo, Jie Song, and Otmar Hilliges. Artigrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation. In 2024 International Conference on 3D Vision (3DV), pages 235–246. IEEE,

    2024

  120. [128]

    Manidext: Hand-object manipulation synthesis via continuous correspondence embeddings and residual-guided diffusion

    Jiajun Zhang, Yuxiang Zhang, Liang An, Mengcheng Li, Hongwen Zhang, Zonghai Hu, and Yebin Liu. Manidext: Hand-object manipulation synthesis via continuous correspondence embeddings and residual-guided diffusion. arXiv preprint arXiv:2409.09300, 2024. 3, 21

    2024 arXiv

  121. [129]

    Generating human motion from textual descriptions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In CVPR, 2023. 19

    2023

  122. [130]

    Motiondiffuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint, 2022. 19

    2022

  123. [131]

    Roam: Robust and object-aware motion generation using neural pose descriptors

    Wanyue Zhang, Rishabh Dabral, Thomas Leimkühler, Vladislav Golyanik, Marc Habermann, and Christian Theobalt. Roam: Robust and object-aware motion generation using neural pose descriptors. In 2024 International Conference on 3D Vision (3DV), pages 1392–1402. IEEE,

    2024

  124. [132]

    Bimart: A unified approach for the synthesis of 3d bimanual interaction with articulated objects

    Wanyue Zhang, Rishabh Dabral, Vladislav Golyanik, Vasileios Choutas, Eduardo Alvarado, Thabo Beeler, Marc Habermann, and Christian Theobalt. Bimart: A unified approach for the synthesis of 3d bimanual interaction with articulated objects. Proceedings of the IEEE/CVF Conference...

    2025

  125. [133]

    Couch: Towards controllable human-chair interactions

    Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons- Moll. Couch: Towards controllable human-chair interactions. In ECCV, 2022. 3

    2022

  126. [134]

    Motiongpt: Finetuned llms are general-purpose motion generators

    Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned llms are general-purpose motion generators. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7368–7376, 2024. 19 17

    2024

  127. [135]

    Diffgrasp: Whole-body grasping synthesis guided by object motion using a diffusion model

    Yonghao Zhang, Qiang He, Yanguang Wan, Yinda Zhang, Xiaoming Deng, Cuixia Ma, and Hongan Wang. Diffgrasp: Whole-body grasping synthesis guided by object motion using a diffusion model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10320–103...

    2025

  128. [136]

    Motion mamba: Efficient and long sequence motion generation

    Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation. In European Conference on Computer Vision, pages 265–282. Springer, 2025. 19

    2025

  129. [137]

    Synthesizing diverse human motions in 3d indoor scenes

    Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In ICCV, 2023. 3

    2023

  130. [138]

    Cams: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis

    Juntian Zheng, Qingyuan Zheng, Lixing Fang, Yun Liu, and Li Yi. Cams: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 585–594, 2023. 3, 8, 21

    2023

  131. [139]

    Gimo: Gaze-informed human motion prediction in context

    Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, C Karen Liu, and Leonidas J Guibas. Gimo: Gaze-informed human motion prediction in context. In ECCV,

  132. [140]

    Toch: Spatio- temporal object-to-hand correspondence for motion refinement

    Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll. Toch: Spatio- temporal object-to-hand correspondence for motion refinement. In European Conference on Computer Vision, pages 1–19. Springer, 2022. 3

    2022

  133. [141]

    Emdm: Efficient motion diffusion model for fast and high-quality motion generation

    Wenyang Zhou, Zhiyang Dou#, Zeyu Cao, Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient motion diffusion model for fast and high-quality motion generation. In ECCV 2024, pages 18–38. Springer Nature Switzerland,...

    2024

  134. [142]

    On the continuity of rotation representations in neural networks

    Yi Zhou, Connelly Barnes, Lu Jingwan, Yang Jimei, and Li Hao. On the continuity of rotation representations in neural networks. In CVPR, 2019. 6

    2019

  135. [143]

    Neural categorical priors for physics- based character control

    Qingxu Zhu, He Zhang, Mengting Lan, and Lei Han. Neural categorical priors for physics- based character control. ACM Trans. Graph. , 42(6), dec 2023. ISSN 0730-0301. doi: 10.1145/3618397. URL https://doi.org/10.1145/3618397. 19 18 A Introduction This supplementary document pro...

    2023 doi

  136. [144]

    Several recent works [77, 109, 119, 18, 120, 61] learn latent representations of human motion that support skill reuse

    to perform various motor skills [124, 143, 104] through reinforcement learning. Several recent works [77, 109, 119, 18, 120, 61] learn latent representations of human motion that support skill reuse. Diffusion noise optimization. Diffusion models [31] have shown great success ...

    2000

  137. [145]

    Whole-body motion generation

    optimizer with a cosine-decayed learning rate 0.05 for 800 steps to calculate the end-effector trajectories. Whole-body motion generation. During inference, we perform noise optimization using DDIM

  138. [146]

    with T =10 for 800 steps and a cosine-decayed learning rate 0.05, following the DNO [ 40] strategy. The optimization loss with different weights λee, λpen, and λreg is as follows: L = λeeLee + λpenLpen + λregLreg, (19) where Lee, Lpen, and Lreg are the end-effector tracking, p...

  139. [147]

    Developing richer textual annotations and grounding them to temporally extended actions is a promising avenue for future work

    provides rule-based captions for hand-object interactions, they are typically segmented into short atomic motions, whereas we aim to model longer and more coherent manipulation sequences. Developing richer textual annotations and grounding them to temporally extended actions i...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.