Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that synchronized video-motion diffusion with a closed-loop vision-aware 3D interaction model generates high-fidelity hand-object videos and explicit 3D motions together, without predefined object models or pose guidance.

desk verdict A genuinely new synchronized video-motion diffusion framework for HOI, but the main quantitative evidence is internally inconsistent across tables, so the SOTA claim is not yet verifiable. read the letter →

arxiv 2506.02444 v3 pith:42FZLRL5 submitted 2025-06-03 cs.CV

Lingwei Dang , Ruizhi Shao , Hongwen Zhang , Wei Min , Yebin Liu , Qingyao Wu This is my paper
classification cs.CV
keywords hand-objectinteractionvideogeneration3Dmotiondiffusionmodelssynchronizedclosed-loopfeedbackmultimodallatent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that video appearance and 3D motion in hand-object interaction should be generated as one synchronized diffusion event, not as separate pipelines. It proposes SViMo, which jointly denoises video latents and motion latents from a reference image and text prompt, with the motion represented as a rendered 2D motion video so a pretrained video model can ingest it. A companion vision-aware 3D interaction diffusion model (VID) reads the denoised output, predicts explicit hand trajectories and object point clouds, and feeds them back into the video denoiser in a closed loop. According to the paper, this removes the need for predefined 3D object models or pose-sequence guidance and yields higher-fidelity, physically plausible HOI sequences that generalize to unseen real-world objects. A reader should care because it is a direct test of whether visual priors and physical constraints can reinforce each other in one diffusion model rather than being traded off.

What carries the argument

The load-bearing mechanism is the closed loop between two diffusion components. SViMo is the synchronized video-motion diffusion model: a pretrained video diffusion Transformer extended so its token sequence contains text, video, and motion tokens, aligned by triple-modality adaptive modulation and fused by 3D full-attention; the motion token stream is a rendered motion video obtained by projecting 3D hand joints and object point clouds onto a 2D image plane and encoding it with the same video VAE. VID is a vision-aware 3D interaction diffusion model that takes the denoised video and motion latents, fuses them in a dual-stream 3D convolutional module, and generates explicit hand poses and object point clouds, supervised by joint mean squared error and Chamfer distance. The closed loop works in two directions: VID's output is projected and encoded to become interaction guidance concatenated with the noised motion latent, and gradients from VID's loss backpropagate into SViMo, so the two models co-evolve during training.

What would settle it

Run the method on annotated contact frames of the test split and measure hand-object penetration depth and contact coverage between predicted hand joints and predicted object point clouds; if these are not better than the baselines, the physical-plausibility benefit of the closed loop is not supported.

Watch

Extended reading notes

Core claim

SViMo's central discovery is that the long-standing trade-off between visually realistic HOI video and physically plausible 3D motion can be dissolved by letting both modalities diffuse together in one Transformer. The paper's architecture feeds text, image-video, and rendered-motion tokens into a single 3D full-attention stack with tri-modal adaptive modulation, then closes the loop with VID, which converts the denoised video and motion latents into explicit 3D hand joint trajectories and object point clouds and re-injects them as interaction guidance plus gradient constraints. The paper reports that this synchronized design outperforms state-of-the-art video and motion baselines on video and 3D metrics, and that ablations show removing VID or splitting video and motion generation degrades both outputs. Its claim is that visual appearance and motion patterns share the same physical laws and can be modeled by a shared diffusion process without needing object models or explicit pose inputs.

Load-bearing premise

The load-bearing premise is that projecting 3D hand joints and object points onto a 2D rendered motion video preserves enough spatial, contact, and depth information for a video model to encode and for the 3D model to recover, and if that projection loses crucial 3D cues the video-motion consistency and generalization claims collapse.

Editorial extensions

If this is right

  • Holding a reference image and text prompt, SViMo produces both a video and an explicit 3D hand-and-object motion sequence from the same denoising run, with no 3D object mesh and no input pose sequence required.
  • The closed-loop feedback keeps video pixels and 3D motion mutually consistent: object point clouds and hand trajectories are regenerated from the video latents, and those 3D outputs refine the video denoising.
  • On the paper's reported evaluations, the joint scheme outperforms sole video generators on overall video score and outperforms motion diffusion baselines by a wide margin on joint error, Chamfer distance, and interaction FID.
  • Ablations imply that synchronized training is necessary: splitting video and motion generation, or removing VID, drops both the video overall score and the motion FID.
  • The same model transfers to unseen household objects in zero-shot real-world tests without retraining on those objects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the rendered-motion-video trick suggests a general recipe for grafting any explicit 3D representation onto a video diffusion foundation model: project the 3D signal to 2D, let the video model ingest it, then lift it back with a small 3D head, so the idea may transfer to full-body interactions, articulated objects, or scenes.
  • The closed-loop feedback is not limited to hand-object pairs; the same design of letting one modality diffuse while a predictor recovers an explicit structure that is re-encoded as guidance could synchronize video with audio, tactile, or robot-action streams where paired supervision is scarce.
  • A testable extension the paper does not run is to quantify physical plausibility directly with contact and penetration metrics between predicted hand joints and object point clouds, rather than only joint error and Chamfer distance, which would sharpen the claim of dynamic plausibility.
  • Because the paper's appendix lists rigid simple objects as a limitation, a natural stress test is to feed objects with complex or deformable geometry and see whether the recovered object point clouds remain coherent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SViMo, a synchronized diffusion framework that jointly generates hand-object interaction (HOI) videos and explicit 3D motion sequences from a reference image and text prompt. The method extends a pretrained video DiT by introducing tri-modal adaptive modulation for text, video, and motion tokens, a 3D full-attention mechanism, and a 'rendered motion video' representation that projects 3D hand joints and object point clouds onto 2D. A vision-aware 3D Interaction Diffusion model (VID) recovers explicit 3D interactions from the synchronized latent codes and feeds them back into the denoising loop, forming a closed-loop optimization. The framework is trained and evaluated on the TACO dataset on a 1:9 split, reporting video quality via VBench metrics and motion quality via MPJPE, motion smoothness, Chamfer distance, and a motion FID. The authors claim improved video-motion consistency and zero-shot generalization to unseen real-world scenarios.

Significance. If the quantitative evidence were internally consistent, this would be a noteworthy contribution: an end-to-end generative model that produces a video and a matched explicit 3D interaction sequence without predefined object models or pose guidance, leveraging large-scale video priors. The idea of a synchronized video-motion diffusion process with closed-loop vision-aware feedback is original and plausible, and the paper provides useful implementation details, including training/inference pseudocode (Algorithms 1 and 2) and a user study with a substantial number of responses. The qualitative results in Figures 4 and 6 are suggestive. However, the current quantitative support has a serious internal inconsistency between Table 2 and Table 3, and several evaluation details are insufficient to verify the claimed state-of-the-art performance. The paper is not ready for acceptance in its present form.

major comments (5)
  1. [Tables 2 and 3] The identical full configuration, listed as 'Ours' in Table 2 and as 'SViMo w/ VID (Ours)' in Table 3, reports vastly different values on the same TACO 1:9 split: MPJPE 0.1087 versus 0.0121, motion smoothness 0.0255 versus 0.0053, Chamfer distance 0.1577 versus 0.0019, and FID 0.1050 versus 0.0100. These are differences of roughly 2x, 5x, 80x, and 10x, respectively, with no explanation of any normalization, unit, subset, or protocol change. Because Table 2 is the only head-to-head motion comparison against MDM and EMDM, the claimed superiority over state-of-the-art motion generators cannot be verified as reported. The authors must clarify the exact evaluation protocol for each table, state the units and normalization, and if necessary rerun the comparisons so that both tables describe the same configuration consistently.
  2. [Section 3.3 and Eq. (7)] The 'rendered motion video' is the core motion representation through which 3D hand joints and object point clouds are fed into the video model, and it is also the space from which VID recovers 3D structure. However, the paper never specifies the projection function: it does not state whether the projection is orthographic or perspective, which camera parameters are used, how hand joints and object point clouds are rasterized, or whether any depth information is retained. Since the whole closed-loop feedback (Eq. 7) depends on the information content of this 2D rendering, the missing specification makes the method not fully reproducible and leaves the central assumption that 3D cues survive the projection uncheckable. Please provide the exact projection procedure and, ideally, a quantitative or qualitative analysis of what 3D information is preserved.
  3. [Appendix B.3 and Section 4.4] The motion FID is computed using a feature extractor that the authors trained on the TACO dataset itself, and the proposed method is fine-tuned on TACO and evaluated on a 1:9 split of the same dataset. This same-distribution evaluation can inflate FID scores, especially since the extractor may have seen the same action and object distributions. The paper should report whether the FID extractor was trained only on the training split, whether any test sequences were used, and provide quantitative cross-dataset evaluation to support the claimed zero-shot generalization; currently, generalization to unseen real-world data is evidenced only by qualitative examples in Figure 6. Without such clarification, the generalization claim is not quantitatively substantiated.
  4. [Tables 1-3 and Section 4.5] No error bars, number of test samples, or random seeds are reported for any of the quantitative results. This is particularly problematic in the ablation study (Table 3), where many differences are tiny (e.g., Subject Consistency 0.9534 versus 0.9522 versus 0.9499; Overall 0.8800 versus 0.8770 versus 0.8723). Without variance estimates or significance testing, it is impossible to tell whether these differences are meaningful or within noise. Please report standard deviations over multiple seeds or at least the number of evaluation videos and the test-split size.
  5. [Tables 2 and 4.3] The motion generation comparison is limited to MDM and EMDM, which are general human motion generation models, not methods specifically designed for hand-object interaction such as Text2HOI, InterDiff, or CG-HOI. While the paper states that the baselines were modified to align the setup, the absence of HOI-specific state-of-the-art baselines weakens the claim of 'superiority over state-of-the-art approaches' for HOI motion generation. The authors should either include HOI-specific baselines or justify why MDM and EMDM are the appropriate comparisons.
minor comments (5)
  1. [Abstract and Section 1] Two different project page URLs are given: 'https://github.com/Droliven/SViMo_project' in the abstract and 'https://droliven.github.io/SViMo_project/' in Section 1. Please unify them.
  2. [Section 3.3, Feature Embedding] There is a typo: 'esulting' should be 'resulting' in the sentence about the patchified visual feature.
  3. [Table 3 caption and Section 4.5] The caption says 'Varients', which should be 'Variants'. Also, in Section 4.5 the text mentions a variant 'VModel w/ GT Mot. Guid.' but the table row is labeled 'VModel w/ Pred. Mot.'; please align the naming.
  4. [Table 1] The 'Training' column with entries 'w/o' and 'w/' is not defined in the caption. It appears to indicate whether each baseline was fine-tuned on TACO or used zero-shot, but this should be stated explicitly.
  5. [Appendix D] The limitations section acknowledges that the generated object point clouds are restricted to rigid, simple objects and that the method relies on paired video-motion data. These are important caveats that affect the central claims and should be summarized in the main text rather than only in the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the joint video-motion diffusion and the closed-loop VID feedback are supervised by external ground-truth 3D annotations, and the central comparisons rest on independent metrics and baselines.

full rationale

SViMo is an empirical generative system rather than a formal derivation, and its claimed outputs are not defined in terms of the inputs. The joint denoising objective in Eq. (5) is supervised by ground-truth video latents and by latents of a rendered motion video that is itself a projection of the target 3D interaction. VID in Eq. (6) is trained with MSE on hand joints and Chamfer distance on object point clouds against the TACO ground truth. These losses are not equivalent to the model's own outputs by construction; they compare against externally captured annotations. The closed-loop feedback in Eq. (7) feeds VID's estimates back into the denoising stream as interaction guidance and gradient constraints. This is a self-conditioning and refinement loop, not a circular derivation: the projection is lossy, and the supervision remains anchored to external 3D ground truth at every stage. The main quantitative claims are supported by comparisons to external baselines (MDM, EMDM, CogVideoX, Animate Anyone, etc.) and by metrics such as MPJPE, Chamfer distance, smoothness, and VBench scores, which are not fitted parameters renamed as predictions. One self-referential point exists: the FID metric in Appendix B.3 uses a 3D action reconstruction model trained by the authors on the same TACO dataset, with the architecture attributed to the first author's prior work [10]. This is a legitimate evaluation-validity concern, but it is not a circular derivation and the central motion-quality conclusions do not depend on FID alone. Separately, the reported motion metrics for the identical 'Ours' configuration differ by large factors between Table 2 and Table 3, which is a reproducibility and correctness issue rather than a circularity issue. The limitations stated in Appendix D and the failure cases in Appendix C.4 are candid and do not indicate that any prediction reduces to its own input. Overall, no equation or fitted quantity in the paper is shown to be equivalent to another by definition, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The method is an empirical learning system, so the ledger mostly contains domain assumptions about datasets, latent spaces, and the projection representation. The free parameters are training loss weights and coefficients. The only new representational entity is the rendered motion video, which has no independent evidence beyond the paper's results.

free parameters (4)
  • loss weight omega2 for LVID = 0.05
    Chosen by hand to balance video and motion losses during joint training; reported in Sec. 4.2.
  • first-order hand loss coefficient = 0.2
    Coefficient for hand velocity loss in Eq. 12, selected as a hyperparameter.
  • second-order hand loss coefficient = 0.05
    Coefficient for hand acceleration loss in Eq. 12, selected as a hyperparameter.
  • object motion dynamic loss coefficient = 0.1
    Weight for the inter-frame Chamfer dynamics loss in Eq. 13, selected as a hyperparameter.
assumptions (5)
  • domain assumption TACO provides sufficiently accurate 3D ground truth for hand joints and object point clouds.
    The entire training and evaluation rely on TACO's annotated 3D motions; no independent verification is provided.
  • domain assumption The frozen video VAE maps both real video frames and rendered motion videos into a shared latent space that supports joint denoising.
    SViMo concatenates rendered motion latents and video latents from the same VAE; if the latent distributions differ too much, the model cannot align them.
  • domain assumption The 2D projection of 3D hand and object motion preserves enough information for the VID to recover explicit 3D interactions.
    The rendered motion video is the only motion representation in SViMo, and the closed-loop feedback depends on this projection being informative.
  • standard math Standard x0-prediction diffusion training is a valid generative objective for both video and motion latents.
    The paper uses the DDPM formulation from Ho et al. [16] without modification.
  • domain assumption The pretrained CogVideoX-5B visual priors transfer to hand-object interaction scenarios after fine-tuning on TACO.
    The method is initialized from CogVideoX-5B and assumes its pretrained visual knowledge improves generation on the target domain.
invented entities (1)
  • Rendered motion video
    purpose: A 2D representation of 3D hand-object motion obtained by projecting 3D joints and point clouds onto the image plane, used as the motion modality in SViMo.
    This representation is introduced by the paper and is only validated through its own experiments; there is no external benchmark or measurement confirming its sufficiency for representing 3D interactions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios." pith.science (2026). https://pith.science/paper/42FZLRL5

@misc{pith2026250602444,
  author       = {Pith},
  title        = {Pith review of: SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/42FZLRL5}},
  note         = {Machine review of arXiv:2506.02444}
}
read the original abstract

Hand-Object Interaction (HOI) generation has significant application potential. However, current 3D HOI motion generation approaches heavily rely on predefined 3D object models and lab-captured motion data, limiting generalization capabilities. Meanwhile, HOI video generation methods prioritize pixel-level visual fidelity, often sacrificing physical plausibility. Recognizing that visual appearance and motion patterns share fundamental physical laws in the real world, we propose a novel framework that combines visual priors and dynamic constraints within a synchronized diffusion process to generate the HOI video and motion simultaneously. To integrate the heterogeneous semantics, appearance, and motion features, our method implements tri-modal adaptive modulation for feature aligning, coupled with 3D full-attention for modeling inter- and intra-modal dependencies. Furthermore, we introduce a vision-aware 3D interaction diffusion model that generates explicit 3D interaction sequences directly from the synchronized diffusion outputs, then feeds them back to establish a closed-loop feedback cycle. This architecture eliminates dependencies on predefined object models or explicit pose guidance while significantly enhancing video-motion consistency. Experimental results demonstrate our method's superiority over state-of-the-art approaches in generating high-fidelity, dynamically plausible HOI sequences, with notable generalization capabilities in unseen real-world scenarios. Project page at https://github.com/Droliven/SViMo_project.

Figures

Figures reproduced from arXiv: 2506.02444 by the authors.

Figure 1
Figure 1. Different HOI generation methods. Approaches like MDM [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our method comprises: (1) A synchronous diffusion model that jointly generates HOI [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of videos. Red boxes highlight artifacts such as deformation, hallucinations, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Quantitative comparison of motion generation results. Our method achieves smoother [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: User studies for generated videos (a) and motions (b). We received 1,066 and 410 valid [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Zero-shot inference on real-world data. We collect data with everyday objects and generate [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Video loss curves of different variants during the training process. The last two rows of Tab. 3 show that modeling video and motion independently not only leads to a 3.88% decrease in video overall score (0.8719 vs. 0.8381), but also results in a 5.31% degradation in …
Figure 8
Figure 8. Figure 8: Visualization of the generated videos and motions. We provide vivid demonstrations in the [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Comparison of video generation results. The artifacts in the videos generated by the [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Comparison of video generation results. The artifacts in the videos generated by the [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Comparison of video generation results. The artifacts in the videos generated by the [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Visualization of failure cases. Red boxes highlight artifacts. D Limitations and Future Work In fine-grained hand-object interaction scenarios, simultaneously generating visually high-fidelity videos and physically plausible motions remains a challenging task. While o…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph

    cs.CV 2026-08 conditional novelty 6.0 of 10

    PhotoHOI turns one RGB photo plus an open-vocabulary instruction into a scene-grounded 3D hand-object motion sequence by parsing the task, recovering objects, planning object motion, and optimizing grasps in a learned...

  2. AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment

    cs.CV 2026-07 conditional novelty 6.0 of 10

    AgentHOI generates human-object interaction videos from text plus one human image and one object image, using multi-agent action planning and implicit text-to-motion feature alignment inside a video diffusion model.

  3. StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A streaming image-to-video model that assigns block-specific sink/local memory and learned RoPE distance scaling preserves long human-object interactions under bounded latency.

Reference graph

Works this paper leans on

72 extracted references · 31 canonical work pages · cited by 3 Pith papers

  1. [1]

    Body of her: A preliminary study on end-to-end humanoid agent

    Tenglong Ao. Body of her: A preliminary study on end-to-end humanoid agent. arXiv preprint arXiv:2408.02879, 2024

    arXiv 2024

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  4. [4]

    Physically plausible full-body hand-object interaction synthesis

    Jona Braun, Sammy Christen, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges. Physically plausible full-body hand-object interaction synthesis. In 2024 International Conference on 3D Vision (3DV), pages 464–473. IEEE, 2024

    work page 2024

  5. [5]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. 2024. URL https://openai. com/research/video-generation-models-as-world-simulators, 3:1, 2024

    2024

  6. [6]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021

    2021

  7. [7]

    Text2hoi: Text-guided 3d motion generation for hand-object interaction

    Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1577–1585, 2024

    2024

  8. [8]

    Dexycb: A benchmark for capturing hand grasping of objects

    Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9044–9053, 2021

    2021

Show all 72 references
  1. [9]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  2. [10]

    Diverse human motion prediction via gumbel-softmax sampling from an auxiliary space

    Lingwei Dang, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. Diverse human motion prediction via gumbel-softmax sampling from an auxiliary space. In Proceedings of the 30th ACM international conference on multimedia, pages 5162–5171, 2022

    2022

  3. [11]

    Cg-hoi: Contact-guided 3d human-object interaction generation

    Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19888– 19901, 2024. 10

    2024

  4. [12]

    Arctic: A dataset for dexterous bimanual hand-object manipulation

    Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12...

    2023

  5. [13]

    Coohoi: Learning cooperative human-object interaction with manipulated object dynamics

    Jiawei Gao, Ziqin Wang, Zeqi Xiao, Jingbo Wang, Tai Wang, Jinkun Cao, Xiaolin Hu, Si Liu, Jifeng Dai, and Jiangmiao Pang. Coohoi: Learning cooperative human-object interaction with manipulated object dynamics. Advances in Neural Information Processing Systems, 37:79741–79763, 2024

    2024

  6. [14]

    Prediction with action: Visual policy learning via joint denoising process

    Yanjiang Guo, Yucheng Hu, Jianke Zhang, Yen-Jen Wang, Xiaoyu Chen, Chaochao Lu, and Jianyu Chen. Prediction with action: Visual policy learning via joint denoising process. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  7. [15]

    Stochastic scene-aware motion prediction

    Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11374–11384, 2021

    2021

  8. [16]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  9. [17]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In The Eleventh International Conference on Learning Representations, 2023

    2023

  10. [18]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024

    2024

  11. [19]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  12. [20]

    The power of sound (tpos): Audio reactive video generation with stable diffusion

    Yujin Jeong, Wonjeong Ryoo, Seunghyun Lee, Dabin Seo, Wonmin Byeon, Sangpil Kim, and Jinkyu Kim. The power of sound (tpos): Audio reactive video generation with stable diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7822–7832, 2023

    2023

  13. [21]

    Vace: All-in-one video creation and editing

    Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.07598, 2025

    2025 arXiv

  14. [22]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  15. [23]

    Nifty: Neural object interaction fields for guided human motion synthesis

    Nilesh Kulkarni, Davis Rempe, Kyle Genova, Abhijit Kundu, Justin Johnson, David Fouhey, and Leonidas Guibas. Nifty: Neural object interaction fields for guided human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9...

    2024

  16. [24]

    Interhandgen: Two-hand interaction generation via cascaded reverse diffusion

    Jihyun Lee, Shunsuke Saito, Giljoo Nam, Minhyuk Sung, and Tae-Kyun Kim. Interhandgen: Two-hand interaction generation via cascaded reverse diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 527–537, 2024

    2024

  17. [25]

    Controllable human-object interaction synthesis

    Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Controllable human-object interaction synthesis. In European Conference on Computer Vision, pages 54–72. Springer, 2024

    2024

  18. [26]

    Object motion guided human motion synthesis

    Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG), 42(6):1–11, 2023

    2023

  19. [27]

    Task-oriented human-object interactions generation with implicit neural representations

    Quanzhou Li, Jingbo Wang, Chen Change Loy, and Bo Dai. Task-oriented human-object interactions generation with implicit neural representations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3035–3044, 2024

    2024

  20. [28]

    Vision-language foundation models as effective robot imitators

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. In The Twelfth International Conference on Learning Representations, 2024

    2024

  21. [29]

    Amt: All-pairs multi-field transforms for efficient frame interpolation

    Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9801–9810, 2023. 11

    2023

  22. [30]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  23. [31]

    Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization

    Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, et al. Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization. arXiv preprint arXiv:2503.23377, 2025

    2025

  24. [32]

    Primitive-based 3d human-object interaction modelling and programming

    Siqi Liu, Yong-Lu Li, Zhou Fang, Xinpeng Liu, Yang You, and Cewu Lu. Primitive-based 3d human-object interaction modelling and programming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3711–3719, 2024

    2024

  25. [33]

    Geneoh diffusion: Towards generalizable hand-object interaction denoising via denoising diffusion

    Xueyi Liu and Li Yi. Geneoh diffusion: Towards generalizable hand-object interaction denoising via denoising diffusion. In The Twelfth International Conference on Learning Representations, 2024

    2024

  26. [34]

    Taco: Benchmarking generalizable bimanual tool-action-object understanding

    Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21740–21751, 2024

    2024

  27. [35]

    Hoi4d: A 4d egocentric dataset for category-level human-object interaction

    Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

    2022

  28. [36]

    Visual-rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  29. [37]

    Omnigrasp: Grasping diverse objects with simulated humanoids

    Zhengyi Luo, Jinkun Cao, Sammy Christen, Alexander Winkler, Kris Kitani, and Weipeng Xu. Omnigrasp: Grasping diverse objects with simulated humanoids. Advances in Neural Information Processing Systems, 37:2161–2184, 2024

    2024

  30. [38]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021

    2021

  31. [39]

    Manivideo: Generating hand-object manipulation video with dexterous and generalizable grasping

    Youxin Pang, Ruizhi Shao, Jiajun Zhang, Hanzhang Tu, Yun Liu, Boyao Zhou, Hongwen Zhang, and Yebin Liu. Manivideo: Generating hand-object manipulation video with dexterous and generalizable grasping. arXiv preprint arXiv:2412.16212, 2024

    2024 arXiv

  32. [40]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  33. [41]

    Hoi-diff: Text- driven synthesis of 3d human-object interactions using diffusion models

    Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. Hoi-diff: Text- driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06553, 2023

    2023 arXiv

  34. [42]

    Hierarchical generation of human- object interactions with diffusion probabilistic models

    Huaijin Pi, Sida Peng, Minghui Yang, Xiaowei Zhou, and Hujun Bao. Hierarchical generation of human- object interactions with diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15061–15073, 2023

    2023

  35. [43]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  36. [44]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  37. [45]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020

    2020

  38. [46]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning , pages 8821–8831. Pmlr, 2021

    2021

  39. [47]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 12

    2022

  40. [48]

    Photorealistic text-to- image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural informatio...

    2022

  41. [49]

    Hand-object interaction pretraining from videos

    Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, and Jitendra Malik. Hand-object interaction pretraining from videos. arXiv preprint arXiv:2409.08273, 2024

    2024 arXiv

  42. [50]

    Grab: A dataset of whole-body human grasping of objects

    Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 581–600. Springer, 2020

    2020

  43. [51]

    Any-to-any generation via composable diffusion

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36:16083–16099, 2023

    2023

  44. [52]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020

    2020

  45. [53]

    Human motion diffusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2023

    2023

  46. [54]

    Deepsimho: Stable pose estimation for hand-object interaction via physics simulation

    Rong Wang, Wei Mao, and Hongdong Li. Deepsimho: Stable pose estimation for hand-object interaction via physics simulation. Advances in Neural Information Processing Systems, 36:79685–79697, 2023

    2023

  47. [55]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475–121499, 2024

    2024

  48. [56]

    Physhoi: Physics-based imitation of dynamic human-object interaction

    Yinhuai Wang, Jing Lin, Ailing Zeng, Zhengyi Luo, Jian Zhang, and Lei Zhang. Physhoi: Physics-based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.04393, 2023

    2023 arXiv

  49. [57]

    Easyanimate: A high-performance long video generation method based on transformer architecture

    Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture. arXiv preprint arXiv:2405.18991, 2024

    2024

  50. [58]

    Interdiff: Generating 3d human-object interactions with physics-informed diffusion

    Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14928–14940, 2023

    2023

  51. [59]

    Intermimic: Towards universal whole-body control for physics-based human-object interactions

    Sirui Xu, Hung Yu Ling, Yu-Xiong Wang, and Liang-Yan Gui. Intermimic: Towards universal whole-body control for physics-based human-object interactions. 2025

    2025

  52. [60]

    Interdreamer: Zero-shot text to 3d dynamic human-object interaction

    Sirui Xu, Yu-Xiong Wang, Liangyan Gui, et al. Interdreamer: Zero-shot text to 3d dynamic human-object interaction. Advances in Neural Information Processing Systems, 37:52858–52890, 2024

    2024

  53. [61]

    Magicanimate: Temporally consistent human image animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  54. [62]

    Anchorcrafter: Animate cyberanchors saling your products via human-object interacting video generation

    Ziyi Xu, Ziyao Huang, Juan Cao, Yong Zhang, Xiaodong Cun, Qing Shuai, Yuchen Wang, Linchao Bao, Jintao Li, and Fan Tang. Anchorcrafter: Animate cyberanchors saling your products via human-object interacting video generation. arXiv preprint arXiv:2411.17383, 2024

    2024 arXiv

  55. [63]

    Oakink: A large- scale knowledge repository for understanding hand-object interaction

    Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large- scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20953–20962, 2022

    2022

  56. [64]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1):1, 2023

    2023 arXiv

  57. [65]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  58. [66]

    Diverse and aligned audio-to- video generation via text-to-video model adaptation

    Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to- video generation via text-to-video model adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6639–6647, 2024. 13

    2024

  59. [67]

    Oakink2: A dataset of bimanual hands-object manipulation in complex task completion

    Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445–456, 2024

    2024

  60. [68]

    Manidext: Hand-object manipulation synthesis via continuous correspondence embeddings and residual-guided diffusion

    Jiajun Zhang, Yuxiang Zhang, Liang An, Mengcheng Li, Hongwen Zhang, Zonghai Hu, and Yebin Liu. Manidext: Hand-object manipulation synthesis via continuous correspondence embeddings and residual-guided diffusion. arXiv preprint arXiv:2409.09300, 2024

    2024 arXiv

  61. [69]

    Couch: Towards controllable human-chair interactions

    Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll. Couch: Towards controllable human-chair interactions. In European Conference on Computer Vision , pages 518–535. Springer, 2022

    2022

  62. [70]

    Emdm: Efficient motion diffusion model for fast and high-quality motion generation

    Wenyang Zhou, Zhiyang Dou, Zeyu Cao, Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient motion diffusion model for fast and high-quality motion generation. In European Conference on Computer Vision, pages 18–38. ...

    2024

  63. [71]

    Champ: Controllable and consistent human image animation with 3d parametric guidance

    Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Zilong Dong, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. In European Conference on Computer Vision, pages 145–162. Springer, 2024

    2024

  64. [72]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 14 SV...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.