Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that one sparse arm skeleton and an object-center dot trajectory are enough for a pretrained multimodal diffusion transformer to generate natural, generalizable human-object interaction videos from a single human image…

desk verdict Useful weak-condition HOI generation with solid ablations, but the headline SOTA claim is tested on full pose, not the advertised weak inputs. read the letter →

arxiv 2506.08797 v3 pith:OOLAVDRR submitted 2025-06-10 cs.CV

Ziyao Huang , Zixiang Zhou , Juan Cao , Yifeng Ma , Yi Chen , Zejing Rao , Zhiyong Xu , Hongmei Wang
show 4 more authors
Qin Lin Yuan Zhou Qinglin Lu Fan Tang
This is my paper
classification cs.CV
keywords human-objectinteractionvideogenerationweakmotioncontrolmultimodaldiffusiontransformerhumanimageanimationobjecttrajectorydotaudio-drivenlipsynchronizationdepth-awaredatacurationHOIadapter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human-object interaction video generation usually demands strong inputs: captured actor motion, depth maps, or per-object fine-tuning. HunyuanVideo-HOMA claims those are unnecessary. Its central proposal is a weak conditioning scheme in which only a single arm skeleton and the trajectory of the object's center point drive the interaction, while a pretrained generative prior fills in everything else, including finger motion, hand pose, and object orientation. The authors argue this makes HOI generation practical for ordinary users, generalizes to novel and non-rigid objects, and still beats specialized systems on interaction naturalness and object consistency in their tests. If the claim holds, creating a video of a person interacting with a specified object reduces to supplying two images and two sparse motion signals.

What carries the argument

The load-bearing design is a two-space context fusion inside an MMDiT video diffusion transformer. Human appearance is channel-concatenated into the VAE latent; object appearance is both token-concatenated as a virtual first frame and copy-pasted at the trajectory's location in latent space. Motion is injected separately: one one-layer CNN encodes sparse arm skeletons, another encodes dot trajectories, and their features are added to the latent. An HOI Adapter, initialized from the pretrained MMDiT self-attention weights, re-injects object tokens at even layers inside a mask, which is what lets the deep network learn object appearance instead of only the input layer. An audio cross-attention adapter gates lip motion to the face region. The claimed effect of this machinery is that the model's pretrained prior fills in everything the weak conditions omit—hand pose, finger motion, and object orientation.

What would settle it

Fix one arm skeleton and one dot trajectory, then vary the object image across rigid, deformable, and articulated objects, measuring object consistency and hand-object contact as a function of dot-to-hand distance and object category; if consistency collapses when the dot sits outside the hand region or when the object is structurally complex, the sparse-condition sufficiency claim is falsified, and the paper's own limitation appendix already shows one such collapse.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the bottleneck in HOI video generation is not model capacity but input strength. The authors design a setting they call weakly conditioned HOI generation: the human is controlled by an arm-only skeleton with fingers optionally absent, and the object is controlled by a dot—its center point over time—rather than a bounding box, depth, or 3D model. They then show that a multimodal diffusion transformer (MMDiT), initialized from a pretrained video model and augmented with a parameter-space HOI adapter, can turn those sparse conditions, together with one reference human image and one object image, into five-second videos with temporally consistent and physically plausible interactions. Their experiments on a self-collected 100-clip test set report the best FID, FVD, and object-consistency scores among the compared methods, and a user study ranks the method first on interaction naturalness. The same machinery is used for text-conditioned motion and audio-driven lip sync, supporting the claim of a generic, multimodal-driven HOI framework.

Load-bearing premise

The load-bearing premise is that a single arm skeleton plus an object-center dot is enough information for the model's generative prior to fill in all the missing detail—other body motion, hand pose, object orientation, and grasp—and that this filled-in motion remains physically plausible.

Editorial extensions

If this is right

  • HOI generation no longer depends on actor-captured motion, depth, or per-object fine-tuning: a single human image, an object image, an arm skeleton, and a dot trajectory are the entire input.
  • Because object appearance comes from an input image plus semantic features rather than a learned object embedding, the same trained model can animate novel objects, including non-rigid ones, without retraining.
  • Text and audio can be layered onto the same weak pose and trajectory conditions, so one framework covers pose-driven animation, text-conditioned motion, and talking with lip sync.
  • Slight edits to the object dot change the interaction while the body pose stays fixed, which is what makes the interactive drag-and-drop demo a practical refinement tool.
  • The depth-aware curation pipeline turns ordinary internet video into HOI training data, which is the basis for the claim that the method generalizes beyond curated motion capture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the weak-condition premise is sound, the same sparse interface could extend to two-handed and multi-object scenes, where the generative prior would decide which hand contacts which object.
  • Editorial extension: the Appendix G dot-hand distance failure indicates the model implicitly learns a reachability region; an explicit distance-aware condition or training penalty could widen the usable range.
  • Editorial extension: because paste size acts as an implicit scale control, other appearance-injection video models could adopt the same latent-space paste mechanism as a general placement and size dial.
  • Editorial extension: the separate-motion-encoder advantage suggests that keeping human and object motion channels disentangled is a generally useful design for weak-condition video generation, not just for this framework.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces HunyuanVideo-HOMA, a weakly conditioned framework for human-object interaction (HOI) video generation. It takes a reference human image and an object image as appearance inputs, and drives animation with sparse arm skeleton sequences plus an object center-point dot trajectory, with text and audio as optional control signals. The architecture builds on HunyuanVideo's MMDiT: context fusion injects human and object appearance via channel and token concatenation in latent and token spaces, a parameter-initialized HOI adapter injects object features at multiple layers, and a facial cross-attention adapter handles audio-driven lip sync. Training is performed in three stages with a depth-aware data curation pipeline for collecting HOI videos. The paper claims state-of-the-art performance in interaction naturalness and generalization under weak supervision, supported by quantitative comparisons on a self-collected test set and the AnchorCrafter test set, along with ablations and user studies.

Significance. The proposed weak-condition paradigm is a genuinely useful research direction: if sparse inputs can reliably determine generated interactions, it would substantially lower the barrier to practical HOI video generation. The technical design is sensible, and the internal ablations in Table 2 consistently support the main architectural choices: dot trajectory outperforms bounding box and Gaussian dot, token concatenation and channel concatenation both contribute, the HOI adapter helps, and separate motion encoders beat a shared encoder. The three-stage training and depth-aware HOI data filtering are principled. However, the headline claim is currently not demonstrated for the actual weak-input setting, and the evaluation has self-referential elements. With additional experiments and careful reframing, the paper could be a solid contribution; in its present form, the central claim outruns the evidence.

major comments (3)
  1. [§4.1.3 and Abstract] The central claim of state-of-the-art performance under weak supervision is not supported by the reported experiments. The manuscript states that for a fair comparison 'our method uses the whole full pose for inference,' so the quantitative results in Table 1 and the user study in Table 3 evaluate the full-pose setting, not the weak condition defined in §3.1 (arm-only skeleton with optional finger exclusion and object-center dot trajectory). No numerical metric is reported for the weak-input configuration; the only evidence there is qualitative (Fig. 10), and Appendix G explicitly shows degradation when the dot is far from the hand. The authors should either report quantitative results under the weak condition, including comparisons with strong-condition baselines, or revise the abstract and conclusion to claim state-of-the-art performance in the full-pose setting.
  2. [Appendix G / §3.1] The paper's load-bearing assumption is that a sparse arm skeleton plus an object-center dot is sufficient for the model to infer hand pose, object orientation, and remaining body motion. Appendix G concedes that when the dot is far from the hand, 'the generated object tends to exhibit reduced consistency and interaction plausibility.' This limitation is stated qualitatively, with no analysis of how performance varies with dot-hand distance or trajectory complexity. Because the entire utility argument for weak supervision rests on this sufficiency, the paper needs a quantitative characterization of the supported input range, for example performance versus dot-hand distance and a comparison against a full-pose baseline on identical test cases. Without this, the claim that the weak condition is a reliable control signal is not established.
  3. [§4.1.2, §4.1.4, Table 1] The evaluation is partly self-referential and omits strong baselines. The OC metric and the AnchorCrafter benchmark come from Xu et al. (2024a), a paper with overlapping authors; the self-collected test set is filtered by the authors' own pipeline (§3.3) and may share distribution with the training data. Strong HOI methods cited in the paper—ManiVideo (Pang et al. 2024) and the fine-tuned AnchorCrafter variant on the self-collected test set—are missing from Table 1; AnchorCrafter appears only on its own test set in the appendix. The SOTA claim would be considerably strengthened by evaluating on an independently constructed benchmark, including ManiVideo and AnchorCrafter on the self-collected set (acknowledging that AnchorCrafter requires per-object fine-tuning), and reporting the AnchorCrafter test set results in the main text.
minor comments (6)
  1. [§4.1.2] The text uses 'AnchorCraft test set' in two places; this should be 'AnchorCrafter' for consistency with the method name and citations.
  2. [§3.2.2, Eq. (3)] The notation Z_i_obj = Z_obj|D_i is unclear; please define D_i and describe the paste operation explicitly.
  3. [Table 3 / §4.4] The user study uses only 15 users and reports no variance, confidence intervals, or significance tests; please describe the protocol and add error bars or a statistical comparison.
  4. [§4.3] All ablations are run at 512×512 resolution with 2,000 training steps, while the final model operates at 512×896 with 23,000 steps; a sentence on how representative these low-budget ablations are of the final model would help readers calibrate the results.
  5. [Appendix F] The long-video generation approach is described but not quantitatively evaluated; please add metrics or at least qualitative comparisons to shorter-video results.
  6. [Figure 21 caption] The caption contains typos: 'relationshuip' should be 'relationship' in both instances.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation circularity found; the weak-supervision evaluation gap is a validity concern, not a circular reduction.

full rationale

HunyuanVideo-HOMA is an empirical systems/architecture paper built on HunyuanVideo's MMDiT; it does not claim a formal derivation whose conclusion is identical to an input definition. The weak-condition formulation in Sec. 3.1 (sparse arm skeleton plus object-center dot trajectory) is a stated design choice, not an equation fitted to the reported outputs. The HOI adapter, context fusion, and audio adapter are trained components with ablations, and their quantitative comparison uses external baselines such as MimicMotion, VACE-14B, and EchoMimic-v2. Some evaluation infrastructure comes from the authors' prior work: the OC metric and AnchorCrafter test set are attributed to Xu et al. (2024a), which shares authors with the present paper, and the self-collected test set is also constructed by the authors. However, OC is a CLIP-based external similarity metric, the AnchorCrafter test set contains real videos and objects, and the main FID/FVD/HAS comparisons are computed from generated videos against independent data; these citations are not used as a uniqueness theorem or fitted constraint, so they are not load-bearing circularity. The most significant concern is that Sec. 4.1.3 states 'For a fair comparison, all compared methods utilize the original motion inputs, and our method uses the whole full pose for inference,' meaning the headline quantitative tables and user study evaluate the full-pose setting rather than the weak arm-plus-dot condition advertised in the abstract. Appendix G further concedes that dot-hand distance degrades object consistency and interaction plausibility. This is a mismatch between the claimed 'weak supervision' result and the evidence actually produced, which is a correctness/validity risk rather than a circular derivation: no metric or prediction is shown to be equivalent to its own training input by construction. Therefore the paper receives no circularity points.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The framework is an empirical system; the load-bearing inputs are the sparse-control assumption, the reuse of the pretrained generative prior, the depth-based validity rule for training and test data, and several hand-chosen hyperparameters (learning rate, step counts, RoPE offsets, resolution schedule). Internal ablations support the control representations and adapters, but external validity is only tested on self-curated data.

free parameters (5)
  • Learning rate = 1e-5
    Fixed learning rate across all training stages (Sec. 4.1.1); hand-chosen and affects convergence and final quality.
  • Training step counts = 16,000 / 2,000 / 5,000
    Stage durations chosen by hand (Sec. 4.1.1); the claimed convergence and quality depend on these budget allocations.
  • RoPE offsets for object tokens = -1 and -2
    Hand-chosen positional offsets for the temporal-concat object token (Eq. 2, offset=-1) and the HOI adapter object token (Eq. 6, offset=-2).
  • Depth filtering criterion
    The mean-depth similarity threshold between segmented object and hand is not specified (Sec. 3.3, Fig. 4); it controls which 140 hours of training and test videos are used.
  • Resolution schedule = 512x512 then 512x896
    Stage two trains at 512x512 and stage three at 512x896 (Sec. 3.3); a hand-chosen schedule that affects the final output quality.
assumptions (5)
  • standard math The flow matching objective (Lipman et al. 2022) is a valid training target for the video diffusion model.
    Invoked in Sec. 3.2.1 as the training objective without proof or alternative.
  • domain assumption The pretrained HunyuanVideo MMDiT weights provide a generative prior that can be repurposed through adapters with only small fine-tuning steps.
    Sec. 3.2.3 states the HOI adapter is initialized from pretrained MMDiT weights to reuse the model's priors; the whole efficiency claim rests on this.
  • domain assumption A single arm skeleton plus an object center dot is a sufficient control signal for the model to infer the rest of the body and object motion.
    Sec. 3.1 defines the weak condition; Appendix G shows degradation when the dot is far from the hand, so the assumption holds only in a limited input range.
  • domain assumption Similar mean depths of the segmented object and hand imply a valid physical human-object interaction.
    Sec. 3.3 Step III uses this rule to filter training and evaluation data, so the model rarely sees interactions that violate the depth rule.
  • domain assumption Average scores from 15 users are a reliable measure of interaction naturalness and video quality.
    Sec. 4.4 presents the user study with 15 raters and no statistical analysis; the conclusion of user-perceived superiority relies on this assumption.
invented entities (3)
  • HOI Adapter independent evidence
    purpose: Multi-layer injection of object appearance and position into the MMDiT at even-numbered layers.
    Ablation in Table 2 (w/o adapter: FID 62.09 vs 38.32, OC 82.01 vs 88.75) provides an internal falsifiable handle.
  • Audio cross-attention adapter independent evidence
    purpose: Injects audio features into the face region for lip synchronization.
    Sync-C scores (4.33 self-collected, 4.864 AnchorCrafter) provide a quantitative handle, though it is not compared against all baselines.
  • Dot trajectory object control independent evidence
    purpose: Represents object motion as a sequence of center points, replacing bounding boxes or dense trajectories.
    Ablations in Table 2 (w/ bbox FID 76.73, w/ Gaussian dot 86.54, proposed dot 38.32) support the representation choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation." pith.science (2026). https://pith.science/paper/OOLAVDRR

@misc{pith2026250608797,
  author       = {Pith},
  title        = {Pith review of: HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OOLAVDRR}},
  note         = {Machine review of arXiv:2506.08797}
}
read the original abstract

To address key limitations in human-object interaction (HOI) video generation -- specifically the reliance on curated motion data, limited generalization to novel objects/scenarios, and restricted accessibility -- we introduce HunyuanVideo-HOMA, a weakly conditioned multimodal-driven framework. HunyuanVideo-HOMA enhances controllability and reduces dependency on precise inputs through sparse, decoupled motion guidance. It encodes appearance and motion signals into the dual input space of a multimodal diffusion transformer (MMDiT), fusing them within a shared context space to synthesize temporally consistent and physically plausible interactions. To optimize training, we integrate a parameter-space HOI adapter initialized from pretrained MMDiT weights, preserving prior knowledge while enabling efficient adaptation, and a facial cross-attention adapter for anatomically accurate audio-driven lip synchronization. Extensive experiments confirm state-of-the-art performance in interaction naturalness and generalization under weak supervision. Finally, HunyuanVideo-HOMA demonstrates versatility in text-conditioned generation and interactive object manipulation, supported by a user-friendly demo interface. The project page is at https://bone-11.github.io/homa-page/.

Figures

Figures reproduced from arXiv: 2506.08797 by the authors.

Figure 1
Figure 1. HunyuanVideo-HOMA is a multimodal-driven human animation framework for human [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. We explore a weak HOI condition, where each component of the human pose can be [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Architecture of HunyuanVideo-HOMA. HunyuanVideo-HOMA is built upon the multi-modal diffusion transformer and is composed of context fusion, HOI adapter, and audio adapter. noise and progressively denoised over T timesteps to produce an estimate Zˆ 0. The final output video is then obtained via decoding, Y = Dec(Zˆ 0) = {y 1:n}. 3.2.2 Context Fusion To effectively leverage the model’s generative prior, we propose a c… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Depth-aware HOI Data Curation. to learn object control while incorporating pose and audio conditions. Etraj is initialized from Epose. Stage three increases the resolution to 512×896 to enhance visual quality and high-resolution generation capability. To address the sc…
Figure 5
Figure 5. Figure 5: Comparison with SOTAs on self-collected test set. More results can be found in the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Ablation results of different object motion. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Ablation results of context fusion. Input Images Input Pose Ours w/ single motion enc [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Ablation results of motion encoder.The qualitative results indicate that employing the [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Ablation results of HOI adapter. Without HOI adapter, the capability to track object motion [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Results with different types of poses. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Influence of Text Prompt. Reference human Object Human poses Object trajectory Reference human Object Human poses Object trajectory [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Results of Non-Rigid Objects. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Various object appearance and motion. Edited pose sequence 1 Edited pose sequence 2 (a) (b) [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Interactive Demo. Our interactive demo supports uploading either a single frame or a [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Qualitative results of AnchorCrafter test set. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Influence of the copy concat. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Network architecture of the MLLM. Reference Image Object Only LLaVA feature w/o LLaVA feature Ours [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Influence of LLaVA feature. D Variations of Different Trajectories We demonstrate the effect of varying object trajectories under the same human pose in [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Variations of different trajectories. Stage 1: Preprocessing and Basic Filtering. Raw videos are first passed through a series of preprocessing steps to remove low-quality or irrelevant content. These include frame rate conversion to standardize temporal resolution, s…
Figure 20
Figure 20. Figure 20: Whole data pipeline. G Limitations We observe that the distance between the dot and the hand significantly influences the final generation quality. As shown in [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]
Figure 21
Figure 21. Figure 21: Limitation. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_21.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A streaming image-to-video model that assigns block-specific sink/local memory and learned RoPE distance scaling preserves long human-object interactions under bounded latency.

  2. MVHOI: Bridge Multi-view Condition to Complex Human-Object Interaction Video Reenactment via 3D Foundation Model

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Using a 3D foundation model to produce viewpoint-aware anchors plus multi-view reference textures enables realistic human-object-interaction reenactment with large out-of-plane rotations.

  3. HOComp: Interaction-Aware Human-Object Composition

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A diffusion-transformer method that composes a foreground object into a human image with MLLM-chosen interaction regions, pose keypoint supervision, and appearance/background consistency losses, plus a new paired dataset.

Reference graph

Works this paper leans on

53 extracted references · 26 canonical work pages · cited by 3 Pith papers

  1. [1]

    Blattmann, T

    A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  2. [2]

    Caron, H

    M. Caron, H. Touvron, I. Misra, H. J \'e gou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pages 9650--9660, 2021

    work page 2021

  3. [3]

    Chang, Y

    D. Chang, Y. Shi, Q. Gao, H. Xu, J. Fu, G. Song, Q. Yan, Y. Zhu, X. Yang, and M. Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion. In Forty-first International Conference on Machine Learning (ICML), 2023

    work page 2023

  4. [4]

    Z. Chen, J. Cao, Z. Chen, Y. Li, and C. Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 2403--2410, 2025

    work page 2025

  5. [5]

    J. S. Chung and A. Zisserman. Out of time: automated lip sync in the wild. In Computer Vision--ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251--263. Springer, 2017

    work page 2016

  6. [6]

    Corona, A

    E. Corona, A. Zanfir, E. G. Bazavan, N. Kolotouros, T. Alldieck, and C. Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthesis. arXiv preprint arXiv:2403.08764, 2024

    arXiv 2024

  7. [7]

    Esser, S

    P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. M \"u ller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024

    2024

  8. [8]

    Y. Fan, Q. Yang, K. Wang, H. Zhou, Y. Li, H. Feng, Y. Wu, and J. Wang. Re-hold: Video hand object interaction reenactment via adaptive layout-instructed diffusion model. arXiv preprint arXiv:2503.16942, 2025

    work page Pith review arXiv 2025

Show all 53 references
  1. [9]

    Ghosh, R

    A. Ghosh, R. Dabral, V. Golyanik, C. Theobalt, and P. Slusallek. Imos: Intent-driven full-body motion synthesis for human-object interactions. In Computer Graphics Forum (CGF), volume 42, pages 1--12. Wiley Online Library, 2023

    2023

  2. [10]

    Heusel, H

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems (NeurIPS), 30, 2017

    2017

  3. [11]

    L. Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8153--8163, 2024

    2024

  4. [12]

    L. Hu, G. Wang, Z. Shen, X. Gao, D. Meng, L. Zhuo, P. Zhang, B. Zhang, and L. Bo. Animate anyone 2: High-fidelity character image animation with environment affordance. arXiv preprint arXiv:2502.06145, 2025

    2025 arXiv

  5. [13]

    Huang, F

    Z. Huang, F. Tang, Y. Zhang, X. Cun, J. Cao, J. Li, and T.-Y. Lee. Make-your-anchor: A diffusion-based 2d avatar generation framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6997--7006, 2024

    2024

  6. [14]

    X. Ji, X. Hu, Z. Xu, J. Zhu, C. Lin, Q. He, J. Zhang, D. Luo, Y. Chen, Q. Lin, et al. Sonic: Shifting focus to global audio perception in portrait animation. arXiv preprint arXiv:2411.16331, 2024

    2024 arXiv

  7. [15]

    Jiang, Z

    N. Jiang, Z. He, Z. Wang, H. Li, Y. Chen, S. Huang, and Y. Zhu. Autonomous character-scene interaction synthesis from text instruction. In SIGGRAPH Asia 2024 Conference Papers, pages 1--11, 2024 a

    2024

  8. [16]

    Jiang, Z

    N. Jiang, Z. Zhang, H. Li, X. Ma, Z. Wang, Y. Chen, T. Liu, Y. Zhu, and S. Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1737--1747, 2024 b

    2024

  9. [17]

    Jiang, Z

    Z. Jiang, Z. Han, C. Mao, J. Zhang, Y. Pan, and Y. Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.07598, 2025

    2025 arXiv

  10. [18]

    D. P. Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  11. [19]

    W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  12. [20]

    J. Li, J. Wu, and C. K. Liu. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG), 42 0 (6): 0 1--11, 2023

    2023

  13. [21]

    G. Lin, J. Jiang, C. Liang, T. Zhong, J. Yang, and Y. Zheng. Cyberhost: Taming audio-driven avatar diffusion model with region codebook attention. arXiv preprint arXiv:2409.01876, 2024

    2024 arXiv

  14. [22]

    G. Lin, J. Jiang, J. Yang, Z. Zheng, and C. Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models. arXiv preprint arXiv:2502.01061, 2025

    2025 arXiv

  15. [23]

    Lipman, R

    Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  16. [24]

    H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems (NeurIPS), 36: 0 34892--34916, 2023

    2023

  17. [25]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision (ECCV), pages 38--55. Springer, 2024

    2024

  18. [26]

    McFee, C

    B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto. librosa: Audio and music signal analysis in python. SciPy, 2015: 0 18--24, 2015

    2015

  19. [27]

    Y. Men, Y. Yao, M. Cui, and L. Bo. Mimo: Controllable character video synthesis with spatial decomposed modeling. arXiv preprint arXiv:2409.16160, 2024

    2024 arXiv

  20. [28]

    R. Meng, X. Zhang, Y. Li, and C. Ma. Echomimicv2: Towards striking, simplified, and semi-body human animation. arXiv preprint arXiv:2411.10061, 2024

    2024

  21. [29]

    Y. Pang, R. Shao, J. Zhang, H. Tu, Y. Liu, B. Zhou, H. Zhang, and Y. Liu. Manivideo: Generating hand-object manipulation video with dexterous and generalizable grasping. arXiv preprint arXiv:2412.16212, 2024

    2024 arXiv

  22. [30]

    X. Peng, Y. Xie, Z. Wu, V. Jampani, D. Sun, and H. Jiang. Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06553, 2023

    2023 arXiv

  23. [31]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML), pages 8748--8763. PMLR, 2021

    2021

  24. [32]

    N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R \"a dle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  25. [33]

    R. Shao, Y. Pang, Z. Zheng, J. Sun, and Y. Liu. Human4dit: 360-degree human video generation with 4d diffusion transformer. ACM Transactions on Graphics (TOG), 43 0 (6), 2024

    2024

  26. [34]

    J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  27. [35]

    L. Tian, Q. Wang, B. Zhang, and L. Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision (ECCV), pages 244--260. Springer, 2024

    2024

  28. [36]

    S. Tu, Z. Xing, X. Han, Z.-Q. Cheng, Q. Dai, C. Luo, and Z. Wu. Stableanimator: High-quality identity-preserving human image animation. arXiv preprint arXiv:2411.17697, 2024

    2024 arXiv

  29. [37]

    Unterthiner, S

    T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018

    2018 arXiv

  30. [38]

    A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025 a

    2025 arXiv

  31. [39]

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 a

    2024 arXiv

  32. [40]

    T. Wang, L. Li, K. Lin, Y. Zhai, C.-C. Lin, Z. Yang, H. Zhang, Z. Liu, and L. Wang. Disco: Disentangled control for realistic human dance generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9326--9336, 2024 b

    2024

  33. [41]

    X. Wang, S. Zhang, C. Gao, J. Wang, X. Zhou, Y. Zhang, L. Yan, and N. Sang. Unianimate: Taming unified video diffusion models for consistent human image animation. arXiv preprint arXiv:2406.01188, 2024 c

    2024 arXiv

  34. [42]

    X. Wang, S. Zhang, L. Tang, Y. Zhang, C. Gao, Y. Wang, and N. Sang. Unianimate-dit: Human image animation with large-scale video diffusion transformer. arXiv preprint arXiv:2504.11289, 2025 b

    2025

  35. [43]

    Z. Wang, Y. Li, Y. Zeng, Y. Fang, Y. Guo, W. Liu, J. Tan, K. Chen, T. Xue, B. Dai, et al. Humanvid: Demystifying training data for camera-controllable human image animation. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024 d

    2024

  36. [44]

    S. Xu, Z. Li, Y.-X. Wang, and L.-Y. Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 14928--14940, 2023

    2023

  37. [45]

    Z. Xu, Z. Huang, J. Cao, Y. Zhang, X. Cun, Q. Shuai, Y. Wang, L. Bao, J. Li, and F. Tang. Anchorcrafter: Animate cyberanchors saling your products via human-object interacting video generation. arXiv preprint arXiv:2411.17383, 2024 a

    2024 arXiv

  38. [46]

    Z. Xu, J. Zhang, J. H. Liew, H. Yan, J.-W. Liu, C. Zhang, J. Feng, and M. Z. Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1481--1490, 2024 b

    2024

  39. [47]

    Z. Xue, M. Luo, C. Chen, and K. Grauman. Hoi-swap: Swapping objects in videos with hand-object interaction awareness. arXiv preprint arXiv:2406.07754, 2024

    2024 arXiv

  40. [48]

    L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao. Depth anything v2. Advances in Neural Information Processing Systems (NeurIPS), 37: 0 21875--21911, 2024

    2024

  41. [49]

    Z. Yang, A. Zeng, C. Yuan, and Y. Li. Effective whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCVW), pages 4210--4220, 2023

    2023

  42. [50]

    Zhang, X

    W. Zhang, X. Cun, X. Wang, Y. Zhang, X. Shen, Y. Guo, Y. Shan, and F. Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2023

  43. [51]

    Zhang, J

    Y. Zhang, J. Gu, L.-W. Wang, H. Wang, J. Cheng, Y. Zhu, and F. Zou. Mimicmotion: High-quality human motion video generation with confidence-aware pose guidance. arXiv preprint arXiv:2406.19680, 2024

    2024 arXiv

  44. [52]

    Y. Zhou, Q. Wang, Y. Cai, and H. Yang. Allegro: Open the black box of commercial-level video generation model. arXiv preprint arXiv:2410.15458, 2024

    2024 arXiv

  45. [53]

    S. Zhu, J. L. Chen, Z. Dai, Y. Xu, X. Cao, Y. Yao, H. Zhu, and S. Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. In European Conference on Computer Vision (ECCV), 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.