Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformer

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A mask-weighted token blend lets a pretrained video diffusion transformer reenact hand-object interactions, including with objects it has never seen.

desk verdict A solid engineering paper with a plausible inpainting-based token blending mechanism, but the generalization claims outrun the evidence and the SOTA numbers are overstated. read the letter →

arxiv 2506.12847 v1 pith:AO3OQLGN submitted 2025-06-15 cs.GR cs.CV

Zhelun Shen , Chenming Wu , Junsheng Zhou , Chen Zhao , Kaisiyuan Wang , Hang Zhou , Yingying Li , Haocheng Feng
show 2 more authors
Wei He Jingdong Wang
This is my paper
classification cs.GRcs.CV
keywords hand-objectinteractionvideoreenactmentdiffusiontransformerinpaintingtokenprocessingobjectswappingin-the-wildgeneralizationkeyframegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that in-the-wild hand-object interaction reenactment — replacing the object in a video while keeping the hand motion — can be done by reusing a pretrained video diffusion transformer as-is, without adding any trainable parameters. The key move is to treat the replacement as an inpainting problem: the masked video tokens are blended with tokens from the reference object image, so the model's existing attention sees the new object as content to propagate rather than as a foreign conditioning signal. The authors support the claim with self-reenactment and cross-reenactment experiments on the Re-HOLD dataset and on a new e-commerce in-the-wild dataset, reporting that their method ranks first on 7 of 9 metrics and that it exceeds the previous state of the art especially for unseen objects. If the claim holds, it makes object swapping in hand videos practical for real-world deployment without per-video tuning or duplicated networks.

What carries the argument

The carrying mechanism is the unified inpainting-based token processing unit (Inp-TPU). It temporally extends the reference object image, spatially aligns it to the masked region, encodes both the masked video and the aligned reference into latent tokens, downsamples the mask, and blends the tokens with $X_{\text{cond}} = (1 - X_M)\cdot \tilde{X} + X_M \cdot X_{\text{ref}}^*$. This lets the pretrained DiT's attention parameters operate directly on the reference tokens, reusing established context perception without adding parameters. A two-stage design — an image DiT for the key frame and a video DiT for subsequent frames — plus a soft adaptive mask whose aspect ratio matches the target object provides shape control.

What would settle it

Take the trained model and attempt a cross-reenactment where the reference object has a shape or material far outside its training distribution (for example, a transparent glass, a torus, or a highly reflective object), then check whether the attention maps in the diffusion transformer's self-attention layers bind the inserted reference tokens to the hand region. If the object's tokens are ignored, blended into the background, or copied without adapting to the hand pose, the claimed generalization fails.

Watch

Extended reading notes

Core claim

Given a masked source video and a single reference object image, iDiT-HOI produces a reenacted video by first generating a key frame that inserts the object into the hand region, then generating subsequent frames conditioned on that key frame, both steps using the same Inp-TPU token process. The central discovery is that the conditional tokens can simply be computed as $X_{\text{cond}} = (1 - X_M)\cdot \tilde{X} + X_M \cdot X_{\text{ref}}^*$, a mask-weighted sum of the source video tokens and the spatially aligned reference tokens, and that this is enough for a fully fine-tuned pretrained diffusion transformer to keep the new object's identity while maintaining temporal coherence. The method is trained in a self-supervised reconstruction mode, yet it is claimed to generalize to genuinely new objects and unfamiliar in-the-wild videos, outperforming prior methods in most evaluated metrics.

Load-bearing premise

The load-bearing premise is that blending masked-video tokens with reference-object tokens is enough to make the pretrained video-diffusion model treat a never-seen object as content to propagate into the hand region, rather than as out-of-distribution noise; the model is only ever trained on self-reconstruction with the same object, so replacing it with a genuinely new object is an extrapolation.

Editorial extensions

If this is right

  • Object swapping in hand-object videos can be done without per-video fine-tuning or duplicated reference networks, lowering deployment cost.
  • Because subsequent frames are generated video-to-video, the last frame of one clip can seed the next, so long videos follow naturally from the same pipeline.
  • A pretrained image-to-video diffusion model can be repurposed for reference-image-controlled editing solely by choosing how tokens are blended at the input.
  • The method works from object images rather than text or layout control, making it more precise for tasks like e-commerce product placement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The approach suggests a general recipe for reference-conditioned video editing: any VAE-latent DiT with an inpainting mask could in principle inject arbitrary visual references through token blending, not just hand-held objects.
  • Because the model is trained only on self-reconstruction, its cross-object performance depends on attention robustness to out-of-distribution tokens; a stress test with radically different object geometries (transparent, non-rigid, highly textured) would clarify how far the generalization actually extends.
  • The same Inp-TPU could be applied to other body parts or to object-in-scene editing, such as replacing furniture or tools in a video, provided the mask and reference alignment are available.
  • If the reported advantage over the concurrent dual-stream baseline in the in-the-wild setting is real, it indicates that parameter duplication is not necessary for identity preservation and that masked-domain attention reuse is a stronger inductive bias than a separate reference stream.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents iDiT-HOI, a two-stage video diffusion transformer framework for hand-object interaction (HOI) reenactment. The core contribution is an inpainting-based token processing unit (Inp-TPU) that blends masked source-video latent tokens with reference-object latent tokens via Eq. (1): X_cond = (1 - X_M) * X_tilde + X_M * X_ref^*. The first stage generates a key frame by inserting the reference object into the hand region; the second stage generates the remaining frames conditioned on that key frame. The model is fully fine-tuned from pretrained Wan-14B and FLUX.1-dev models, and is trained in self-reconstruction mode on 19,000 video clips. At inference, the reference object can be replaced, enabling cross-reenactment. The authors evaluate on the Re-HOLD dataset and a self-collected HOI-ITW dataset, reporting quantitative metrics, ablations, and a user study, and claim state-of-the-art performance and strong generalization to unseen objects.

Significance. The proposed method is practically appealing: it avoids adding new network modules or parameters, uses a two-stage key-frame-then-video design for long-video generation, and builds on strong pretrained DiT backbones. If the generalization claims hold, the method could make in-the-wild object-swapping HOI reenactment practical. The authors provide machine-verifiable tables and ablations, but no released code or dataset, and the reported superiority is not consistently supported by the presented numbers. The central generalization claim rests on an extrapolation from self-reconstruction training to cross-object inference, with only a 10-video in-the-wild test set as direct evidence. Strengths of the paper include the formulation of a parameter-free token blending mechanism and a two-stage pipeline that naturally extends to long videos. The empirical support, however, is weaker than the claims and needs careful reanalysis.

major comments (5)
  1. [§4.3.1 and Table 1] The claim that the method "achieves top performance in 4 out of 6 metrics and ranks second in 1" is not supported by Table 1. In cross-reenactment, Ours* has subj. cons. 0.948, which is below Re-HOLD* (0.958) and AnimateAnyone (0.950); mot. smth. 0.995 ties with Re-HOLD* and V ACE*; PSNR 33.74 is below V ACE* (35.86). Only FID (12.07) is clearly best. In self-reenactment, mot. smth. is best (0.996) and subj. cons. ties with Re-HOLD* (0.958). A strict count gives at most 2 clear firsts and 1 clear second, plus a few ties. Similarly, the statement in §4.3.2 that the method achieves top performance in 7 out of 9 metrics is contradicted by Table 2, where on HOI-ITW temporal consistency V ACE (3.575) exceeds Ours (3.55). The reported counts should be corrected and any tie-breaking rule stated explicitly.
  2. [§3.3 and Eq. (1)] The central generalization claim is that Eq. (1), which linearly blends masked source tokens with reference-object tokens, lets the pretrained DiT propagate the novel object's identity into the masked hand region. In training, X_ref^* is the VAE latent of the source object from the same video; the model never sees a mismatch between the reference object identity and the target hand pose at training time. At inference, cross-reenactment inserts an entirely different object. The paper asserts that this works because the model "reuses established context perception capabilities," but the model is fully fine-tuned, so the pretrained capabilities are substantially overwritten, and no training signal enforces that inserted arbitrary object tokens are bound to the hand region. The only direct evidence is the 10-video HOI-ITW cross-reenactment set, with no analysis of how far those objects are from the training distribution, no attention-map analysis, and no contact-quality metric. This is a load-bearing assumption that should be tested explicitly or the generalization claim should be softened.
  3. [§4.2 and Table 2] The user study uses only 10 participants, reports no inter-rater agreement, no error bars, and no statistical significance tests. The text claims that the method "consistently outperforms competitors across all three evaluated dimensions," but Table 2 shows that on HOI-ITW temporal consistency V ACE (3.575) is higher than Ours (3.55). Moreover, subject consistency and motion smoothness do not measure hand-object contact quality, which is the central challenge of HOI reenactment. The evaluation section should either add sufficient statistics and a contact-aware metric or temper the claims to match what the evidence supports.
  4. [§4.1, Dataset] The HOI-ITW dataset is not released, and the statement that "we ensure that the data or similar data has not been seen by the trained models" is an unverifiable assertion. Because the training dataset is a private collection of 19,000 clips, a reader cannot check whether the test objects, scenes, or actions are truly out-of-distribution. For a paper whose main claim is generalization to unseen scenarios, the evaluation should either release the test data or provide a measurable distance between training and test distributions (for example, object-class overlap statistics or retrieval-based nearest-neighbor analysis).
  5. [§4.1 and §3.3] The claim of "reusing pretrained context perception without adding new trainable parameters" is only literally true in the sense that no new modules are introduced. The implementation section states that the models are trained in a full fine-tuning manner, meaning every existing parameter is updated. This is a different and weaker form of reuse than zero-shot prompting or frozen-backbone conditioning. The phrasing in the abstract and introduction should be revised to acknowledge that all pretrained parameters are fine-tuned, so the efficiency comparison with Re-HOLD and V ACE, which duplicate modules, should be stated relative to parameter count rather than to preservation of the original pretrained weights.
minor comments (6)
  1. [Abstract and Conclusion] The abstract and conclusion use absolute superlatives such as "outperforms existing methods" and "state-of-the-art performance" without the caveats that the quantitative table only partially supports; consider making these statements conditional on the corrected metric counts.
  2. [Figure 2 and Figure 3 captions] The method labels in the captions are missing spaces ("Re-HOLDOursVA C E"); this should be fixed for readability.
  3. [§2.2] The text uses both "V ASE" (reference [45]) and "V ACE" (reference [25]); these are different methods and the names should be clearly distinguished to avoid confusion.
  4. [Table 1] The footnote says "* denotes that metrics are computed on a fixed length of 81 frames," but only some methods are marked with the asterisk; the table would be clearer if all compared methods used the same evaluation protocol or if the marking were explained in the caption.
  5. [§4.4] The text reports a "27.08% drop in FID" when object information fusion is removed. Since a lower FID is better, this is actually an improvement in FID for the full model; the phrase should be clarified as an improvement or a degradation from the ablated model to the full model.
  6. [§4.3.2] The sentence "achieving only 1.1, 1.7, and 1.1 in 3 aspects of the user study" should include the table reference and the exact values from Table 2 (1.125, 1.7, 1.1) for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Eq. (1) is an input conditioning rule, not a fitted target; generalization claims are tested on held-out data.

full rationale

The paper's central derivation does not reduce to its own inputs. Equation (1), X_cond = (1 - X_M) * X_tilde + X_M * X_ref^*, is an explicit method for composing conditional tokens from the masked video and the encoded reference; it is not a fitted quantity, an evaluation metric, or a renamed target. Training is self-supervised reconstruction: both the masked video and the reference image are derived from the same source video, and the objective is to reconstruct that source video. Cross-reenactment is then evaluated on the Re-HOLD benchmark and on a separately collected HOI-ITW set, with reported metrics independent of any parameter fit to those metrics. The use of Re-HOLD as a baseline and dataset is a citation to prior work by overlapping authors, but it is not load-bearing for the method's derivation or for the claimed generalization. The train/inference gap -- the reference is from the same video during training but a novel object during inference -- is a genuine extrapolation risk and a legitimate correctness concern, but it is not circularity: the paper does not assert that Eq. (1) mathematically guarantees generalization, and it does not fit any parameter to the evaluation outcomes. No equation is shown to be equivalent to its target by construction, and no fitted parameter is renamed as a prediction. The central claim therefore has independent empirical content, and no specific circular step can be identified under the stated criteria.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method relies on pretrained Wan-14B and FLUX models, full fine-tuning on a private 19k-clip dataset, and a few hand-designed mask parameters (sqrt(2) enlargement, unspecified softness). No new physical entities are introduced. The core token blending is a simple weighted average, so there are no fitted constants in the mathematical sense. The main unknowns are training hyperparameters and the mask-generation constants.

free parameters (2)
  • Ellipse enlargement factor = sqrt(2) (approx 1.414)
    Section 3.5: the ellipse axes are enlarged by sqrt(2) to cover the oriented bounding box. This is a hand-chosen heuristic without an optimality derivation.
  • Mask softness = not reported
    Section 3.5 states the mask should be soft, but no parameter such as the blur kernel or standard deviation is given. This affects inpainting quality and is chosen by hand.
assumptions (4)
  • domain assumption Pretrained Wan-14B and FLUX models possess transferable context perception capabilities that survive full fine-tuning on 19k HOI clips.
    Invoked in Sections 3.2 and 3.3 as the basis for reusing attention parameters without adding architecture.
  • domain assumption Self-supervised reconstruction, where the reference object is the same video's own object, is a sufficient proxy for cross-reenactment with novel objects.
    The training objective in Section 3.2 reconstructs V from masked V and I_ref; generalization to unseen objects is an extrapolation from this setup.
  • domain assumption Oriented bounding box or mask annotations are available for all training and test videos.
    The method requires masks as input; the paper does not discuss how these are obtained in practice or whether errors propagate.
  • ad hoc to paper An ellipse with axes scaled by sqrt(2) and aspect-ratio matched to the target object will fully cover the source object region and support scale-aware inpainting.
    Section 3.5 introduces this geometric heuristic as the adaptive masking strategy without validating it across object shapes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformer." pith.science (2026). https://pith.science/paper/AO3OQLGN

@misc{pith2026250612847,
  author       = {Pith},
  title        = {Pith review of: iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AO3OQLGN}},
  note         = {Machine review of arXiv:2506.12847}
}
read the original abstract

Digital human video generation is gaining traction in fields like education and e-commerce, driven by advancements in head-body animation and lip-syncing technologies. However, realistic Hand-Object Interaction (HOI) - the complex dynamics between human hands and objects - continues to pose challenges. Generating natural and believable HOI reenactments is difficult due to issues such as occlusion between hands and objects, variations in object shapes and orientations, and the necessity for precise physical interactions, and importantly, the ability to generalize to unseen humans and objects. This paper presents a novel framework iDiT-HOI that enables in-the-wild HOI reenactment generation. Specifically, we propose a unified inpainting-based token process method, called Inp-TPU, with a two-stage video diffusion transformer (DiT) model. The first stage generates a key frame by inserting the designated object into the hand region, providing a reference for subsequent frames. The second stage ensures temporal coherence and fluidity in hand-object interactions. The key contribution of our method is to reuse the pretrained model's context perception capabilities without introducing additional parameters, enabling strong generalization to unseen objects and scenarios, and our proposed paradigm naturally supports long video generation. Comprehensive evaluations demonstrate that our approach outperforms existing methods, particularly in challenging real-world scenes, offering enhanced realism and more seamless hand-object interactions.

Figures

Figures reproduced from arXiv: 2506.12847 by the authors.

Figure 1
Figure 1. Our proposed method begins with Data Preprocessing, where object images and input video frames with corresponding masks [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Additional self-reenactment (left column) and cross-reenactment (right column) videos generated by our method and baseline [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Additional cross-reenactment videos generated by our method and baseline methods on the HOI-ITW dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Cross-reenactment results on the HOI-ITW dataset produced by our proposed method and other baseline methods. 4.2. Evaluation Setting We evaluate the proposed method from two perspectives: (1) Self-reenactment. The method reconstructs the original video using the masked…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer

    cs.CV 2026-07 conditional novelty 7.0 of 10

    Foundation-model HOI work is organized into eight geometric, semantic, and visual sub-priors that enter six reconstruction/generation tasks and three robot-transfer routes.

  2. AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment

    cs.CV 2026-07 conditional novelty 6.0 of 10

    AgentHOI generates human-object interaction videos from text plus one human image and one object image, using multi-agent action planning and implicit text-to-motion feature alignment inside a video diffusion model.

  3. StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A streaming image-to-video model that assigns block-specific sink/local memory and learned RoPE distance scaling preserves long human-object interactions under bounded latency.

Reference graph

Works this paper leans on

86 extracted references · 67 canonical work pages · cited by 3 Pith papers

  1. [1]

    Person image synthesis via de- noising diffusion model

    Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Mubarak Shah, and Fahad Shahbaz Khan. Person image synthesis via de- noising diffusion model. InCVPR, 2023. 2

    2023

  2. [2]

    Videopainter: Any- length video inpainting and editing with plug-and-play con- text control

    Yuxuan Bian, Zhaoyang Zhang, Xuan Ju, Mingdeng Cao, Liangbin Xie, Ying Shan, and Qiang Xu. Videopainter: Any- length video inpainting and editing with plug-and-play con- text control. InSIGGRAPH, 2025. 3

    2025

  3. [3]

    Smpler-x: Scaling up expressive human pose and shape estimation.NeurIPS, 2024

    Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qing- ping Sun, Wang Yanjun, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al. Smpler-x: Scaling up expressive human pose and shape estimation.NeurIPS, 2024. 2

    2024

  4. [4]

    Pix2video: Video editing using image diffusion

    Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. InCVPR,

  5. [5]

    Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion

    Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang, and Mo- hammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion. InICML, 2024. 2

    2024

  6. [6]

    Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023. 3

    arXiv

  7. [7]

    Control-a-video: Controllable text-to-video generation with diffusion models

    Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv, 2023. 3

    2023

  8. [8]

    Cg-hoi: Contact-guided 3d human-object interaction generation

    Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction generation. InCVPR, 2024. 3

    work page 2024

Show all 86 references
  1. [9]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. InCVPR, 2023. 3

    2023

  2. [10]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InForty-first international conference on machi...

  3. [11]

    Re-hold: Video hand object interaction reen- actment via adaptive layout-instructed diffusion model

    Yingying Fan, Quanwei Yang, Kaisiyuan Wang, Hang Zhou, Yingying Li, Haocheng Feng, Errui Ding, Yu Wu, and Jing- dong Wang. Re-hold: Video hand object interaction reen- actment via adaptive layout-instructed diffusion model. In CVPR, 2025. 1, 3, 7

    2025

  4. [12]

    Ccedit: Creative and controllable video editing via diffusion models

    Ruoyu Feng, Wenming Weng, Yanhui Wang, Yuhui Yuan, Jianmin Bao, Chong Luo, Zhibo Chen, and Baining Guo. Ccedit: Creative and controllable video editing via diffusion models. InCVPR, 2024. 3

    2024

  5. [13]

    Humandit: Pose-guided diffusion transformer for long- form human motion video generation.arXiv preprint arXiv:2502.04847, 2025

    Qijun Gan, Yi Ren, Chen Zhang, Zhenhui Ye, Pan Xie, Xiang Yin, Zehuan Yuan, Bingyue Peng, and Jianke Zhu. Humandit: Pose-guided diffusion transformer for long- form human motion video generation.arXiv preprint arXiv:2502.04847, 2025. 2

    2025 arXiv

  6. [14]

    Tokenflow: Consistent diffusion features for consistent video editing.arXiv, 2023

    Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing.arXiv, 2023. 2, 3

    2023

  7. [15]

    Emu video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factoriz- ing text-to-video generation by explicit image conditioning. arXiv, 2023. 3

    2023

  8. [16]

    Videoswap: Customized video subject swapping with interactive seman- tic point correspondence

    Yuchao Gu, Yuchao Zhou, Gu, Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Jun- hao Zhang, Mike Zheng Shou, and Kevin Tang. Videoswap: Customized video subject swapping with interactive seman- tic point correspondence. InCVPR, 2024. 2, 7

    2024

  9. [17]

    Talk-act: Enhance textural- awareness for 2d speaking avatar reenactment with diffusion model.SIGGRAPH Asia, 2024

    Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Hang Zhou, Shengyi He, Zhiliang Xu, Haocheng Feng, Errui Ding, Jing- dong Wang, Hongtao Xie, et al. Talk-act: Enhance textural- awareness for 2d speaking avatar reenactment with diffusion model.SIGGRAPH Asia, 2024. 2

    2024

  10. [18]

    Livepor- trait: Efficient portrait animation with stitching and retarget- ing control.arXiv, 2024

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control.arXiv, 2024. 1

    2024

  11. [19]

    Resolving 3d human pose ambigui- ties with 3d scene constraints

    Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. Resolving 3d human pose ambigui- ties with 3d scene constraints. InCVPR, 2019. 3

    2019

  12. [20]

    Populating 3d scenes by learning human-scene interaction

    Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J Black. Populating 3d scenes by learning human-scene interaction. InCVPR, 2021

    2021

  13. [21]

    Synthesizing physi- cal character-scene interactions

    Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physi- cal character-scene interactions. InSIGGRAPH, 2023. 3

    2023

  14. [22]

    Hand-object interaction image generation.NeurIPS,

    Hezhen Hu, Weilun Wang, Wengang Zhou, and Houqiang Li. Hand-object interaction image generation.NeurIPS,

  15. [23]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. InCVPR,

  16. [24]

    Animate anyone 2: High-fidelity character image animation with environment affordance.arXiv preprint arXiv:2502.06145, 2025

    Li Hu, Guangyuan Wang, Zhen Shen, Xin Gao, Dechao Meng, Lian Zhuo, Peng Zhang, Bang Zhang, and Liefeng Bo. Animate anyone 2: High-fidelity character image animation with environment affordance.arXiv preprint arXiv:2502.06145, 2025. 2

    2025 arXiv

  17. [25]

    Vace: All-in-one video creation and editing.arXiv preprint arXiv:2503.07598, 2025

    Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing.arXiv preprint arXiv:2503.07598, 2025. 3, 4, 7

    2025 arXiv

  18. [26]

    Dreampose: Fashion image-to-video synthesis via stable diffusion

    Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shlizerman. Dreampose: Fashion image-to-video synthesis via stable diffusion. InICCV,

  19. [27]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024. 3

    2024 arXiv

  20. [28]

    Anyv2v: A plug-and-play framework for any video- to-video editing tasks.arXiv, 2024

    Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen. Anyv2v: A plug-and-play framework for any video- to-video editing tasks.arXiv, 2024. 7

    2024

  21. [29]

    Flux.https://github.com/ black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/ black-forest-labs/flux, 2024. 3, 6

    2024

  22. [30]

    Lego: Learning egocentric ac- tion frame generation via visual instruction tuning

    Bolin Lai, Xiaoliang Dai, Lawrence Chen, Guan Pang, James M Rehg, and Miao Liu. Lego: Learning egocentric ac- tion frame generation via visual instruction tuning. InECCV,

  23. [31]

    Shape-aware text-driven lay- ered video editing

    Yao-Chih Lee, Ji-Ze Genevieve Jang, Yi-Ting Chen, Eliza- beth Qiu, and Jia-Bin Huang. Shape-aware text-driven lay- ered video editing. InCVPR, 2023. 2, 3

    2023

  24. [32]

    Generative om- nimatte: Learning to decompose video into layers.arXiv,

    Yao-Chih Lee, Erika Lu, Sarah Rumbley, Michal Geyer, Jia- Bin Huang, Tali Dekel, and Forrester Cole. Generative om- nimatte: Learning to decompose video into layers.arXiv,

  25. [33]

    Vidtome: Video token merging for zero-shot video editing

    Xirui Li, Chao Ma, Xiaokang Yang, and Ming-Hsuan Yang. Vidtome: Video token merging for zero-shot video editing. InCVPR, 2024. 2

    2024

  26. [34]

    Video generation from text

    Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. InAAAI, 2018. 3

    2018

  27. [35]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding.arXiv preprint arXiv:2405.08748, 2024

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding.arXiv preprint arXiv:2405.0874...

    2024 arXiv

  28. [36]

    Flowvid: Taming imperfect opti- cal flows for consistent video-to-video synthesis

    Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kun- peng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, et al. Flowvid: Taming imperfect opti- cal flows for consistent video-to-video synthesis. InCVPR,

  29. [37]

    Video-p2p: Video editing with cross-attention control

    Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. InCVPR, 2024. 3

    2024

  30. [38]

    Iterative ensemble training with anti-gradient control for mitigating memorization in diffusion models

    Xiao Liu, Xiaoliu Guan, Yu Wu, and Jiaxu Miao. Iterative ensemble training with anti-gradient control for mitigating memorization in diffusion models. InECCV, 2024. 3

    2024

  31. [39]

    Live speech por- traits: real-time photorealistic talking-head animation.TOG,

    Yuanxun Lu, Jinxiang Chai, and Xun Cao. Live speech por- traits: real-time photorealistic talking-head animation.TOG,

  32. [40]

    Dreamactor-m1: Holistic, expressive and robust human image animation with hybrid guidance, 2025

    Yuxuan Luo, Zhengkun Rong, Lizhen Wang, Longhao Zhang, Tianshu Hu, and Yongming Zhu. Dreamactor-m1: Holistic, expressive and robust human image animation with hybrid guidance, 2025. 2

    2025

  33. [41]

    Ace++: Instruction- based image creation and editing via context-aware content filling.arXiv preprint arXiv:2501.02487, 2025

    Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, and Jingren Zhou. Ace++: Instruction- based image creation and editing via context-aware content filling.arXiv preprint arXiv:2501.02487, 2025. 5

    2025 arXiv

  34. [42]

    Mimo: Controllable character video synthesis with spatial decomposed modeling

    Yifang Men, Yuan Yao, Miaomiao Cui, and Liefeng Bo. Mimo: Controllable character video synthesis with spatial decomposed modeling. InCVPR, 2025. 2

    2025

  35. [43]

    Sora: Creating video from text.https:// openai.com/sora/, 2024

    OpenAI. Sora: Creating video from text.https:// openai.com/sora/, 2024. Accessed: 2025-05-16. 3

    2024

  36. [44]

    Reconstruct- ing hands in 3d with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstruct- ing hands in 3d with transformers. InCVPR, 2024. 2

    2024

  37. [45]

    Vase: Object-centric appearance and shape manipulation of real videos.arXiv, 2024

    Elia Peruzzo, Vidit Goel, Dejia Xu, Xingqian Xu, Yifan Jiang, Zhangyang Wang, Humphrey Shi, and Nicu Sebe. Vase: Object-centric appearance and shape manipulation of real videos.arXiv, 2024. 3

    2024

  38. [46]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. InACM MM, 2020. 1

    2020

  39. [47]

    Fatezero: Fus- ing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fus- ing attentions for zero-shot text-based video editing. In CVPR, 2023. 2, 3

    2023

  40. [48]

    Stable diffu- sion 2 inpainting.https : / / huggingface

    Robin Rombach and Patrick Esser. Stable diffu- sion 2 inpainting.https : / / huggingface . co / stabilityai / stable - diffusion - 2 - inpainting., 2022. Accessed: 2025-05-16. 3

    2022

  41. [49]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, 2022. 3

    2022

  42. [50]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMICCAI, 2015. 2

    2015

  43. [51]

    Human4dit: 360-degree human video generation with 4d diffusion transformer.arXiv preprint arXiv:2405.17405, 2024

    Ruizhi Shao, Youxin Pang, Zerong Zheng, Jingxiang Sun, and Yebin Liu. Human4dit: 360-degree human video generation with 4d diffusion transformer.arXiv preprint arXiv:2405.17405, 2024. 2

    2024 arXiv

  44. [52]

    Edit-a-video: Single video editing with object-aware consistency

    Chaehun Shin, Heeseung Kim, Che Hyun Lee, Sang-gil Lee, and Sungroh Yoon. Edit-a-video: Single video editing with object-aware consistency. InACML, 2024. 3

    2024

  45. [53]

    Make-a-video: Text-to-video generation without text-video data.arXiv, 2022

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data.arXiv, 2022. 3

    2022

  46. [54]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv, 2024. 1

    2024

  47. [55]

    Attention is all you need.NeurIPS, 2017

    A Vaswani. Attention is all you need.NeurIPS, 2017. 2 11

    2017

  48. [56]

    Wan: Open and advanced large-scale video gen- erative models, 2025

    Team Wan. Wan: Open and advanced large-scale video gen- erative models, 2025. 3, 6

    2025

  49. [57]

    Robust video portrait reenact- ment via personalized representation quantization

    Kaisiyuan Wang, Changcheng Liang, Hang Zhou, Jiaxiang Tang, Qianyi Wu, Dongliang He, Zhibin Hong, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Robust video portrait reenact- ment via personalized representation quantization. InAAAI,

  50. [58]

    Efficient video portrait reenactment via grid-based codebook

    Kaisiyuan Wang, Hang Zhou, Qianyi Wu, Jiaxiang Tang, Zhiliang Xu, Borong Liang, Tianshu Hu, Errui Ding, Jing- tuo Liu, Ziwei Liu, et al. Efficient video portrait reenactment via grid-based codebook. InSIGGRAPH, 2023. 1

    2023

  51. [59]

    Diffusion in diffusion: Cyclic one-way diffusion for text-vision-conditioned generation.ICLR, 2023

    Ruoyu Wang, Yongqi Yang, Zhihao Qian, Ye Zhu, and Yu Wu. Diffusion in diffusion: Cyclic one-way diffusion for text-vision-conditioned generation.ICLR, 2023. 2

    2023

  52. [60]

    Disco: Disentangled control for realistic human dance generation

    Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung- Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. InCVPR, 2024. 2

    2024

  53. [61]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. InICCV, 2021. 1

    2021

  54. [62]

    Videocomposer: Compositional video synthesis with motion controllability.NeurIPS, 2024

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability.NeurIPS, 2024. 3

    2024

  55. [63]

    Unianimate-dit: Human image animation with large-scale video diffusion transformer.arXiv preprint arXiv:2504.11289, 2025

    Xiang Wang, Shiwei Zhang, Longxiang Tang, Yingya Zhang, Changxin Gao, Yuehuan Wang, and Nong Sang. Unianimate-dit: Human image animation with large-scale video diffusion transformer.arXiv preprint arXiv:2504.11289, 2025. 2

    2025

  56. [64]

    Latent image animator: Learning to animate im- ages via latent space navigation.arXiv, 2022

    Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate im- ages via latent space navigation.arXiv, 2022. 1

    2022

  57. [65]

    Foundationpose: Unified 6d pose estimation and tracking of novel objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. InCVPR, pages 17868–17879, 2024. 10

    2024

  58. [66]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In CVPR, 2023. 2, 3

    2023

  59. [67]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv, 2024. 1

    2024

  60. [68]

    Hoi-swap: Swapping objects in videos with hand-object in- teraction awareness.NeurIPS, 2024

    Zihui Xue, Mi Luo, Changan Chen, and Kristen Grauman. Hoi-swap: Swapping objects in videos with hand-object in- teraction awareness.NeurIPS, 2024. 1, 3, 5, 7

    2024

  61. [69]

    Showmaker: Creating high-fidelity 2d human video via fine-grained diffusion mod- eling.NeurIPS, 2024

    Quanwei Yang, Jiazhi Guan, Kaisiyuan Wang, Lingyun Yu, Wenqing Chu, Hang Zhou, ZhiQiang Feng, Haocheng Feng, Errui Ding, Jingdong Wang, et al. Showmaker: Creating high-fidelity 2d human video via fine-grained diffusion mod- eling.NeurIPS, 2024. 2

    2024

  62. [70]

    Rerender a video: Zero-shot text-guided video-to-video translation

    Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. InSIGGRAPH Asia, 2023. 2, 3

    2023

  63. [71]

    Effec- tive whole-body pose estimation with two-stages distillation

    Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effec- tive whole-body pose estimation with two-stages distillation. InCVPR, 2023. 2

    2023

  64. [72]

    Space-time diffusion features for zero-shot text-driven motion transfer

    Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. InCVPR, 2024. 2, 3

    2024

  65. [73]

    Diffusion-guided reconstruction of everyday hand- object interaction clips

    Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tul- siani. Diffusion-guided reconstruction of everyday hand- object interaction clips. InCVPR, 2023. 3

    2023

  66. [74]

    Affordance diffusion: Synthesizing hand-object inter- actions

    Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object inter- actions. InCVPR, 2023. 3

    2023

  67. [75]

    Moonshot: To- wards controllable video generation and editing with multi- modal conditions.arXiv, 2024

    David Junhao Zhang, Dongxu Li, Hung Le, Mike Zheng Shou, Caiming Xiong, and Doyen Sahoo. Moonshot: To- wards controllable video generation and editing with multi- modal conditions.arXiv, 2024. 3

    2024

  68. [76]

    Graspxl: Generating grasping motions for di- verse objects at scale

    Hui Zhang, Sammy Christen, Zicong Fan, Otmar Hilliges, and Jie Song. Graspxl: Generating grasping motions for di- verse objects at scale. InECCV, 2025. 3

    2025

  69. [77]

    Hoidiffusion: Generating realistic 3d hand-object interaction data

    Mengqi Zhang, Yang Fu, Zheng Ding, Sifei Liu, Zhuowen Tu, and Xiaolong Wang. Hoidiffusion: Generating realistic 3d hand-object interaction data. InCVPR, 2024. 3

    2024

  70. [78]

    Place: Proximity learning of articulation and con- tact in 3d environments

    Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang. Place: Proximity learning of articulation and con- tact in 3d environments. In3DV, 2020. 3

    2020

  71. [79]

    Generating 3d people in scenes with- out people

    Yan Zhang, Mohamed Hassan, Heiko Neumann, Michael J Black, and Siyu Tang. Generating 3d people in scenes with- out people. InCVPR, 2020. 3

    2020

  72. [80]

    Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance.arXiv preprint arXiv:2406.19680, 2024

    Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance.arXiv preprint arXiv:2406.19680, 2024. 2

    2024 arXiv

  73. [81]

    Avid: Any-length video inpainting with diffusion model

    Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. Avid: Any-length video inpainting with diffusion model. InCVPR, pages 7162–7172, 2024. 3

    2024

  74. [82]

    Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation

    Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation. InCVPR, 2021. 1

    2021

  75. [83]

    Realisdance: Equip controllable character ani- mation with realistic hands.arXiv, 2024

    Jingkai Zhou, Benzhi Wang, Weihua Chen, Jingqi Bai, Dongyang Li, Aixi Zhang, Hao Xu, Mingyang Yang, and Fan Wang. Realisdance: Equip controllable character ani- mation with realistic hands.arXiv, 2024. 2, 7

    2024

  76. [84]

    Discrete contrastive diffusion for cross-modal music and image generation.ICLR, 2022

    Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for cross-modal music and image generation.ICLR, 2022. 3

    2022

  77. [85]

    Cococo: Improving text-guided video inpainting for better consistency, controllability and compatibility

    Bojia Zi, Shihao Zhao, Xianbiao Qi, Jianan Wang, Yukai Shi, Qianyu Chen, Bin Liang, Rong Xiao, Kam-Fai Wong, and Lei Zhang. Cococo: Improving text-guided video inpainting for better consistency, controllability and compatibility. In AAAI, pages 11067–11076, 2025. 3

    2025

  78. [86]

    Cut-and-paste: Subject- driven video editing with attention control.Neural Networks,

    Zhichao Zuo, Zhao Zhang, Yan Luo, Yang Zhao, Haijun Zhang, Yi Yang, and Meng Wang. Cut-and-paste: Subject- driven video editing with attention control.Neural Networks,

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.