Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AniCrafter claims that by feeding a video diffusion model a coarse 3D-consistent avatar-background condition, open-domain human animation becomes a restoration task, yielding better identity preservation, body structure, and overall…

desk verdict A sensible engineering contribution — avatar-background conditioning from 3DGS — whose quantitative support mostly comes from a self-reenactment setting that stacks the deck; the cross-identity case is under-evaluated. read the letter →

arxiv 2505.20255 v2 pith:AGTAMJ3W submitted 2025-05-26 cs.CV

Muyao Niu , Mingdeng Cao , Yifan Zhan , Qingtian Zhu , Mingze Ma , Jiancheng Zhao , Yanhong Zeng , Zhihang Zhong
show 2 more authors
Xiao Sun Yinqiang Zheng
This is my paper
classification cs.CV
keywords avatar-backgroundconditioninghumanimageanimationvideodiffusionmodels3DGaussianSplattingavatarSMPL-Xposeimage-to-videogenerationopen-domaincharacterrestoration-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that open-domain human animation—inserting a given person into an arbitrary background video while following a motion sequence—becomes far more reliable if the diffusion model is given, as an extra condition, a coarse but 3D-consistent video of an animated avatar of that person blended into the target background. With this avatar-background condition, the generative task is reframed as restoration: the model refines the degraded avatar rendering and adds realistic non-rigid motion such as hair and loose clothing, rather than synthesizing the person from scratch. The authors report consistently better identity preservation, body structure, and overall quality than prior animation models on both static-background and dynamic-background benchmarks, along with a user study favoring their outputs. If this holds, the practical gain is that a single reference photo plus a motion sequence is enough to place a recognizable person into arbitrary dynamic scenes with stable structure.

What carries the argument

The load-bearing object is the avatar-background condition: a video formed by rendering a 3D Gaussian Splatting avatar (reconstructed from one reference frame by LHM), animating it with the estimated SMPL-X body poses, and alpha-blending the renderings, using the rasterizer's opacity mask, into a background video with the human region inpainted away. Three supporting mechanisms carry the argument: the SMPL-X mesh video supplies pure geometric motion; a compatible mask embedding, derived from blurred SMPL-X masks, tells the model to preserve background pixels and concentrate refinement on the human region; and 3D convolutional layers extract spatio-temporal features from both condition videos before they enter the frozen diffusion transformer, which is adapted with LoRA layers.

What would settle it

Take a fixed test set and replace the avatar-background condition with (a) a randomly selected person's avatar and (b) the ground-truth video itself; if the model's output quality does not monotonically improve from (a) to (b), the restoration framing is not the mechanism driving the gains. Alternatively, measure the pixel and pose alignment error between the rendered avatar and the target frame; a strong negative correlation between that error and output FVD would confirm the conditioning mechanism, while a flat relation would suggest the gains come from the extra network capacity or training data.

Watch

Extended reading notes

Core claim

The central discovery is that a pose-aligned coarse appearance condition—rather than only sparse structural cues like skeletons or meshes—determines the stability of diffusion-based character animation. The authors construct a single-image 3D Gaussian avatar, animate it with SMPL-X poses, alpha-blend the renderings into an inpainted background video, and feed this composite alongside the reference image and the SMPL-X mesh into an image-to-video diffusion transformer. Because the avatar rendering is grounded in one unified 3D representation, it supplies frame-consistent structure and approximate appearance; the diffusion model's job reduces to restoring high-frequency texture and synthesizing non-rigid dynamics. The paper argues this is why their model outperforms HumanVid, Champ, StableAnimator, and Unianimate-DiT on identity preservation, body structure, and overall quality, with the largest margins on dynamic-background open-domain scenarios.

Load-bearing premise

The method works only if the single-image 3D avatar reconstruction is faithful enough to the reference person and the estimated SMPL-X poses align well enough with the target motion that the blended avatar video is actually a degraded version of the desired output; if the avatar drifts, the diffusion model is guided by a misleading condition rather than a helpful one.

Editorial extensions

If this is right

  • A single reference photo is enough to animate a recognizable person into arbitrary dynamic backgrounds, since the avatar supplies pose-aligned appearance that the diffusion model then refines.
  • The restoration framing separates structure from dynamics: coarse 3D-consistent geometry comes from the avatar, while non-rigid motion like hair and loose garments is generated by the diffusion prior.
  • Because background pixels are preserved by the mask, the model can keep open-domain dynamic backgrounds largely intact while animating the foreground.
  • The method inherits the generalization of the base image-to-video model, so improvements in the base model should transfer directly to the animation quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same conditioning recipe might extend beyond humans to other articulated or deformable objects, wherever a parametric 3D model can be fitted to a reference image and driven by a pose sequence.
  • Since the paper itself notes depth misalignment in blending, a natural testable extension is to incorporate per-frame depth or a joint 3D reconstruction of person and scene, which could improve cases with occlusions or contact.
  • The training pipeline generates avatar-background conditions from monocular videos automatically, so scaling to larger and more diverse datasets is straightforward and may further close the gap to full scene-level control.
  • One could test the restoration premise directly by measuring how much output quality degrades as the avatar-to-ground-truth alignment error increases, which would isolate the contribution of the 3D-consistent condition from the rest of the architecture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AniCrafter, a video-diffusion model for human character animation that animates a given reference character into an open-domain dynamic background. The key novelty is an 'avatar-background' conditioning signal: the authors construct a 3D Gaussian Splatting avatar of the reference character with LHM, animate it with SMPL-X poses estimated from the target video, alpha-blend the renderings into an inpainted background video, and feed this composite video to a Wan2.1-I2V diffusion model along with the reference image and SMPL-X mesh sequence. The task is thereby reframed as restoring/refining the degraded avatar renderings while synthesizing non-rigid dynamics. The method is evaluated with self-reenactment experiments on static-background dancing videos and dynamic-background HumanVid videos, plus a user study on cross character-background samples.

Significance. The avatar-background conditioning idea is genuinely useful: it injects 3D-consistent structure and pose-aligned coarse appearance into the diffusion model, and the proposed mask-embedding strategy is compatible with the pretrained Wan2.1-I2V prior. The architecture is clearly described, the ablations isolate the main components, and the code is promised publicly. However, the central claim of open-domain cross-identity animation is not supported by the quantitative evidence: the only automated metrics are computed in a self-reenactment setting where the conditions are derived from the same target video, so the task is close to reconstruction. The cross-identity evaluation relies on a small user study with a single baseline and no statistical reporting. The method may well work, but the current paper overstates the strength of the evidence.

major comments (4)
  1. [§4.2, Tables 2–3] In the self-reenactment evaluation, all conditioning signals are derived from the target video itself: the 3DGS avatar is built from a randomly selected reference frame of that video, the SMPL-X parameters are estimated from that video, and the background is inpainted from that video (Sec. 3.1). Consequently, the large improvements in SSIM, PSNR, LPIPS, FID, and FVD over pose-only baselines are partly a copying/denoising effect rather than evidence of open-domain generalization. To support the paper's central claim, the authors should report automated identity and structure metrics in a true cross-identity protocol (novel character, novel motion, novel background), or at minimum clearly state that the quantitative tables measure reenactment fidelity, not cross-identity animation quality.
  2. [§4.1, Table 1] The cross character-background evaluation, which is the only experiment that directly tests the advertised use case, is a user study over 10 characters and 20 backgrounds with a single baseline (HumanVid). The paper reports neither the number of participants nor variance or significance information for the scores in Table 1. Given that this table carries the load of the cross-identity claim, the authors should add error bars, significance tests, and at least one additional modern baseline, or complement the user study with automated identity-preservation and structure metrics.
  3. [§4.3, Table 4] Table 4 does not consistently support the claim that the full model achieves optimal performance. On the HumanVid dataset, the variant 'w/o Mask' obtains higher PSNR (27.574 vs. 27.511) and lower LPIPS (0.0721 vs. 0.0734) than the full model, while the full model only wins on SSIM, FID, and FVD. The authors should either explain this trade-off or adjust the claim, and should report error bars or repeated-seed variance given that several differences are within 0.01–0.02 in SSIM/PSNR.
  4. [§3.3 and §4.2.2] The training procedure fine-tunes the model on a filtered HumanVid dataset of roughly 5,000 clips, and the dynamic-background self-reenactment test set is described as 100 test videos 'from the filtered HumanVid dataset.' The paper should explicitly confirm that the test videos are disjoint from the training clips and describe how the filtered test split was created. Without this clarification, there is a risk of train/test leakage that would inflate the Table 3 results.
minor comments (5)
  1. [§3.2] 'Casual V AE encoding' appears to be a typo and should read 'Causal VAE encoding'.
  2. [Title/Abstract] The title contains an extra space in 'A vatar-Background'; this should be fixed to 'Avatar-Background'.
  3. [§4.2, Tables 2–3] The table captions should state the number of frames, resolution, and whether metrics are computed over the full video or a subset, and should include error bars across videos.
  4. [§3.2] The phrase '0 denotes frames to be generated' has a missing space and should read '0 denotes frames to be generated.'
  5. [Figure 5 caption] The caption 'We respectively visualize DWPose and the avatar-background condition for HumanVid and our model' is ambiguous; it should clarify which visualization belongs to which method.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the avatar-background condition is an openly described self-supervised degradation of the target, and the central open-domain claim has independent, if limited, user-study evidence.

full rationale

The paper is an empirical systems paper and does not claim a formal derivation, so the circularity check reduces to whether any stated prediction is equivalent to its conditioning input by construction. The closest candidate is the self-reenactment evaluation (Sec. 4.2): the avatar-background condition is built from the same video used as ground truth, since Sec. 3.1 selects a random reference frame from that video for the LHM avatar, estimates SMPL-X from that video, and inpaints the background from that video. The paper openly describes this as making the condition 'basically ground-truth video itself' and the model's task a 'restoration.' That is a self-supervised degradation-recovery setup, not a hidden equivalence: the output is not logically forced, and the model must still synthesize high-frequency detail and non-rigid dynamics absent from the degraded input. The paper does not use the self-reenactment numbers as proof of open-domain insertion; the cross character-background setting (Sec. 4.1) is evaluated separately by a user study, with the explicit caveat that 'no ground-truth videos are available.' The stated limitation in Sec. 4.4 about missing scene-level 3D attributes is an honest scope restriction. No load-bearing self-citation was found: [3] and [37] are related-work citations, LHM [44] and SMPL-X [38] are external estimators, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The self-referential nature of the self-reenactment protocol is a limitation on what those quantitative tables can prove about open-domain generalization, but it is not a circular derivation.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The method is empirical, so the ledger lists the external tools and modeling choices the central claim depends on. No new physical entities are introduced, and no fitted constants are derived.

assumptions (5)
  • domain assumption Wan2.1-I2V-14B provides a strong image-to-video prior that can be adapted with LoRA without destroying generation ability.
    Sec 3.2 initializes from Wan2.1-I2V and adds LoRA while freezing original parameters; the method inherits the base model's quality.
  • domain assumption LHM reconstructs a high-fidelity animatable 3DGS avatar from a single image.
    Sec 3.1 uses LHM to build the avatar from a reference frame; if reconstruction fails for a novel character, the condition is poor. This is an external tool assumption.
  • domain assumption Monocular SMPL-X estimation returns poses accurate enough to animate the avatar and to serve as structural guidance.
    Sec 3.1 recovers SMPL-X parameters from the input video and uses them for both avatar animation and mesh rendering; pose quality is inherited from the unstated estimator.
  • domain assumption SAM-2 segmentation and ProPainter inpainting produce temporally coherent pseudo-backgrounds without visible artifacts.
    Sec 3.1 uses these tools to isolate the person and fill the background; failures would corrupt both the condition and the training target.
  • domain assumption The avatar-background composite is a valid degraded version of the desired output, differing only in texture detail and non-rigid dynamics.
    Sec 1 defines the diffusion task as restoration; this requires the avatar rendering to be pose-aligned and the alpha-blending to be seamless. This is the core assumption of the conditioning scheme.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models." pith.science (2026). https://pith.science/paper/AGTAMJ3W

@misc{pith2026250520255,
  author       = {Pith},
  title        = {Pith review of: AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AGTAMJ3W}},
  note         = {Machine review of arXiv:2505.20255}
}
read the original abstract

Recent advances in video diffusion models have significantly improved character animation techniques. However, current approaches rely on basic structural conditions such as DWPose or SMPL-X to animate character images, limiting their effectiveness in open-domain scenarios with dynamic backgrounds or challenging human poses. In this paper, we introduce \textbf{AniCrafter}, a diffusion-based human-centric animation model that can seamlessly integrate and animate a given character into open-domain dynamic backgrounds while following given human motion sequences. Built on cutting-edge Image-to-Video (I2V) diffusion architectures, our model incorporates an innovative ''avatar-background'' conditioning mechanism that reframes open-domain human-centric animation as a restoration task, enabling more stable and versatile animation outputs. Experimental results demonstrate the superior performance of our method. Codes are available at https://github.com/MyNiuuu/AniCrafter.

Figures

Figures reproduced from arXiv: 2505.20255 by the authors.

Figure 1
Figure 1. AniCrafter inserts and animates anyone into any open-domain dynamic backgrounds following given motion patterns. Besides the widely adopted SMPL-X mesh condition (blue boxes), an “avatar-background” condition signal (yellow boxes) is leveraged to achieve robust and diverse human-centric animation. Abstract Recent advances in video diffusion models have signifi￾cantly improved character animation techniques. However,… view at source ↗
Figure 2
Figure 2. The data processing pipeline for training our model. Given an in-the-wild video sequence, we first parse the human mask and use it to obtain the background videos through background inpainting. Next, we estimate the SMPL-X parameters from the monocular video. After that, we randomly select one reference image to generate a 3DGS avatar. Using the estimated SMPL-X parameters, we animate the human avatar and obtain the… view at source ↗
Figure 3
Figure 3. The Overall architecture of our proposed method. Instead of predicting the target video, our model predicts “reference image + target video” based on the pretrained Image-to-Video (I2V) diffusion transformer model. 3D Convolution layers and mask embeddings are leveraged to formulate useful spatial-temporal features from both the avatar-background video and the SMPL-X mesh video. presents the technical implementation… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Illustration of the mask strategy of (a) Wan2.1- I2V [57] and (b) our method. Our compatible mask embedding strategy follows the convention of Wan2.1, while further exploit￾ing the information of the conditioned 3DGS avatar video. to be generated. We extend this to a s…
Figure 5
Figure 5. Figure 5: Qualitative comparisons for cross character-background evaluation. We respectively visualize DWPose and the avatar￾background condition for HumanVid and our model. Zoom in for the best view. We show more results in the video demo. our method to better model both human …
Figure 6
Figure 6. Figure 6: Qualitative comparison results on the Dancing dataset. Zoom in for the best view. We show more results in the video demo [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison results on the HumanVid dataset. Zoom in for the best view. Our method achieves better structure preservation and ID consistency. Please refer to the video demo for more qualitative video results [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Qualitative ablation results for our model. Zoom in for the best view. We show more results in the video demo [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: More samples of qualitative comparisons for cross character-background evaluation. Zoom in for the best view. We show more results in the video demo. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 4DHumanDiff: Direct Text-to-4DGS Generation for Consistent 360-Degree Dynamic Humans

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A diffusion model trained on 60,000 fitted 4D Gaussian Splatting human clips generates text-prompted, view-consistent dynamic humans directly in 4D, over 10x faster than video-first pipelines.

  2. 3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Presents a scene-adaptive 3D human image animation framework using ground-adaptive motion retargeting and viewpoint-adaptive latent fusion to control human and camera trajectories, claiming improvements on two benchmarks.

  3. Proxy-GS: Unified Occlusion Priors for Training and Inference in Structured 3D Gaussian Splatting

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A proxy mesh rendered through hardware rasterization provides a cheap occlusion depth prior that culls hidden anchors at inference and guides densification at training, giving Octree-GS-like MLP splatting a 3 to 4x sp...

Reference graph

Works this paper leans on

77 extracted references · 17 canonical work pages · cited by 3 Pith papers

  1. [1]

    Ac3d: Analyzing and improving 3d camera control in video diffusion trans- formers

    Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Ali- aksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera control in video diffusion trans- formers. arXiv preprint arXiv:2411.18673, 2024. 3

    arXiv 2024

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1, 2

    arXiv 2023

  3. [3]

    Consistent Human Image and Video Generation with Spatially Conditioned Diffusion

    Mingdeng Cao, Chong Mou, Ziyang Yuan, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Yinqiang Zheng. Consis- tent human image and video generation with spatially condi- tioned diffusion. arXiv preprint arXiv:2412.14531, 2024. 1, 3

    work page Pith review arXiv 2024

  4. [4]

    Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion

    Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang, and Mo- hammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion. arXiv preprint arXiv:2311.12052, 2023

    arXiv 2023

  5. [5]

    X-dyna: Expressive dynamic hu- man image animation

    Di Chang, Hongyi Xu, You Xie, Yipeng Gao, Zhengfei Kuang, Shengqu Cai, Chenxu Zhang, Guoxian Song, Chao Wang, Yichun Shi, et al. X-dyna: Expressive dynamic hu- man image animation. arXiv preprint arXiv:2501.10021 ,

    arXiv

  6. [6]

    Motion-zero: Zero-shot mov- ing object control framework for diffusion-based video gen- eration

    Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He, Changbo Wang, and Yang Li. Motion-zero: Zero-shot mov- ing object control framework for diffusion-based video gen- eration. arXiv preprint arXiv:2401.10150, 2024. 3

    arXiv 2024

  7. [7]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 2

    arXiv

  8. [8]

    Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models. In Proceedings of the IEEE/CVF Conference 8 on Computer Vision and Pattern Recognition , pages 7310– 7320, 2024. 1, 2

    work page 2024

Show all 77 references
  1. [9]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 2

    2021

  2. [10]

    Mo- tion prompting: Controlling video generation with motion trajectories

    Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. Mo- tion prompting: Controlling video generation with motion trajectories. arXiv preprint arXiv:2412.02700, 2024. 3

    2024 arXiv

  3. [11]

    Densepose: Dense human pose estimation in the wild

    Rıza Alp G ¨uler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7297–7306, 2018. 1, 3

    2018

  4. [12]

    Ltx-video: Realtime video latent diffusion

    Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103 ,

    arXiv

  5. [13]

    Cameractrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 1, 3

    2024 arXiv

  6. [14]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  7. [15]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 7

    2017

  8. [16]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1, 2

    2020

  9. [17]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 2

    2022

  10. [18]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 2

    2022 arXiv

  11. [19]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 4

    2022

  12. [20]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 1, 3

    2024

  13. [21]

    Animate anyone 2: High-fidelity character image animation with environment affordance

    Li Hu, Guangyuan Wang, Zhen Shen, Xin Gao, Dechao Meng, Lian Zhuo, Peng Zhang, Bang Zhang, and Liefeng Bo. Animate anyone 2: High-fidelity character image animation with environment affordance. arXiv preprint arXiv:2502.06145, 2025. 1, 2, 3

    2025 arXiv

  14. [22]

    Fine-grained controllable video generation via object appearance and con- text

    Hsin-Ping Huang, Yu-Chuan Su, Deqing Sun, Lu Jiang, Xuhui Jia, Yukun Zhu, and Ming-Hsuan Yang. Fine-grained controllable video generation via object appearance and con- text. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3698–3708. IEEE, 2025. 3

    2025

  15. [23]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1,

  16. [24]

    Text2video-zero: Text- to-image diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text- to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Comput...

    2023

  17. [25]

    Auto-encoding vari- ational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding vari- ational bayes, 2013. 2

    2013

  18. [26]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 2

    2024 arXiv

  19. [27]

    Animateanything: Consistent and con- trollable animation for video generation

    Guojun Lei, Chi Wang, Hong Li, Rong Zhang, Yikai Wang, and Weiwei Xu. Animateanything: Consistent and con- trollable animation for video generation. arXiv preprint arXiv:2411.10836, 2024. 3

    2024 arXiv

  20. [28]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  21. [29]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding. arXiv preprint arXiv:2405.087...

    2024 arXiv

  22. [30]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models. arXiv preprint arXiv:2502.01061, 2025. 1, 3

    2025 arXiv

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 2

    2023

  24. [32]

    Animateanywhere: Rouse the background in human image animation

    Xiaoyu Liu, Mingshuai Yao, Yabo Zhang, Xianhui Lin, Peiran Ren, Xiaoming Li, Ming Liu, and Wangmeng Zuo. Animateanywhere: Rouse the background in human image animation. arXiv preprint arXiv:2504.19834, 2025. 2

    2025 arXiv

  25. [33]

    Smpl: A skinned multi- person linear model

    Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi- person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851–866. 2023. 1, 3, 4

    2023

  26. [34]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  27. [35]

    Trailblazer: Trajectory control for diffusion-based video 9 generation

    Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video 9 generation. In SIGGRAPH Asia 2024 Conference Papers , pages 1–11, 2024. 3

    2024

  28. [36]

    Mimo: Controllable character video synthesis with spatial decomposed modeling

    Yifang Men, Yuan Yao, Miaomiao Cui, and Liefeng Bo. Mimo: Controllable character video synthesis with spatial decomposed modeling. arXiv preprint arXiv:2409.16160 ,

    arXiv

  29. [37]

    Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model. In European Con- ference on Computer Vision, pages 111–128. Springer, 2024. 1, 3

    2024

  30. [38]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pa...

    2019

  31. [39]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 4195–4205,

  32. [40]

    Fatezero: Fus- ing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fus- ing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15932–15942, 2023. 3

    2023

  33. [41]

    Moviecharacter: A tuning-free framework for controllable character video syn- thesis

    Di Qiu, Zheng Chen, Rui Wang, Mingyuan Fan, Changqian Yu, Junshi Huang, and Xiang Wen. Moviecharacter: A tuning-free framework for controllable character video syn- thesis. arXiv preprint arXiv:2410.20974, 2024. 3

    2024 arXiv

  34. [42]

    Freetraj: Tuning-free tra- jectory control in video diffusion models

    Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free tra- jectory control in video diffusion models. arXiv preprint arXiv:2406.16863, 2024. 3

    2024 arXiv

  35. [43]

    Anigs: Animatable gaussian avatar from a single image with inconsistent gaussian reconstruction

    Lingteng Qiu, Shenhao Zhu, Qi Zuo, Xiaodong Gu, Yuan Dong, Junfei Zhang, Chao Xu, Zhe Li, Weihao Yuan, Liefeng Bo, et al. Anigs: Animatable gaussian avatar from a single image with inconsistent gaussian reconstruction. arXiv preprint arXiv:2412.02684, 2024. 2, 3

    2024 arXiv

  36. [44]

    Lhm: Large animatable human reconstruction model from a single image in seconds

    Lingteng Qiu, Xiaodong Gu, Peihao Li, Qi Zuo, Weichao Shen, Junfei Zhang, Kejie Qiu, Weihao Yuan, Guanying Chen, Zilong Dong, and Liefeng Bo. Lhm: Large animatable human reconstruction model from a single image in seconds. In arXiv preprint arXiv:2503.10625, 2025. 4

    2025 arXiv

  37. [45]

    Motionshop2

    Lingteng Qiu, Junfei Zhang, Chao Xu, Xiaodong Gu, Shenhao Zhu, Qi Zuo, Xiaodan Ye, Weichao Shen, Peihao Li, Weihao Yuan, Mingyang Huang, Gang Cheng, Zilong Dong, and Liefeng Bo. Motionshop2. https://aigc3d.github.io/motionshop-2/, 2025. 2, 3

    2025

  38. [46]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  39. [47]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2

    2020

  40. [48]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 4

    2024 arXiv

  41. [49]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2

    2022

  42. [50]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  43. [51]

    Invi: Object insertion in videos using off-the-shelf diffusion models

    Nirat Saini, Navaneeth Bodla, Ashish Shrivastava, Avinash Ravichandran, Xiao Zhang, Abhinav Shrivastava, and Bharat Singh. Invi: Object insertion in videos using off-the-shelf diffusion models. arXiv preprint arXiv:2407.10958, 2024. 3

    2024 arXiv

  44. [52]

    Motionstone: Decoupled motion in- tensity modulation with diffusion transformer for image-to- video generation

    Shuwei Shi, Biao Gong, Xi Chen, Dandan Zheng, Shuai Tan, Zizheng Yang, Yuyuan Li, Jingwen He, Kecheng Zheng, Jingdong Chen, et al. Motionstone: Decoupled motion in- tensity modulation with diffusion transformer for image-to- video generation. arXiv preprint arXiv:2412.05848 , 2024. 1

    2024 arXiv

  45. [53]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2

    2010 arXiv

  46. [54]

    Stableanimator: High- quality identity-preserving human image animation

    Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, and Zuxuan Wu. Stableanimator: High- quality identity-preserving human image animation. arXiv preprint arXiv:2411.17697, 2024. 6

    2024 arXiv

  47. [55]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 7

    2018 arXiv

  48. [56]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 2

    2017

  49. [57]

    Wan: Open and advanced large-scale video gen- erative models

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video gen- erative models. arXiv preprint arXiv:2503.20314, 2025. 2, 4, 5, 6

    2025 arXiv

  50. [58]

    Boximator: Generat- ing rich and controllable motions for video synthesis

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guo- qiang Wei, Liping Yuan, and Hang Li. Boximator: Generat- ing rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566, 2024. 3

    2024 arXiv

  51. [59]

    Disco: Disentangled control for realistic human dance generation

    Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung- Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,...

    2024

  52. [60]

    Unianimate: Taming unified video diffusion mod- els for consistent human image animation

    Xiang Wang, Shiwei Zhang, Changxin Gao, Jiayu Wang, Xiaoqiang Zhou, Yingya Zhang, Luxin Yan, and Nong Sang. Unianimate: Taming unified video diffusion mod- els for consistent human image animation. arXiv preprint arXiv:2406.01188, 2024. 1, 3

    2024 arXiv

  53. [61]

    Unianimate-dit: Human image animation with large-scale video diffusion transformer

    Xiang Wang, Shiwei Zhang, Longxiang Tang, Yingya Zhang, Changxin Gao, Yuehuan Wang, and Nong Sang. Unianimate-dit: Human image animation with large-scale video diffusion transformer. arXiv preprint arXiv:2504.11289, 2025. 5, 6

    2025

  54. [62]

    Humanvid: Demystifying training data for camera-controllable human image animation

    Zhenzhi Wang, Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan, Kai Chen, Tianfan Xue, Bo Dai, et al. Humanvid: Demystifying training data for camera-controllable human image animation. In The Thirty- eight Conference on Neural Information Processing Syst...

    2024

  55. [63]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Pa- pers, pages 1–11, 2024. 3

    2024

  56. [64]

    LIFe- GoM: Generalizable Human Rendering with Learned Iter- ative Feedback Over Multi-Resolution Gaussians-on-Mesh

    Jing Wen, Alex Schwing, and Shenlong Wang. LIFe- GoM: Generalizable Human Rendering with Learned Iter- ative Feedback Over Multi-Resolution Gaussians-on-Mesh. In ICLR, 2025. 2, 3

    2025

  57. [65]

    Mo- tionbooth: Motion-aware customized text-to-video genera- tion

    Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Mo- tionbooth: Motion-aware customized text-to-video genera- tion. arXiv preprint arXiv:2406.17758, 2024. 3

    2024 arXiv

  58. [66]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Eu- ropean Conference on Computer Vision , pages 399–417. Springer, 2024. 1, 2

    2024

  59. [67]

    Motioncanvas: Cinematic shot design with controllable image-to-video generation

    Jinbo Xing, Long Mai, Cusuh Ham, Jiahui Huang, Aniruddha Mahapatra, Chi-Wing Fu, Tien-Tsin Wong, and Feng Liu. Motioncanvas: Cinematic shot design with controllable image-to-video generation. arXiv preprint arXiv:2502.04299, 2025. 3

    2025 arXiv

  60. [68]

    Mo- tion dreamer: Realizing physically coherent video genera- tion through scene-aware motion reasoning

    Tianshuo Xu, Zhifei Chen, Leyi Wu, Hao Lu, Yuying Chen, Lihui Jiang, Bingbing Liu, and Yingcong Chen. Mo- tion dreamer: Realizing physically coherent video genera- tion through scene-aware motion reasoning. arXiv preprint arXiv:2412.00547, 2024. 3

    2024

  61. [69]

    Magicanimate: Temporally consistent human im- age animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human im- age animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  62. [70]

    Effec- tive whole-body pose estimation with two-stages distillation

    Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effec- tive whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4210–4220, 2023. 1, 3, 6

    2023

  63. [71]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 1, 2

    2024 arXiv

  64. [72]

    Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024

    Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024. 1, 3

    2024 arXiv

  65. [73]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 7

    2018

  66. [74]

    Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance

    Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance. arXiv preprint arXiv:2406.19680, 2024. 1, 3

    2024 arXiv

  67. [75]

    Motiondirector: Motion customization of text-to-video diffusion models

    Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Jun- hao Zhang, Jia-Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. In European Conference on Computer Vision, pages 273–290. Springer, 2024. 3

    2024

  68. [76]

    Propainter: Improving propagation and transformer for video inpainting

    Shangchen Zhou, Chongyi Li, Kelvin CK Chan, and Chen Change Loy. Propainter: Improving propagation and transformer for video inpainting. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10477–10486, 2023. 4

    2023

  69. [77]

    Champ: Controllable and consistent human image an- imation with 3d parametric guidance

    Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Zilong Dong, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image an- imation with 3d parametric guidance. In European Confer- ence on Computer Vision, pages 145–162. Springer, 2024. 1, ...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.