Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

StableAnimator++: Overcoming Pose Misalignment and Face Distortion for Human Image Animation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read StableAnimator++ preserves identity in human image animation even when poses are badly misaligned, and does it without face-swapping post-processing.

desk verdict A genuine systems advance in pose alignment for human animation, with honest ablations, but the headline misalignment benchmark matches its own training augmentation and the HJB theory is mostly decoration. read the letter →

arxiv 2507.15064 v1 pith:QZNR3SU7 submitted 2025-07-20 cs.CV cs.AI

Shuyuan Tu , Zhen Xing , Xintong Han , Zhi-Qi Cheng , Qi Dai , Chong Luo , Zuxuan Wu , Yu-Gang Jiang This is my paper
classification cs.CVcs.AI
keywords humanimageanimationidentitypreservationposemisalignmentvideodiffusionSVD-guidedalignmentdistribution-awareIDadapterHamilton-Jacobi-Bellmanfaceoptimizationtest-timeguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that identity-preserving human image animation can be made robust to the routine mismatch between a reference photo and a driving pose sequence—different body size, camera distance, or subject position—without falling back on face-swapping post-processing. Its proposal, StableAnimator++, is a video diffusion model with three training-time interventions and one inference-time intervention: a learnable SVD-guided aligner that predicts rotation, scale, and translation between reference and driving skeletons; a global content-aware Face Encoder; a distribution-aware ID Adapter that reconciles face and image cross-attention features before temporal layers; and an HJB-equation-based face optimization embedded in denoising. The paper reports that this system outperforms all compared methods in identity consistency (CSIM 0.831 on TikTok, 0.802 on MisAlign100) and video fidelity (FVD 122.47 and 384.27) while remaining a single end-to-end model. A sympathetic reader would care because the work targets the exact failure mode that makes current animation tools unusable outside studio-aligned inputs: the face drifts or the body distorts as soon as the reference and the driving video are not already framed alike.

What carries the argument

The main engine is the learnable pose aligner: a Transformer encoder receives the repeated reference keypoints concatenated with driving keypoints, while a parallel SVD branch computes an intermediate rigid alignment from centered keypoint covariance; a Pose Fusion Block injects the SVD-aligned features via cross-attention, and an MLP predicts $R'$, $S'$, $t'$ so the driven pose becomes $P_d^{\text{align}} = S'_d (R'_d \cdot P_d) + t'_d$, trained against ground-truth keypoint distance. The second mechanism is the distribution-aware ID Adapter: after separate cross-attention with image and face embeddings, it enforces $\frac{z^{\text{img}}_i - \mu_{\text{img}}}{\sigma_{\text{img}}} = \frac{z^{\text{face}}_i - \mu_{\text{face}}}{\sigma_{\text{face}}}$ and adds the renormalized face feature to the image feature before temporal layers. The third is the HJB inference module: treating the predicted sample $x_{\text{pred}}$ as the terminal state of an optimal control problem, the paper derives $c^*_t = \frac{r(x_1 - X_t)}{1 + r(1 - t)}$, shows the resulting controlled SDE coincides with the diffusion score SDE, and implements the optimal direction by minimizing ArcFace cosine distance between the decoded latent and the reference with Adam over ten steps inside the EDM sampler. The SVD-guided aligner carries the misalignment argument; the ID Adapter carries the temporal-interference argument; the HJB step carries the post-processing-free face argument.

What would settle it

A reader can check the central claim by rerunning StableAnimator++ and the strongest competitor on 100 naturally misaligned videos, measuring CSIM and FVD under identical conditions without synthetic transforms. If the CSIM gap over the best competitor collapses or the learnable aligner's contribution disappears when replaced by an oracle alignment that uses ground-truth poses, then the claim that learned SVD-guided alignment drives robustness would be falsified. A second check: the paper reports average Euclidean alignment error 0.105 for the full aligner versus 0.345 for SVD-only on MisAlign100; if forcing the predicted alignment to match an oracle rigid transform does not further improve CSIM, alignment error is not the limiting factor.

Watch

Extended reading notes

Core claim

The paper's central claim is that pose misalignment and face distortion in human image animation are better solved inside the diffusion pipeline than by post-processing. Concretely, it claims that a similarity transformation between reference keypoints and driving keypoints can be learned, with a single SVD solution used as cross-attention guidance to make learning tractable, and that this learned alignment reduces the Euclidean distance between aligned and ground-truth poses from 0.345 (SVD-only) or 0.430 (ControlNeXt's alignment) to 0.105 on the paper's MisAlign100 set. It further claims that identity is preserved by refining ArcFace face embeddings with a content-aware Face Encoder and by enforcing that the cross-attention output distributions from face and image embeddings match before every temporal layer, so that temporal modeling no longer erases identity. At inference, the paper claims that solving the HJB equation yields an optimal control direction for the denoising trajectory, and that ten Adam steps on the predicted sample against ArcFace cosine similarity progressively sharpens the face; this replaces FaceFusion-style post-processing and avoids its domain mismatch artifacts. The quantitative case rests on comparisons where all competitors were retrained on the authors' collected 5K-video dataset and evaluated on both TikTok and the self-built MisAlign100.

Load-bearing premise

The evaluation rests on a self-built test set (MisAlign100) in which misalignment is simulated by randomly scaling, rotating, and translating poses, and on retraining all competing methods on the authors' private 5K-video dataset; if those simulated transforms or that retraining protocol advantage the proposed modules, the reported margins may not transfer to naturally misaligned real-world footage.

Editorial extensions

If this is right

  • Animators can animate from a single reference photo even when the driving video is shot from a different distance or framing, without manually rescaling skeletons or running face-swap cleanup.
  • Removing face-swapping post-processing should eliminate the domain-mismatch artifacts, such as blurry or over-smoothed faces and degraded backgrounds, that the paper attributes to tools like FaceFusion.
  • The component design transfers: the paper shows that adding the Face Encoder, ID Adapter, and HJB optimization to a different video diffusion backbone preserves identity for text-to-video portrait generation.
  • Long animations of 500 or more frames remain stable under strong misalignment, which is the operating regime of dance and vlog animation products.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would be to evaluate on naturally misaligned footage rather than randomly transformed poses; the MisAlign100 construction leaves open whether the learned aligner generalizes to real-world scale and position gaps that are not rigid transforms.
  • The distribution-alignment step is not specific to faces: the same mean-variance matching between two cross-attention streams could protect clothing, object, or style conditioning from temporal-layer interference in other video generators.
  • The HJB optimizer is a per-sample test-time adaptation on the decoded face; one could ablate step count and learning rate per video to trade sharpness against latency, or apply the same control law to other identity-bearing regions such as hands.
  • Because every competitor was retrained on the authors' private 5K-video dataset, an open benchmark with released checkpoints would be needed to know whether the reported margins are due to the architecture or to the training data distribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes StableAnimator++, a video diffusion framework for human image animation that targets identity preservation under pose misalignment. The system adds a learnable SVD-guided pose-alignment module, a global content-aware face encoder, a distribution-aware ID adapter, and an inference-time face-optimization procedure presented as an approximate Hamilton-Jacobi-Bellman (HJB) solver integrated with the denoising process. The authors report state-of-the-art results on the TikTok dataset and on a self-constructed MisAlign100 benchmark, claiming improved CSIM and FVD without face-swapping post-processing.

Significance. If the empirical claims hold, the work is significant: it addresses a practical gap in human image animation, replaces brittle hand-crafted pose alignment with a learnable module, and eliminates face-swapping post-processing, which often degrades video quality. The modular design is tested across multiple backbones (Stable Video Diffusion, MagicAnimate, CogVideoX-I2V), and the ablations are reasonably thorough. However, the significance is currently constrained by three load-bearing weaknesses: the theoretical link between the HJB derivation and the actual algorithm is not established, the evaluation benchmark is private and its misalignment distribution matches the training augmentation, and no error bars or statistical tests accompany the headline numbers.

major comments (4)
  1. [§III-C, Algorithm 1, Eq. (15)-(24)] The derivation does not produce Algorithm 1. The chain from the HJB equation (15) through Eqs. (16)-(24) concludes with a score-based SDE that is structurally identical to a known reverse-time diffusion SDE, but Algorithm 1 performs an Adam-based optimization of a face-cosine loss on the predicted sample, with no explicit control variable, no denominator (1-t), no Brownian motion term, and no gradient log p(X_t) step. The claimed identification between solving the HJB equation and the diffusion denoising process is therefore not demonstrated by the given equations. Either provide a precise formulation in which the Algorithm 1 objective and Adam updates are the value function and control dynamics of the HJB problem, or reframe the method as a heuristic latent optimization; as written, the theoretical justification does not support the algorithm.
  2. [§III-A (Eq. 12) and §IV-B (MisAlign100)] The MisAlign100 benchmark is constructed by applying random scaling, rotation, and translation to driven poses, which is the exact family of transformations used to train the alignment module in Eq. (12). Since only StableAnimator++ is trained to predict these synthetic transformations, the large CSIM margins on MisAlign100 (0.802 vs. 0.391 for Animate-X) may be inflated by train/test distribution match rather than by genuine robustness to natural misalignment. The paper should report performance on a test set with naturally occurring body-size/position mismatches (not synthetically augmented), and should compare with a version of the competitors that also receives the same alignment augmentation during training, or at least with their released models on the same natural-misalignment videos.
  3. [Table I] The headline claims are presented without error bars, number of runs, or statistical significance. All quantitative comparisons are single point estimates, including FVD (122.47 on TikTok, 384.27 on MisAlign100), and the evaluation protocol is not fully specified (number of frames, resolution, start-frame selection, number of seeds). For a state-of-the-art claim, the authors should report mean±std over multiple assessment runs or per-video confidence intervals, and should make the evaluation code available so the protocol can be reproduced independently.
  4. [§IV-A, §IV-D, and Table I] The comparison protocol is not equally controlled across settings. TikTok results for competitors are cited from their original papers, while MisAlign100 results are obtained after retraining all competitors on the authors' private 5K-video dataset. No training code, checkpoints, or hyperparameters for the retrained competitors are released, so readers cannot verify that each competitor received an equal amount of tuning effort. This asymmetry makes the reported margins on MisAlign100 difficult to interpret as a fair comparison, and the absence of released code/checkpoints for the proposed method further limits reproducibility.
minor comments (4)
  1. [Eq. (6)] The denominator in the scale factor is ambiguous: it should be written as \sum_i \|X_d^i\|^2 (the squared norm of each centered keypoint vector), and the summation index and variable types should be defined consistently with Eq. (2).
  2. [Algorithm 1] The pseudocode calls the Adam optimizer on xop before setting xop.requires_grad = True; in standard frameworks, the requires_grad flag must be set before constructing the optimizer. Also, the loop variable γ_i is set to 0 at the top of the loop and then conditionally overwritten; the initial assignment is misleading even if functionally harmless.
  3. [Conclusion] There is a verb-tense error in the conclusion: 'StableAnimator++ then used off-the-shelf models to gain image and face embeddings' should be present tense, and 'StableAnimator introduced a Face Encoder' should refer to StableAnimator++.
  4. [Throughout] There is inconsistent spelling of ControlNeXt (also written 'ControlNext' in Table IX), and reference [4] lists the venue as 'EECV' instead of 'ECCV'. These should be corrected before publication.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the core modules are trained with supervised keypoint and reconstruction losses and evaluated on external-plus-self-constructed benchmarks; the HJB and MisAlign100 issues are framing and generalization concerns, not derivation-circularity.

full rationale

The paper's derivation chain is not circular in the sense defined here. The learnable alignment block is trained with a supervised keypoint loss (Eq. 12, Lalign = Avg(DisEuc(P_gt, S'*(R'·P_d)+t'))) against ground-truth poses, and the ID-related modules (Face Encoder, ID Adapter, PoseNet) are trained with the reconstruction loss in Eq. 25; neither of these training objectives is identical to the headline evaluation metrics, and the external TikTok benchmark plus comparisons against public methods provide independent support. The only notable self-citation is the use of StableAnimator [36] as the 'w/o Pose Align' ablation baseline in Section IV-E; this is a natural continuation of the authors' prior work, and the central quantitative comparison is against external methods such as AnimateAnyone, Champ, Unianimate, and Animate-X, so the self-citation is not load-bearing. The HJB section is a post-hoc theoretical framing: its derivation culminates in Eq. 24, which the paper itself identifies as structurally the same as the standard score-based SDE [14], and the actual face optimization in Algorithm 1 directly minimizes ArcFace cosine distance to the reference. This is an overclaim or a non-sequitur rather than a circular reduction, because the HJB formalism is not what forces the algorithm's behavior. The MisAlign100 benchmark is self-constructed by applying random scaling, rotation, and translation to driven poses (Section IV-B), which is the same transformation family used to train the alignment block (Section III-A); this is a train/test distribution-match concern that may inflate absolute robustness numbers, but it does not make any reported metric equal to an input by construction. The paper also acknowledges a limitation of its HJB optimization, namely that increasing optimization steps raises inference latency and excessive steps over-sharpen facial details (Section IV-E). Overall, no step of the paper's own equations reduces its output to its input, so no circular step is identified.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The system introduces new trainable modules (learnable alignment, Face Encoder, ID Adapter, HJB optimizer), but no new physical entities or conserved quantities. The main free parameters are hand-selected inference settings and the unspecified data augmentation ranges.

free parameters (4)
  • Number of HJB optimization steps k = 10
    Set empirically as a quality-latency trade-off (Sec. IV-E, Fig. 11); the paper states excessive steps over-sharpen details.
  • HJB optimization learning rate eta = not reported
    Used in Algorithm 1 (Adam on x_op); the paper defines eta but does not specify its value, so replication requires guessing it.
  • HJB optimization denoising window = first 10 denoising steps
    Applied only during the first 10 denoising steps (Sec. IV-A); the choice of which steps to optimize is hand-chosen and affects results.
  • Misalignment augmentation range = unspecified
    Random scaling/rotation/translation applied to pose sequences during alignment training and MisAlign100 construction (Sec. III-A, IV-B); ranges are not quantified, affecting benchmark difficulty.
assumptions (5)
  • domain assumption SVD-computed alignment provides a useful intermediate state for the learnable alignment layers
    Sec. III-A states that although the SVD output is not strictly accurate, injecting it as cross-attention guidance enhances the model's ability; this is an empirical assumption.
  • domain assumption Matching mean and variance of face and image cross-attention features brings them into the same domain
    Sec. III-B Eq. 14: the ID Adapter assumes that if the normalized features are equal, the distributions are basically in the same domain; this is a modeling assumption without proof.
  • ad hoc to paper The HJB optimal-control solution can be identified with the diffusion denoising process
    Sec. III-C: the derivation ends with an SDE 'structurally the same' as a diffusion SDE; the equivalence is asserted to justify Algorithm 1, but the actual objective (face similarity) does not appear in the HJB cost.
  • standard math EDM sampler and Stable Video Diffusion backbone provide the base distributions
    The entire method builds on EDM [65] and SVD [52], treated as given.
  • standard math Tweedie's formula and Pontryagin Maximum Principle
    Used in the HJB derivation (Eqs. 22, 17-19) as standard tools.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StableAnimator++: Overcoming Pose Misalignment and Face Distortion for Human Image Animation." pith.science (2026). https://pith.science/paper/QZNR3SU7

@misc{pith2026250715064,
  author       = {Pith},
  title        = {Pith review of: StableAnimator++: Overcoming Pose Misalignment and Face Distortion for Human Image Animation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZNR3SU7}},
  note         = {Machine review of arXiv:2507.15064}
}
read the original abstract

Current diffusion models for human image animation often struggle to maintain identity (ID) consistency, especially when the reference image and driving video differ significantly in body size or position. We introduce StableAnimator++, the first ID-preserving video diffusion framework with learnable pose alignment, capable of generating high-quality videos conditioned on a reference image and a pose sequence without any post-processing. Building upon a video diffusion model, StableAnimator++ contains carefully designed modules for both training and inference, striving for identity consistency. In particular, StableAnimator++ first uses learnable layers to predict the similarity transformation matrices between the reference image and the driven poses via injecting guidance from Singular Value Decomposition (SVD). These matrices align the driven poses with the reference image, mitigating misalignment to a great extent. StableAnimator++ then computes image and face embeddings using off-the-shelf encoders, refining the face embeddings via a global content-aware Face Encoder. To further maintain ID, we introduce a distribution-aware ID Adapter that counteracts interference caused by temporal layers while preserving ID via distribution alignment. During the inference stage, we propose a novel Hamilton-Jacobi-Bellman (HJB) based face optimization integrated into the denoising process, guiding the diffusion trajectory for enhanced facial fidelity. Experiments on benchmarks show the effectiveness of StableAnimator++ both qualitatively and quantitatively.

Figures

Figures reproduced from arXiv: 2507.15064 by the authors.

Figure 1
Figure 1. Pose-driven Human image animations generated by our StableAnima [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Architecture of StableAnimator++. (a) and (b) refer to the structure of the Face Encoder and each block in the U-Net. We first apply our learnable [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples from MisAlign100. The first row, the second row, and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Animation results generated by StableAnimator++. The images with red borders are the reference images. The presented pose skeletons are dramatically [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparisons with state-of-the-art methods. The skeletons in the third and fourth rows are misaligned with the reference image in terms [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ablations on core components of StableAnimator++. The presented [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Ablations on the alignment. The poses in the last two rows are [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Ablation study on face enhancement strategies. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Ablation study on different backbones. Reference Step=0 Step=4 Step=8 Step=10 [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Visual comparison of HJB-based face optimization at different [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Long animation results. The presented skeletons are misaligned with the reference image in body size and position. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: (a), (b), and (c) refer to multiple-person animation results, general [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Animate-X++: Universal Character Image Animation with Dynamic Backgrounds

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Animate-X++ turns cartoon images into pose-driven animations with text-controlled moving backgrounds, claiming state-of-the-art results on a new synthetic anthropomorphic benchmark.

Reference graph

Works this paper leans on

73 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    Disco: Disentangled control for realistic human dance generation,

    T. Wang, L. Li, K. Lin, Y . Zhai, C.-C. Lin, Z. Yang, H. Zhang, Z. Liu, and L. Wang, “Disco: Disentangled control for realistic human dance generation,” in CVPR, 2024

    work page 2024

  2. [2]

    Magicanimate: Temporally consistent human image animation using diffusion model,

    Z. Xu, J. Zhang, J. H. Liew, H. Yan, J.-W. Liu, C. Zhang, J. Feng, and M. Z. Shou, “Magicanimate: Temporally consistent human image animation using diffusion model,” in CVPR, 2024

    work page 2024

  3. [3]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation,

    L. Hu, “Animate anyone: Consistent and controllable image-to-video synthesis for character animation,” in CVPR, 2024

    work page 2024

  4. [4]

    Champ: Controllable and consistent human image animation with 3d parametric guidance,

    S. Zhu, J. L. Chen, Z. Dai, Y . Xu, X. Cao, Y . Yao, H. Zhu, and S. Zhu, “Champ: Controllable and consistent human image animation with 3d parametric guidance,” in EECV, 2024

    work page 2024

  5. [5]

    Unianimate: Taming unified video diffusion models for consistent human image animation,

    X. Wang, S. Zhang, C. Gao, J. Wang, X. Zhou, Y . Zhang, L. Yan, and N. Sang, “Unianimate: Taming unified video diffusion models for consistent human image animation,” arXiv preprint arXiv:2406.01188 , 2024

    arXiv 2024

  6. [6]

    Mimicmotion: High-quality human motion video generation with confidence-aware pose guidance,

    Y . Zhang, J. Gu, L.-W. Wang, H. Wang, J. Cheng, Y . Zhu, and F. Zou, “Mimicmotion: High-quality human motion video generation with confidence-aware pose guidance,” arXiv preprint arXiv:2406.19680 , 2024

    arXiv 2024

  7. [7]

    Controlnext: Powerful and efficient control for image and video generation,

    B. Peng, J. Wang, Y . Zhang, W. Li, M.-C. Yang, and J. Jia, “Controlnext: Powerful and efficient control for image and video generation,” arXiv preprint arXiv:2408.06070, 2024

    arXiv 2024

  8. [8]

    Animate-x: Universal character image animation with enhanced motion representation,

    S. Tan, B. Gong, X. Wang, S. Zhang, D. Zheng, R. Zheng, K. Zheng, J. Chen, and M. Yang, “Animate-x: Universal character image animation with enhanced motion representation,” in ICLR, 2025

    work page 2025

Show all 73 references
  1. [9]

    Multiple biological granularities network for person re-identification,

    S. Tu, T. Guan, and L. Kuang, “Multiple biological granularities network for person re-identification,” in ICMR, 2022

    2022

  2. [10]

    Implicit temporal modeling with learnable alignment for video recognition,

    S. Tu, Q. Dai, Z. Wu, Z.-Q. Cheng, H. Hu, and Y .-G. Jiang, “Implicit temporal modeling with learnable alignment for video recognition,” in ICCV, 2023

    2023

  3. [11]

    Diffusion models beat gans on image synthesis,

    P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” in NeurIPS, 2021

    2021

  4. [12]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in NeurIPS, 2020

    2020

  5. [13]

    Cascaded diffusion models for high fidelity image generation,

    J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans, “Cascaded diffusion models for high fidelity image generation,” JMLR, 2022

    2022

  6. [14]

    Score-based generative modeling through stochastic differ- ential equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differ- ential equations,” in ICLR, 2021

    2021

  7. [15]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in ICLR, 2021

    2021

  8. [16]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in CVPR, 2022

    2022

  9. [17]

    Sdedit: Guided image synthesis and editing with stochastic differential equations,

    C. Meng, Y . He, Y . Song, J. Song, J. Wu, J.-Y . Zhu, and S. Ermon, “Sdedit: Guided image synthesis and editing with stochastic differential equations,” in ICLR, 2021

    2021

  10. [18]

    Plug-and-play diffusion features for text-driven image-to-image translation,

    N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, “Plug-and-play diffusion features for text-driven image-to-image translation,” in CVPR, 2023

    2023

  11. [19]

    Genrec: Unifying video generation and recognition with diffusion models,

    Z. Weng, X. Yang, Z. Xing, Z. Wu, and Y .-G. Jiang, “Genrec: Unifying video generation and recognition with diffusion models,” arXiv preprint arXiv:2408.15241, 2024

    2024 arXiv

  12. [20]

    A survey on video diffusion models,

    Z. Xing, Q. Feng, H. Chen, Q. Dai, H. Hu, H. Xu, Z. Wu, and Y .-G. Jiang, “A survey on video diffusion models,” ACM Computing Surveys , vol. 57, no. 2, pp. 1–42, 2024

    2024

  13. [21]

    Simda: Simple diffusion adapter for efficient video generation,

    Z. Xing, Q. Dai, H. Hu, Z. Wu, and Y .-G. Jiang, “Simda: Simple diffusion adapter for efficient video generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 7827–7839

    2024

  14. [22]

    Aid: Adapting image2video diffusion models for instruction-guided video prediction,

    Z. Xing, Q. Dai, Z. Weng, Z. Wu, and Y .-G. Jiang, “Aid: Adapting image2video diffusion models for instruction-guided video prediction,” arXiv preprint arXiv:2406.06465 , 2024

    2024 arXiv

  15. [23]

    Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance,

    Q. Li, Z. Xing, R. Wang, H. Zhang, Q. Dai, and Z. Wu, “Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance,” arXiv preprint arXiv:2503.16421 , 2025

    2025

  16. [24]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models,

    H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang, “Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models,” arXiv preprint arxiv:2308.06721 , 2023

    2023 arXiv

  17. [25]

    Instantid: Zero-shot identity-preserving generation in seconds,

    Q. Wang, X. Bai, H. Wang, Z. Qin, and A. Chen, “Instantid: Zero-shot identity-preserving generation in seconds,” arXiv preprint arXiv:2401.07519, 2024. JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 13

    2024 arXiv

  18. [26]

    Consistentid: Portrait generation with multimodal fine-grained identity preserving,

    J. Huang, X. Dong, W. Song, H. Li, J. Zhou, Y . Cheng, S. Liao, L. Chen, Y . Yan, S. Liaoet al., “Consistentid: Portrait generation with multimodal fine-grained identity preserving,” arXiv preprint arXiv:2404.16771 , 2024

    2024 arXiv

  19. [27]

    Pulid: Pure and lightning id customization via contrastive alignment,

    Z. Guo, Y . Wu, Z. Chen, L. Chen, and Q. He, “Pulid: Pure and lightning id customization via contrastive alignment,” in NeurIPS, 2024

    2024

  20. [28]

    Facefusion,

    R. Henry, “Facefusion,” https://github.com/facefusion/facefusion, 2024

    2024

  21. [29]

    Towards real-world blind face restoration with generative facial prior,

    X. Wang, Y . Li, H. Zhang, and Y . Shan, “Towards real-world blind face restoration with generative facial prior,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021

    2021

  22. [30]

    Towards robust blind face restoration with codebook lookup transformer,

    S. Zhou, K. C. Chan, C. Li, and C. C. Loy, “Towards robust blind face restoration with codebook lookup transformer,” in NeurIPS, 2022

    2022

  23. [31]

    Motioneditor: Editing video motion via content-aware diffusion,

    S. Tu, Q. Dai, Z.-Q. Cheng, H. Hu, X. Han, Z. Wu, and Y .-G. Jiang, “Motioneditor: Editing video motion via content-aware diffusion,” in CVPR, 2024

    2024

  24. [32]

    Arcface: Additive angular margin loss for deep face recognition,

    J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in CVPR, 2019

    2019

  25. [33]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in ICML, 2021

    2021

  26. [34]

    Bardi, I

    M. Bardi, I. C. Dolcetta et al. , Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations . Springer, 1997

    1997

  27. [35]

    Stochastic hamilton–jacobi–bellman equations,

    S. Peng, “Stochastic hamilton–jacobi–bellman equations,” SIAM Journal on Control and Optimization , 1992

    1992

  28. [36]

    Sta- bleanimator: High-quality identity-preserving human image animation,

    S. Tu, Z. Xing, X. Han, Z.-Q. Cheng, Q. Dai, C. Luo, and Z. Wu, “Sta- bleanimator: High-quality identity-preserving human image animation,” arXiv preprint arXiv:2411.17697 , 2024

    2024 arXiv

  29. [37]

    Improved denoising diffusion proba- bilistic models,

    A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion proba- bilistic models,” in ICML, 2021

    2021

  30. [38]

    Prompt-to-prompt image editing with cross attention control,

    A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y . Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” arXiv preprint arXiv:2208.01626 , 2022

    2022 arXiv

  31. [39]

    Make-a-video: Text-to-video generation without text-video data,

    U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni et al. , “Make-a-video: Text-to-video generation without text-video data,” arXiv preprint arXiv:2209.14792 , 2022

    2022 arXiv

  32. [40]

    Animatediff: Animate your personalized text-to- image diffusion models without specific tuning,

    Y . Guo, C. Yang, A. Rao, Z. Liang, Y . Wang, Y . Qiao, M. Agrawala, D. Lin, and B. Dai, “Animatediff: Animate your personalized text-to- image diffusion models without specific tuning,” in ICLR, 2024

    2024

  33. [41]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,

    J. Z. Wu, Y . Ge, X. Wang, S. W. Lei, Y . Gu, Y . Shi, W. Hsu, Y . Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” in CVPR, 2023

    2023

  34. [42]

    Magicvideo-v2: Multi-stage high-aesthetic video generation,

    W. Wang, J. Liu, Z. Lin, J. Yan, S. Chen, C. Low, T. Hoang, J. Wu, J. H. Liew, H. Yan et al. , “Magicvideo-v2: Multi-stage high-aesthetic video generation,” arXiv preprint arXiv:2401.04468 , 2024

    2024 arXiv

  35. [43]

    Video generation models as world simulators,

    T. Brooks, B. Peebles, C. Holmes, W. DePue, Y . Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh, “Video generation models as world simulators,” 2024. [Online]. Available: https: //openai.com/research/video-generation-models-as-world-simulators

    2024

  36. [44]

    Motionfollower: Editing video motion via lightweight score-guided diffusion,

    S. Tu, Q. Dai, Z. Zhang, S. Xie, Z.-Q. Cheng, C. Luo, X. Han, Z. Wu, and Y .-G. Jiang, “Motionfollower: Editing video motion via lightweight score-guided diffusion,” arXiv preprint arXiv:2405.20325 , 2024

    2024 arXiv

  37. [45]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in ICCV, 2023

    2023

  38. [46]

    Videogpt: Video gener- ation using vq-vae and transformers,

    W. Yan, Y . Zhang, P. Abbeel, and A. Srinivas, “Videogpt: Video gener- ation using vq-vae and transformers,” arXiv preprint arXiv:2104.10157 , 2021

    2021 arXiv

  39. [47]

    Magvit: Masked generative video transformer,

    L. Yu, Y . Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang, Y . Hao, I. Essa et al. , “Magvit: Masked generative video transformer,” in CVPR, 2023, pp. 10 459–10 469

    2023

  40. [48]

    Latte: Latent diffusion transformer for video generation,

    X. Ma, Y . Wang, G. Jia, X. Chen, Z. Liu, Y .-F. Li, C. Chen, and Y . Qiao, “Latte: Latent diffusion transformer for video generation,”arXiv preprint arXiv:2401.03048, 2024

    2024 arXiv

  41. [49]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models,

    F. Bao, C. Xiang, G. Yue, G. He, H. Zhu, K. Zheng, M. Zhao, S. Liu, Y . Wang, and J. Zhu, “Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models,” arXiv preprint arXiv:2405.04233, 2024

    2024 arXiv

  42. [50]

    Cogvideo: Large- scale pretraining for text-to-video generation via transformers,

    W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang, “Cogvideo: Large- scale pretraining for text-to-video generation via transformers,” arXiv preprint arXiv:2205.15868, 2022

    2022 arXiv

  43. [51]

    Hunyuanvideo: A systematic framework for large video generative models,

    W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang et al. , “Hunyuanvideo: A systematic framework for large video generative models,” arXiv preprint arXiv:2412.03603 , 2024

    2024 arXiv

  44. [52]

    Stable video diffusion: Scaling latent video diffusion models to large datasets,

    A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y . Levi, Z. English, V . V oleti, A. Lettset al. , “Stable video diffusion: Scaling latent video diffusion models to large datasets,” arXiv preprint arXiv:2311.15127, 2023

    2023 arXiv

  45. [53]

    First order motion model for image animation,

    A. Siarohin, S. Lathuili `ere, S. Tulyakov, E. Ricci, and N. Sebe, “First order motion model for image animation,” in NeurIPS, 2019

    2019

  46. [54]

    Motion representations for articulated animation,

    A. Siarohin, O. J. Woodford, J. Ren, M. Chai, and S. Tulyakov, “Motion representations for articulated animation,” in CVPR, 2021

    2021

  47. [55]

    Few-shot human motion transfer by personalized geometry and texture modeling,

    Z. Huang, X. Han, J. Xu, and T. Zhang, “Few-shot human motion transfer by personalized geometry and texture modeling,” in CVPR, 2021

    2021

  48. [56]

    Generative adversarial networks,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial networks,” Communications of the ACM , 2020

    2020

  49. [57]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,” in ICML, 2024

    2024

  50. [58]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in ICLR, 2021

    2021

  51. [59]

    Photomaker: Customizing realistic human photos via stacked id embedding,

    Z. Li, M. Cao, X. Wang, Z. Qi, M.-M. Cheng, and Y . Shan, “Photomaker: Customizing realistic human photos via stacked id embedding,” in CVPR, 2024

    2024

  52. [60]

    Facestudio: Put your face everywhere in seconds,

    Y . Yan, C. Zhang, R. Wang, Y . Zhou, G. Zhang, P. Cheng, G. Yu, and B. Fu, “Facestudio: Put your face everywhere in seconds,”arXiv preprint arXiv:2312.02663, 2023

    2023 arXiv

  53. [61]

    Auto-encoding variational bayes,

    D. P. Kingma, “Auto-encoding variational bayes,” in ICLR, 2014

    2014

  54. [62]

    Effective whole-body pose estimation with two-stages distillation,

    Z. Yang, A. Zeng, C. Yuan, and Y . Li, “Effective whole-body pose estimation with two-stages distillation,” in ICCV, 2023

    2023

  55. [63]

    Method for registration of 3-d shapes,

    P. J. Besl and N. D. McKay, “Method for registration of 3-d shapes,” in Sensor fusion IV: control paradigms and data structures , vol. 1611. Spie, 1992, pp. 586–606

    1992

  56. [64]

    Generative modeling with phase stochastic bridges,

    T. Chen, J. Gu, L. Dinh, E. A. Theodorou, J. Susskind, and S. Zhai, “Generative modeling with phase stochastic bridges,” in ICLR, 2024

    2024

  57. [65]

    Elucidating the design space of diffusion-based generative models,

    T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the design space of diffusion-based generative models,” in NeurIPS, 2022

    2022

  58. [66]

    D. E. Kirk, Optimal control theory: an introduction . Courier Corpora- tion, 2004

    2004

  59. [67]

    W. H. Fleming and R. W. Rishel, Deterministic and stochastic optimal control. Springer Science & Business Media, 2012, vol. 1

    2012

  60. [68]

    Tweedie’s formula and selection bias,

    B. Efron, “Tweedie’s formula and selection bias,” Journal of the Amer- ican Statistical Association , 2011

    2011

  61. [69]

    Learning high fidelity depths of dressed humans by watching social media dance videos,

    Y . Jafarian and H. S. Park, “Learning high fidelity depths of dressed humans by watching social media dance videos,” in CVPR, 2021

    2021

  62. [70]

    Image quality metrics: Psnr vs. ssim,

    A. Hore and D. Ziou, “Image quality metrics: Psnr vs. ssim,” in 2010 20th international conference on pattern recognition , 2010

    2010

  63. [71]

    Liveportrait: Efficient portrait animation with stitching and retargeting control,

    J. Guo, D. Zhang, X. Liu, Z. Zhong, Y . Zhang, P. Wan, and D. Zhang, “Liveportrait: Efficient portrait animation with stitching and retargeting control,” arXiv preprint arXiv:2407.03168 , 2024

    2024 arXiv

  64. [72]

    Cotracker: It is better to track together,

    N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rup- precht, “Cotracker: It is better to track together,” in ECCV, 2024

    2024

  65. [73]

    Cogvideox: Text-to-video diffusion models with an expert transformer,

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Feng et al., “Cogvideox: Text-to-video diffusion models with an expert transformer,” arXiv preprint arXiv:2408.06072 , 2024. Shuyuan Tu is currently pursuing his Ph.D. degree in Computer Scie...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.