Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MotionStone claims that decoupling object and camera motion intensity, with scores learned from pairwise human comparisons and tracking-based pseudo-labels, gives image-to-video diffusion models stable, human-aligned control over motion.

desk verdict A useful decoupled motion-intensity estimator with a novel ranking-based annotation scheme, but the absolute 1-10 scale is underspecified and the control evaluation is circular. read the letter →

arxiv 2412.05848 v1 pith:WDPABYSB submitted 2024-12-08 cs.CV

Shuwei Shi , Biao Gong , Xi Chen , Dandan Zheng , Shuai Tan , Zizheng Yang , Yuyuan Li , Jingwen He
show 4 more authors
Kecheng Zheng Jingdong Chen Ming Yang Yinqiang Zheng
This is my paper · ORCID
classification cs.CV
keywords image-to-videogenerationmotionintensityestimationobjectcameracontrastivelearningdiffusiontransformercontroldecoupledconditioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MotionStone is built on the idea that motion intensity for image-to-video generation should be two separate numbers, one for object motion and one for camera motion, rather than one global value. Because people find it hard to label absolute intensity, the paper trains a motion estimator with simple pairwise comparisons—which of two clips moves more—and then anchors its 1-to-10 scale with tracking-derived pseudo-labels. The frozen estimator's two scores are injected into a pretrained diffusion transformer through separate embeddings added to the time-step condition, so the model can tell camera control from object control. The paper reports that this decoupled scheme improves temporal consistency and image quality over fixed-intensity, SSIM-based, and feature-difference baselines, and that users can independently dial object and camera intensity. The central claim is that a reliable, human-aligned decoupled motion estimator is what makes fine-grained motion control in image-to-video generation work.

What carries the argument

The load-bearing mechanism is the decoupled motion estimator plus its injection scheme. The estimator uses a TAdaConv backbone to extract spatiotemporal motion features, then two MLP heads predict object motion and camera motion scores in the 1-to-10 range. It is trained with a pairwise ranking loss on relative human annotations and a regression loss whose pseudo-labels come from tracking trajectories combined with segmentation masks. In the generator, the two scores pass through separate MLPs, are concatenated, and are added to the time embedding of a diffusion transformer, so each control signal is kept distinct during denoising.

What would settle it

Render or record clips with calibrated motion—a camera panning at known speeds and an object moving at known velocities—then run the estimator on them and check whether its object and camera scores are monotone in the true speeds and whether pairwise rankings match the ground truth; non-monotone or inverted scores on these calibrated cases would falsify the claim that the estimator aligns with human perception.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a lightweight motion estimator can decompose the motion in a raw video clip into an object score and a camera score, and that these decoupled scores are the right conditioning signal for a video diffusion model. The estimator is trained in two complementary ways: contrastive ranking on 5,000 video pairs labeled by relative human judgment, and regression on pseudo-labels computed from tracking trajectories combined with segmentation masks. During generation training the estimator is frozen, and its two outputs are mapped through separate MLPs, concatenated, and added to the diffusion time embedding, which keeps the two control channels from blurring together. On the reported benchmarks MotionStone improves background consistency, aesthetic quality, and imaging quality over the compared image-to-video models, and the estimator itself predicts human-annotated relative motion in video pairs with 72.80% accuracy versus 44.56% for an SSIM-based proxy.

Load-bearing premise

Everything rests on the assumption that the tracking-and-segmentation pseudo-labels, which fix the estimator's 1-to-10 scale, are faithful proxies for what people mean by object and camera motion intensity.

Editorial extensions

If this is right

  • Users can set object motion and camera motion independently on a 1-to-10 scale, and the generated video's movement follows the requested levels.
  • Training with estimated, per-video motion intensities yields better background consistency and frame quality than training with a fixed default intensity.
  • Replacing whole-video motion proxies such as inter-frame SSIM or feature difference with the decoupled estimator improves both generation quality and motion controllability.
  • Because the estimator is frozen and only supplies additional conditioning, it can be attached to an existing diffusion-transformer video generator as a plug-in enhancer.
  • On the validation video-pair set, the estimator predicts human-annotated relative motion with 72.80% accuracy, well above an SSIM-based proxy's 44.56%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same contrastive setup could be extended to other subjective video properties, such as motion naturalness or speed categories, without changing the annotation protocol, since it only requires relative labels.
  • A concrete offline use of the plug-in framing would be corpus filtering: the estimator could score large unlabeled video collections and curate training sets with a desired balance of object and camera motion.
  • The paper validates relative ranking and generation quality but does not compare its 1-to-10 scores against human absolute ratings; a direct calibration study would be the quickest way to test whether the estimator's scale is truly human-aligned.
  • The decoupled injection design suggests that other paired or opposing conditions, such as content motion versus style motion, could benefit from separate embedding channels added to the time-step modulation rather than a single merged condition.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes MotionStone, an image-to-video (I2V) generation model built on CogVideoX that adds decoupled control of object and camera motion intensity. The authors train a motion estimator using pairwise human annotations of relative motion strength on 5,000 video pairs, together with a regression term whose pseudo-labels are derived from tracking and SAM-based masks. The resulting object and camera intensity scores are injected into the diffusion transformer through two separate MLPs concatenated into the time embedding. Experiments compare against I2VGen-XL, SVD, AnimateAnything, and CogVideoX using VBench metrics, user studies, ablations, and motion-strength error measurements, reporting improved quantitative scores and qualitative controllability.

Significance. If the claims hold, the decoupled motion estimator is a plausible scalable alternative to SSIM or optical-flow-based motion intensity for I2V training, and the injection scheme offers a practical way to expose object-versus-camera motion control to users. The paper's strengths include the relative annotation protocol (which is easier for annotators than absolute scoring), the compact TAdaConv-based estimator with dual heads, and ablations that consistently show each proposed module contributing. However, the absolute 1-10 scale is not independently validated, the regression pseudo-label construction is under-specified, and the control-precision evaluation in Table 6 is partly circular because it reuses the same estimator that provided the training signal.

major comments (5)
  1. [Section 3.3, Eq. (7)] The regression targets y_object and y_camera are said to be approximated from tracking trajectories [49] combined with SAM masks [50], but the conversion from trajectories to absolute intensity scores is never specified. This conversion defines the estimator's output scale and the user-facing 1-10 controls, so without it the absolute scale has no demonstrated semantics. Please specify the algorithm (e.g., how trajectory magnitudes are pooled, normalized, and mapped to the 1-10 range) and ideally validate the resulting pseudo-labels against human absolute ratings.
  2. [Table 6, Section F.1] The motion strength error is computed by applying a motion estimator to generated videos and comparing the result with the user-specified input intensity. If this is the same estimator used to produce training labels and to condition the generator during training, the low error (2.52) largely measures self-consistency between the generator and the estimator; the generator could learn the estimator's biases rather than human-perceived intensity. Please evaluate control precision with an independent human study (e.g., users judge which generated video matches a requested intensity) or with a separately-trained estimator not used in the pipeline.
  3. [Section 3.3, Eqs. (5)-(6) and Appendix B] The annotation labels in Appendix B are -2, -1, 0, 1, 2, and the text says that significant differences are amplified by doubling the loss, but the main-text losses are simple max(0, s2 - s1) forms that assume one direction and no label-dependent weighting. As written, Eqs. (5) and (6) do not implement the described training procedure. Please provide the full loss that uses the signed labels and the amplification factor.
  4. [Tables 1, 2, 5, 7] The headline improvements are small in absolute terms (e.g., Background Consistency 95.76% vs. 94.91% for CogVideoX-5B; Imaging Quality 62.29% vs. 61.99%). No error bars, confidence intervals, or significance tests are reported, and the evaluation set size is not stated. Without variance estimates, the state-of-the-art claim and the ablation conclusions are not statistically supported.
  5. [Table 3 and Abstract] Table 3 evaluates the motion estimator only on the validation split of the same human-annotated video-pair dataset used for training. There is no cross-dataset or in-the-wild evaluation, so the abstract's claim that the estimator is a 'general plug-in enhancer' for arbitrary videos is not supported. Please add a generalization study, e.g., zero-shot evaluation on another video dataset or on WebVID clips with independently collected relative judgments.
minor comments (5)
  1. [Table 5, Row 2] The row labeled 'MotionStone w/o S' appears to be the same as 'MotionStone w/ S' in Table 2 (both report 94.97% Background Consistency, 46.13% Aesthetic Quality, 60.73% Imaging Quality). The label should be 'MotionStone w/ S' or the contents should be corrected to avoid confusion.
  2. [Section 4.1 and References [1], [22]] The evaluation set is referred to as 'WebVID validation set [1]' in one place and 'WebVID [22]' in another, but reference [22] is WebVision, not WebVid. Please use the correct citation consistently.
  3. [Appendix B] The amplification rule for significant differences is described only in prose. Even after the main-text loss is corrected, please give the exact formula for how the magnitude labels (1 vs. 2) enter the loss.
  4. [Section 4.1] The 100,000 training videos are described only as 'high-quality videos collected by ourselves'; please provide details on curation, filtering criteria, licenses, and whether there is any overlap with the WebVID evaluation set.
  5. [Figures 4-9] Several qualitative claims about camera pan/zoom and object motion are supported by still frames; since these effects are temporal, please point readers explicitly to the supplemental video files rather than relying on static snapshots for verification.

Circularity Check

1 steps flagged · score 6.0 of 10

Control-precision evidence is self-referential: Table 6 measures generated motion with the same estimator used to condition training.

  1. fitted input called prediction [Appendix F.1, Table 6 caption/text, with Sec. 3.2 and Sec. 3.4 training loop]
    "Using predefined motion intensity values, we generate videos and subsequently apply a motion estimator to obtain the corresponding motion intensities. The mean squared error (MSE) between the generated video intensities and the input values is then calculated. As shown in Tab. 6, the motion estimator proposed in this work provides more stable motion guidance and ensures that the motion intensities in the generated videos align more closely with the user-specified values."

    The 'motion estimator proposed in this work' in Table 6 is the same estimator whose outputs were used as the conditioning signal when training MotionStone: the paper states that during diffusion training it 'freeze[s] the pre-trained motion estimator and use[s] its prediction result as an additional input for noise prediction at each step.' Thus the generator is explicitly trained to produce videos whose estimated object/camera scores match the injected intensity values. Table 6 then evaluates control precision by running that same estimator on generated videos and computing MSE against the user-specified values. Low error therefore measures self-consistency between the generator and the estimator, not agreement with human-perceived motion magnitude.

full rationale

The paper has substantial independent content. Table 3 validates the estimator's relative ordering against manually annotated video-pair ground truth, and the main generation-quality comparisons use external metrics (VBench) and user studies. No load-bearing self-citation chain or imported uniqueness theorem is present. The circularity is localized to the absolute control-precision claim: the only quantitative evidence that user-specified 1-10 intensities are faithfully realized is Table 6, which measures error with the same estimator used to create the training conditioning signal, making the low error partly a by-construction artifact. Separately, the paper never specifies how tracking trajectories from [49] and SAM masks from [50] are converted into the absolute regression pseudo-labels y_object and y_camera (Sec. 3.3), so the semantic anchoring of the 1-10 scale rests on an unstated calibration; this is a serious limitation but not itself a circular step. Because one central quantitative claim reduces to self-consistency by construction, while relative ranking and generation quality retain independent evidence, a score of 6 is appropriate.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the motion estimator's ability to produce calibrated 1-10 scores from relative annotations and tracking-derived pseudo-labels. The pseudo-label conversion is the largest unstated scaffolding item. No new physical entities are introduced.

free parameters (2)
  • lambda (regression loss weight) = 0.1
    Set in Supplementary Sec A; balances ranking and regression losses without sensitivity analysis.
  • amplification factor for significant-difference pairs = 2
    Supplementary Sec B states losses are twice as large for significantly different pairs, but Eq 5-6 do not encode this; the factor is chosen by hand.
assumptions (4)
  • domain assumption Relative human annotations of object/camera motion intensity in video pairs are consistent and can be scaled to a 1-10 absolute scale.
    Supplementary Sec B reports 95% agreement on 20% of the 5,000 pairs; no external validation of the absolute scale is provided.
  • domain assumption Object motion and camera motion are separable in real-world videos.
    Sec 1 and Fig 2 assume disentanglement; no quantitative verification beyond generated examples is given.
  • domain assumption Tracking trajectories with SAM masks provide reliable regression pseudo-labels.
    Sec 3.3 uses tracking [49] and masks [50] to produce y_object and y_camera; the conversion to intensity is unspecified.
  • domain assumption TAdaConv backbone produces a video motion representation that supports the two heads.
    Adopted from [19]; no comparison to alternative motion backbones is provided in the estimator ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation." pith.science (2026). https://pith.science/paper/WDPABYSB

@misc{pith2026241205848,
  author       = {Pith},
  title        = {Pith review of: MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WDPABYSB}},
  note         = {Machine review of arXiv:2412.05848}
}
read the original abstract

The image-to-video (I2V) generation is conditioned on the static image, which has been enhanced recently by the motion intensity as an additional control signal. These motion-aware models are appealing to generate diverse motion patterns, yet there lacks a reliable motion estimator for training such models on large-scale video set in the wild. Traditional metrics, e.g., SSIM or optical flow, are hard to generalize to arbitrary videos, while, it is very tough for human annotators to label the abstract motion intensity neither. Furthermore, the motion intensity shall reveal both local object motion and global camera movement, which has not been studied before. This paper addresses the challenge with a new motion estimator, capable of measuring the decoupled motion intensities of objects and cameras in video. We leverage the contrastive learning on randomly paired videos and distinguish the video with greater motion intensity. Such a paradigm is friendly for annotation and easy to scale up to achieve stable performance on motion estimation. We then present a new I2V model, named MotionStone, developed with the decoupled motion estimator. Experimental results demonstrate the stability of the proposed motion estimator and the state-of-the-art performance of MotionStone on I2V generation. These advantages warrant the decoupled motion estimator to serve as a general plug-in enhancer for both data processing and video generation training.

Figures

Figures reproduced from arXiv: 2412.05848 by the authors.

Figure 1
Figure 1. Samples generated by MotionStone. Our model achieves accurate motion instruction following (rows-1 and rows-2), and is controllable, easily adapting to specified object motion intensities (row-3) and camera motion intensities (row-4). Abstract The image-to-video (I2V) generation is conditioned on the static image, which has been enhanced recently by the motion intensity as an additional control signal. These motion-… view at source ↗
Figure 2
Figure 2. Illustration of the motion decoupling. Decoupling these two types of motion helps the diffusion model learn specific motion patterns, thereby improving the dynamics and controllabil￾ity of the generated video. freeze the pre-trained motion estimator and use its predic￾tion result as an additional input for noise prediction at each step. In particular, we design a decoupled motion score injection method that allows t… view at source ↗
Figure 3
Figure 3. The framework of MotionStone. The first frame of the video serves as the conditioning image, while object and camera motion intensities (ranging from 1 to 10) are predicted by the motion estimator and can be customized by users during inference. At the top, the object and camera motion intensities predicted by the motion estimator are processed through an MLP respectively to obtain corresponding embeddings, which ar… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison with other methods. We compare our MotionStone with I2VGEN-XL [53], SVD [2], AnimateAnything [11] and CogvideoX [52]. MotionStone demonstrates superior alignment with text and image inputs compared to other methods (Example 2 and Example 4). Addi…
Figure 5
Figure 5. Figure 5: Illustrations of camera motion intensity guidance. We present two common camera movements: Zoom and Pan. Since the camera movement often impacts object motion in scenes with moving subjects, we fix the object motion intensity at 5 to isolate and highlight the effect of…
Figure 6
Figure 6. Figure 6: Illustrations of object motion intensity guidance. To emphasize control over object motion intensity and speed, we exclude camera motion prompts from the text and set the camera motion intensity to its minimum value of 1 while varying the object motion intensity. As th…
Figure 7
Figure 7. Figure 7: Qualitative ablation for proposed modules. Using inter-frame SSIM [8] and feature difference [11] (MotionStone w/ SSIM and MotionStone w/ S) causes varying degrees of unnatural background motion (In the first row, the snow block in the upper left corner of the third co…
Figure 8
Figure 8. Figure 8: Illustrations of object and camera motion intensity guidance. MotionStone can decouple and independently control camera motion and object motion intensities. When either camera motion or object motion is increased, the generated videos exhibit excellent adherence to th…
Figure 9
Figure 9. Figure 9: More cases generated by MotionStone. MotionStone demonstrates impressive generation quality across various scenarios. Here, the default object motion intensity or camera motion intensity (if applicable) is set to 5 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A diffusion model animates a character into arbitrary dynamic backgrounds by conditioning on a rendered 3D-avatar video, reframing open-domain animation as a restoration problem.

  2. Animate-X++: Universal Character Image Animation with Dynamic Backgrounds

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Animate-X++ turns cartoon images into pose-driven animations with text-controlled moving backgrounds, claiming state-of-the-art results on a new synthetic anthropomorphic benchmark.

Reference graph

Works this paper leans on

57 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [49]

    Spatialtracker: Tracking any 2d pixels in 3d space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406–20417, 2024. 4 10

    work page 2024

  2. [50]

    Moving object segmentation: All you need is sam (and flow)

    Junyu Xie, Charig Yang, Weidi Xie, and Andrew Zisserman. Moving object segmentation: All you need is sam (and flow). arXiv preprint arXiv:2404.12389, 2024. 4

    arXiv 2024

  3. [1]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1728– 1738, 2021. 6

    work page 2021

  4. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 5, 6

    arXiv 2023

  5. [3]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22563–22575, 2023. 3

    work page 2023

  6. [4]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021. 3

    work page 2021

  7. [5]

    Stablevideo: Text-driven consistency-aware diffusion video editing

    Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency-aware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23040–23050, 2023. 2

    work page 2023

  8. [6]

    Magicpose: Realistic human poses and facial expressions retargeting with identity-aware dif- fusion

    Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang, and Mohammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware dif- fusion. In Forty-first International Conference on Machine Learning, 2023. 2

    work page 2023

Show all 57 references
  1. [7]

    Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- α: Fast training of diffusion transformer for photorealistic text-to- image synthesis. ArXiv, abs/2310.00426, 2023. 3

    arXiv

  2. [8]

    Livephoto: Real image animation with text-guided motion control

    Xi Chen, Zhiheng Liu, Mengting Chen, Yutong Feng, Yu Liu, Yujun Shen, and Hengshuang Zhao. Livephoto: Real image animation with text-guided motion control. In European Conference on Computer Vision, pages 475–491. Springer, 2025. 2, 3, 6, 8, 4, 5

    2025

  3. [9]

    Time flies: Animating a still image with time-lapse video as reference

    Chia-Chi Cheng, Hung-Yu Chen, and Wei-Chen Chiu. Time flies: Animating a still image with time-lapse video as reference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5641– 5650, 2020. 2

    2020

  4. [10]

    Animating pictures with stochastic motion textures

    Yung-Yu Chuang, Dan B Goldman, Ke Colin Zheng, Brian Curless, David H Salesin, and Richard Szeliski. Animating pictures with stochastic motion textures. In ACM SIG- GRAPH 2005 Papers, pages 853–860. 2005. 2

    2005

  5. [11]

    Animateanything: Fine-grained open domain image animation with motion guidance

    Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animateanything: Fine-grained open domain image animation with motion guidance. arXiv e-prints , pages arXiv–2311, 2023. 2, 3, 5, 6, 8, 4

    2023

  6. [12]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023. 2

    2023

  7. [13]

    Perceptual quality assessment of smartphone photography

    Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3677–3686,

  8. [14]

    Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion trans- formers

    Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion trans- formers. arXiv preprint arXiv:2405.05945, 2024. 3

    2024 arXiv

  9. [15]

    Check locate rectify: A training- free layout calibration system for text-to-image generation

    Biao Gong, Siteng Huang, Yutong Feng, Shiwei Zhang, Yuyuan Li, and Yu Liu. Check locate rectify: A training- free layout calibration system for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6624–6634, 2024. 3

    2024

  10. [16]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2, 3

    2023 arXiv

  11. [17]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  12. [18]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 2

    2024

  13. [19]

    Tada! temporally-adaptive convolutions for video understanding

    Ziyuan Huang, Shiwei Zhang, Liang Pan, Zhiwu Qing, Mingqian Tang, Ziwei Liu, and Marcelo H Ang Jr. Tada! temporally-adaptive convolutions for video understanding. arXiv preprint arXiv:2110.06178, 2021. 2, 4

    2021 arXiv

  14. [20]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  15. [21]

    Musiq: Multi-scale image quality transformer

    Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5148–5157, 2021. 3

    2021

  16. [22]

    WebVision Database: Visual Learning and Understanding from Web Data

    Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. WebVision Database: Visual Learning and Understanding from Web Data. ArXiv, abs/1708.02862,

    arXiv

  17. [23]

    Amt: All-pairs multi-field transforms for efficient frame interpolation

    Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun- Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 9801–9810, 2023. 3

    2023

  18. [24]

    Magicedit: High-fidelity and temporally 9 coherent video editing

    Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng. Magicedit: High-fidelity and temporally 9 coherent video editing. arXiv preprint arXiv:2308.14749 ,

    arXiv

  19. [25]

    Rankiqa: Learning from rankings for no-reference image quality assessment

    Xialei Liu, Joost Van De Weijer, and Andrew D Bagdanov. Rankiqa: Learning from rankings for no-reference image quality assessment. In Proceedings of the IEEE international conference on computer vision, pages 1040–1049, 2017. 4

    2017

  20. [26]

    Cinemo: Consistent and controllable image animation with motion diffusion models

    Xin Ma, Yaohui Wang, Gengyu Jia, Xinyuan Chen, Yuan- Fang Li, Cunjian Chen, and Yu Qiao. Cinemo: Consistent and controllable image animation with motion diffusion models. arXiv preprint arXiv:2407.15642, 2024. 2, 3

    2024 arXiv

  21. [27]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 3

    2024 arXiv

  22. [28]

    Controllable animation of fluid elements in still images

    Aniruddha Mahapatra and Kuldeep Kulkarni. Controllable animation of fluid elements in still images. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3667–3676, 2022. 2

    2022

  23. [29]

    Revideo: Remake a video with motion and content control

    Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Jian Zhang. Revideo: Remake a video with motion and content control. arXiv preprint arXiv:2405.13865, 2024. 2

    2024 arXiv

  24. [30]

    Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Controllable image animation via generative motion field adaptions in frozen image-to-video diffusion model. arXiv preprint arXiv:2405.20222, 2024. 2

    2024 arXiv

  25. [31]

    Animating pictures of fluid using video examples

    Makoto Okabe, Ken Anjyo, Takeo Igarashi, and Hans-Peter Seidel. Animating pictures of fluid using video examples. In Computer Graphics Forum, pages 677–686. Wiley Online Library, 2009. 2

    2009

  26. [32]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  27. [33]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  28. [34]

    Consisti2v: Enhancing visual consistency for image-to-video generation

    Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. arXiv preprint arXiv:2402.04324, 2024. 2

    2024 arXiv

  29. [35]

    Image animation with perturbed masks

    Yoav Shalev and Lior Wolf. Image animation with perturbed masks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3647– 3656, 2022. 2

    2022

  30. [36]

    Resmaster: Mastering high- resolution image generation via structural and fine-grained guidance

    Shuwei Shi, Wenbo Li, Yuechen Zhang, Jingwen He, Biao Gong, and Yinqiang Zheng. Resmaster: Mastering high- resolution image generation via structural and fine-grained guidance. arXiv preprint arXiv:2406.16476, 2024. 3

    2024 arXiv

  31. [37]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019. 2

    2019

  32. [38]

    Motion representations for articulated animation

    Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13653–13662, 2021. 2

    2021

  33. [39]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 3

    2010 arXiv

  34. [40]

    Animate-x: Universal character image animation with enhanced motion representation

    Shuai Tan, Biao Gong, Xiang Wang, Shiwei Zhang, Dandan Zheng, Ruobing Zheng, Kecheng Zheng, Jingdong Chen, and Ming Yang. Animate-x: Universal character image animation with enhanced motion representation. arXiv preprint arXiv:2410.10306, 2024. 2

    arXiv

  35. [41]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  36. [42]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  37. [43]

    Unianimate: Taming unified video diffusion models for consistent human image animation

    Xiang Wang, Shiwei Zhang, Changxin Gao, Jiayu Wang, Xiaoqiang Zhou, Yingya Zhang, Luxin Yan, and Nong Sang. Unianimate: Taming unified video diffusion models for consistent human image animation. arXiv preprint arXiv:2406.01188, 2024. 2

    2024 arXiv

  38. [44]

    Latent image animator: Learning to ani- mate images via latent space navigation

    Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to ani- mate images via latent space navigation. arXiv preprint arXiv:2203.09043, 2022. 2

    2022 arXiv

  39. [45]

    Lavie: High-quality video generation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 3

    2023 arXiv

  40. [46]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 2

    2004

  41. [47]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  42. [48]

    Automatic animation of hair blowing in still portrait photos

    Wenpeng Xiao, Wentao Liu, Yitong Wang, Bernard Ghanem, and Bing Li. Automatic animation of hair blowing in still portrait photos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22963– 22975, 2023. 2

    2023

  43. [51]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Eu- ropean Conference on Computer Vision , pages 399–417. Springer, 2025. 2

    2025

  44. [52]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 3, 5, 6, 7

    2024 arXiv

  45. [53]

    I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023. 2, 5, 6

    2023 arXiv

  46. [54]

    Pia: Your personalized image animator via plug-and-play modules in text-to-image models

    Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image animator via plug-and-play modules in text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7747–7756, 2024. 2, 3

    2024

  47. [55]

    Tora: Trajectory-oriented diffu- sion transformer for video generation

    Zhenghao Zhang, Junchao Liao, Menghao Li, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented diffu- sion transformer for video generation. arXiv preprint arXiv:2407.21705, 2024. 2

    2024 arXiv

  48. [56]

    Thin-plate spline motion model for image animation

    Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3657–3666, 2022. 2

    2022

  49. [57]

    Camera zooms out. A penguin is dancing

    Ruiqi Zhao, Tianyi Wu, and Guodong Guo. Sparse to dense motion transfer for face image animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1991–2000, 2021. 2 11 MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transform...

    1991

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.