Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

ATI: Any Trajectory Instruction for Controllable Video Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One trajectory input controls camera, object, and local motion in video generation.

desk verdict ATI is a plausible trajectory-conditioning method with nice qualitative demos, but the quantitative evidence for its headline superiority claim is missing – no baselines, a small self-built test set, and a tracker reused from training labels. read the letter →

arxiv 2505.22944 v3 pith:VB6PWMS3 submitted 2025-05-28 cs.CV cs.AI

Angtian Wang , Haibin Huang , Jacob Zhiyuan Fang , Yiding Yang , Chongyang Ma This is my paper · ORCID
classification cs.CVcs.AI
keywords trajectory-basedvideogenerationmotioncontrolimage-to-videolatentfeatureinjectiondiffusiontransformercameraobjecttaildropoutregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that one input — trajectories of 2D points drawn on a single frame — can control every major kind of motion in generated video: camera movement such as pan and zoom, translation of whole objects, and fine-grained local deformation. Its wager is that these seemingly different motion types are all the same thing at the level of the diffusion model's latent space, and that a lightweight motion injector can translate point paths into latent instructions for a pretrained image-to-video model without task-specific modules. The authors build this injector into two large DiT backbones and report better controllability and quality than separate-module systems and commercial products, with one drawing interface for all motion. If the claim holds, the practical payoff is a single intuitive control surface for video generation and a control recipe that ports across backbone models. The paper itself flags where the equivalence strains: purely planar trajectory sets can leave content static and only produce 2D camera movement.

What carries the argument

The load-bearing mechanism is the Gaussian motion injector. For each trajectory point it extracts a $C$-dimensional appearance feature $f$ from the first-frame latent by bilinear sampling at the start location $\varphi_0$, then defines a spatial mask $P(f \mid l_{i,j,t}) = \exp(-\|\varphi_t - (i,j)\|^2 / (2\sigma))$ centered at the point's location $\varphi_t$ in each later frame, with $\sigma = 1/440$ so the weight falls to half at the nearest diagonal pixel. Blending these masked features into the latent stream before the DiT patchify layer gives the denoiser a per-timestep reading of "this appearance belongs at this location now," converting sparse user strokes into dense soft guidance without changing the backbone architecture beyond fine-tuning all DiT parameters for around 50,000 iterations.

What would settle it

Run the trained model on a first frame with strong depth layering — a near object in front of a distant wall — and specify a purely horizontal pan via parallel point trajectories across the whole image. If the output is a rigid 2D translation with no relative displacement between near and far content, while a physical camera pan would produce depth-dependent parallax, the claim that 2D trajectories alone encode camera dynamics is contradicted. A second decisive test: draw trajectories that force an object to split apart, and check whether the model follows the instruction or substitutes a natural alternative, as the paper reports it often does.

Watch

Extended reading notes

Core claim

ATI's central discovery is that a point trajectory becomes a latent instruction through two steps: sample the first frame's VAE feature at the point's starting location, then, in every later frame, re-deposit that feature over a small Gaussian neighborhood centered at the point's prescribed position. Each trajectory point thereby carries the appearance of the content it anchors, and the denoiser learns to move that content along the drawn path. The same mechanism covers local deformation, object translation, and camera dynamics, because camera motion is just many background points moving together. A tail-dropout regularizer — randomly truncating trajectories during training with probability 0.2 — teaches the model that a vanished point means "no more instruction" rather than "something occludes it," which suppresses hallucinated occluders at early trajectory endings. On 100 hand-drawn trajectory sets, TAP-Net tracking shows points staying within 5% of the image diagonal on 55.9–59.0% of frames, with appearance rates of 65.5–67.9%, across the two backbones.

Load-bearing premise

The load-bearing premise is that every motion a user wants, including camera dynamics, can be captured by the 2D movement of image-plane points with no explicit depth or 3D structure — a premise the paper itself strains when it notes that purely planar trajectory sets yield static content with only 2D camera movement.

Editorial extensions

If this is right

  • A single drawing interface can replace separate camera-control, object-drag, and local-deformation modules; users mix zooms, pans, and local edits in one set of strokes on the first frame.
  • The injector ports across DiT image-to-video backbones with a standard fine-tuning pass, so trajectory control becomes a general adapter rather than a bespoke model.
  • Trajectories that end before the final frame no longer invite hallucinated occluders, since tail dropout teaches the model that a vanished point simply stops being tracked.
  • Combining trajectories — radial background zoom plus a static subject point — reproduces compound effects like a dolly-zoom without task-specific code.
  • Extreme instructions (a point crossing half the frame width in two frames, or an object forced to split) are the documented failure boundary of the unified control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The injector couples only to the latent grid and the patchify input, so the same Gaussian feature-carrying scheme should transfer to any latent transformer video model, including text-to-video generators that synthesize a first frame; this is a testable extension the paper does not run.
  • The static-content failure on planar trajectories points to a concrete upgrade the paper leaves implicit: add a per-point depth or disparity channel so background trajectories scale with depth and produce genuine parallax, replacing the hand-placed static point its dolly-zoom needs.
  • Trajectory sets behave like a composition algebra — an affine transform applied to all points is a camera move, editing a subset is an object move — which an editor UI could expose as pan/zoom buttons that transform every point rather than as a separate camera module.
  • Because TAP-Net builds the training labels and also scores the outputs, the reported follow-rates partly measure tracker agreement with the generator's appearance changes; a human study or independent point-correspondence benchmark would separate control fidelity from tracking artifacts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ATI, a trajectory-conditioned image-to-video generation framework. It samples latent feature vectors at user-selected keypoints in the first frame and injects them into later frames through Gaussian spatial masks, with a tail-dropout regularizer to avoid spurious occlusions when trajectories terminate early. Training data are generated by tracking uniformly sampled points on 2.4M clips with TAP-Net, and the module is fine-tuned on Seaweed-7B and Wan2.1-14B. The paper claims unified control of camera, object, and local motion through point trajectories, with superior controllability and visual quality relative to prior methods and commercial tools.

Significance. If the claims were supported, the contribution would be valuable: a single lightweight module that controls camera, object, and local motion through point trajectories in pretrained I2V models would simplify workflows and improve composability. The Gaussian feature injection and tail-dropout idea are simple and plausible, the training-data scale is substantial, and demonstrations on two backbones suggest compatibility. However, the current experimental evidence does not establish the headline superiority claims: the only quantitative table compares two ATI variants, and the evaluation metric shares its tracker with the label generator. The conceptual and methodological contribution is nonetheless a reasonable starting point for a stronger evaluation.

major comments (4)
  1. [§4.3, Table 1] The quantitative evaluation in Table 1 compares only ATI with Seaweed-7B against ATI with Wan2.1-14B, yet the abstract and introduction claim 'superior performance' and 'significantly better controllability and visual quality compared to prior approaches and commercial solutions.' With no baseline method or commercial system measured, this central claim is unsupported by the reported numbers; the authors should add comparisons against at least one trajectory-based prior method (e.g., Motion Prompting, ReVideo, Trailblazer) and one commercial tool, together with error bars or significance tests.
  2. [§3.4, §4.3] The evaluation metric is circular in a mild but important way: TAP-Net is used both to generate the training trajectory labels (Section 3.4) and to track user-specified points in the generated videos during evaluation (Section 4.3). Because the model is trained to satisfy TAP-Net's tracking behavior, high Acc@0.05/0.01 may reflect TAP-Net's inductive biases rather than genuine point correspondence in the generated video. The authors should re-evaluate with an independent tracker (e.g., CoTracker or PIPs) or with human-annotated correspondences on a subset of the test set.
  3. [§4.3] The paper claims better 'visual quality' but reports no image or video quality metric (e.g., FVD, CLIP score, or human preference). Table 1 only measures trajectory-following accuracy. A user study or standard quality metric is needed to support the visual-quality component of the central claim, which is currently asserted without quantitative evidence.
  4. [§4.2, Figure 5] The self-reported observation that purely planar horizontal shifts or zooms can leave content static shows that not all camera dynamics are faithfully encoded by point trajectories; the dolly-zoom requires a special combination of static subject and background zoom trajectories. This limits the claimed unification of camera and object motion and should be analyzed systematically, with a characterization of which camera motions are representable and quantitative camera-control evaluation (e.g., pose estimation) rather than only qualitative examples.
minor comments (4)
  1. [§3.4, §4.2, Reference [24]] There are several typos: 'filted' should be 'filtered' in Section 3.4; 'we observer' should be 'we observe' in Section 4.2; and the team name in reference [24] is spelled 'Seawead' instead of 'Seaweed.'
  2. [§4.3] The definition of 'Appearance Rate' is vague ('whenever the user-specified trajectory is present'); please clarify whether this counts all frames in which the user trajectory exists and whether the tracker's visibility prediction is compared against user intent or against ground-truth visibility.
  3. [§3.2] The Gaussian width σ = 1/440 is given only in pixel units of the input image; since the latent grids of Seaweed-7B and Wan2.1-14B differ, please specify how σ scales across models and resolutions.
  4. [§4.2] The qualitative section does not state how representative the shown examples are or how failures were counted; a success-rate breakdown over the 100 test pairs, including the noted failure cases (rapid motion, object disassembly), would help calibrate the method's reliability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TAP-Net used for both training labels and evaluation is a same-tool dependency, not a by-construction reduction.

full rationale

The paper's derivation chain is self-contained. Trajectory features are injected into a pretrained DiT via a Gaussian-based motion injector, and the model is fine-tuned with a standard noise-prediction objective (Eq. 3), not with a loss that optimizes TAP-Net's tracker on generated frames. TAP-Net is used both to create training labels (Sec. 3.4) and to evaluate trajectory agreement (Sec. 4.3); this is a methodological same-tool dependency that could bias measurements, but it is not a by-construction equivalence: the training objective does not predict TAP-Net outputs, the test trajectories are manually drawn on unseen images, and the reported metrics are not a fitted parameter renamed as a prediction. No load-bearing self-citation is present; the only author-overlapping references (e.g., [36], [39]) appear in the related-work survey and do not justify the core mechanism. No uniqueness theorem is imported, and no ansatz is smuggled in by citation. The abstract's claim of superiority over prior methods and commercial products is not supported by Table 1, which compares only ATI variants on two backbones, but that is an evidence gap and a correctness concern, not circularity. No circular step satisfies the requirement of exhibiting a specific reduction by construction, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. The main assumptions are about representation sufficiency and the reliability of the tracking-based supervision and evaluation.

free parameters (2)
  • Gaussian sigma = 1/440
    Chosen so the Gaussian weight decays to half its maximum at the nearest diagonal pixel; a hand-set design decision that controls the spatial support of the injected feature.
  • Tail dropout probability p = 0.2
    Hyperparameter for random trajectory truncation during training; chosen empirically, not derived.
assumptions (4)
  • standard math Diffusion model forward and reverse processes (Eqs. 1-3)
    Background probabilistic formulation of diffusion, taken from prior literature.
  • domain assumption DiT backbone and VAE encode/decode are treated as fixed and suitable for trajectory feature injection
    The method assumes the pretrained I2V model's latent space can accept the injected trajectory features and that fine-tuning all DiT parameters preserves base capabilities.
  • domain assumption TAP-Net provides accurate long-range point tracks for training
    Training data is generated by TAP-Net on 2.4M clips; if TAP-Net trajectories are noisy or mis-track, the learned conditioning is corrupted.
  • domain assumption Point trajectories are a sufficient representation for camera and object motion
    The unified design assumes camera pans, zooms, and rotations can be expressed as background-point trajectories; the paper notes limitations when all shifts are planar.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ATI: Any Trajectory Instruction for Controllable Video Generation." pith.science (2026). https://pith.science/paper/VB6PWMS3

@misc{pith2026250522944,
  author       = {Pith},
  title        = {Pith review of: ATI: Any Trajectory Instruction for Controllable Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VB6PWMS3}},
  note         = {Machine review of arXiv:2505.22944}
}
read the original abstract

We propose a unified framework for motion control in video generation that seamlessly integrates camera movement, object-level translation, and fine-grained local motion using trajectory-based inputs. In contrast to prior methods that address these motion types through separate modules or task-specific designs, our approach offers a cohesive solution by projecting user-defined trajectories into the latent space of pre-trained image-to-video generation models via a lightweight motion injector. Users can specify keypoints and their motion paths to control localized deformations, entire object motion, virtual camera dynamics, or combinations of these. The injected trajectory signals guide the generative process to produce temporally consistent and semantically aligned motion sequences. Our framework demonstrates superior performance across multiple video motion control tasks, including stylized motion effects (e.g., motion brushes), dynamic viewpoint changes, and precise local motion manipulation. Experiments show that our method provides significantly better controllability and visual quality compared to prior approaches and commercial solutions, while remaining broadly compatible with various state-of-the-art video generation backbones. Project page: https://anytraj.github.io/.

Figures

Figures reproduced from arXiv: 2505.22944 by the authors.

Figure 1
Figure 1. ATI is able to generate a video given an initial frame (left) and a set of user-specified trajectories. Green dots denote the starting [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. ATI takes an image and user specified trajectories as inputs. The point-wise trajectories are injected into the latent condition for [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Trajectory Instruction module computes a latent feature from a point’s trajectory. During inference, given the point’s location [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Object Motion Control. Left: the input image overlaid with user-specified trajectories—green dots mark each trajectory’s start [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Video generation results with camera control. Left: Input image superimposed with user specified trajectories. Right: Five [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Video generation results with coherent control of camera and object motion. Left: Input image superimposed with user specified [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison for ATI video generation with different backend models. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Motion4Motion: Motion Transfer Across Subjects at Inference

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Training-free motion transfer across species works by extracting source motion flows, matching semantic points, and injecting them into DiT self-attention via TransPE positional padding.

  2. SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generation

    cs.CV 2026-04 conditional novelty 6.0 of 10

    SymphoMotion jointly controls camera trajectories and depth-aware object dynamics inside one video diffusion model, supported by the new RealCOD-25K real-world paired-motion dataset.

  3. From World Action Models to Embodied Brains: A Roadmap for Open-World Physical Intelligence

    cs.RO 2026-07 conditional novelty 4.0 of 10

    Physical intelligence needs an embodied brain that reasons over interventions and emits capability requests, grounded by a physical harness and shared experience contracts rather than direct actuator policies.

Reference graph

Works this paper leans on

44 extracted references · 20 canonical work pages · cited by 3 Pith papers

  1. [1]

    Lumiere: A space-time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, and et.al. Lumiere: A space-time diffusion model for video generation. 2024. 1

    work page 2024

  2. [2]

    Stable video diffusion: Scaling la- tent video diffusion models to large datasets.arXiv preprint, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling la- tent video diffusion models to large datasets.arXiv preprint, 2023

    work page 2023

  3. [3]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. 2023

    work page 2023

  4. [4]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. 2024. 1

    work page 2024

  5. [5]

    Perception-as-control: Fine-grained control- lable image animation with 3d-aware motion representation

    Yingjie Chen, Yifang Men, Yuan Yao, Miaomiao Cui, and Liefeng Bo. Perception-as-control: Fine-grained control- lable image animation with 3d-aware motion representation. arXiv preprint arXiv:2501.05020, 2025. 3

    arXiv 2025

  6. [6]

    TAP-vid: A benchmark for track- ing any point in a video.Advances in Neural Information Processing Systems, 35:13610–13626, 2022

    Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Re- casens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-vid: A benchmark for track- ing any point in a video.Advances in Neural Information Processing Systems, 35:13610–13626, 2022. 4

    work page 2022

  7. [7]

    I2vcontrol-camera: Precise video camera control with adjustable motion strength.arXiv preprint arXiv:2411.06525, 2024

    Wanquan Feng, Jiawei Liu, Pengqi Tu, Tianhao Qi, Mingzhen Sun, Tianxiang Ma, Songtao Zhao, Siyu Zhou, and Qian He. I2vcontrol-camera: Precise video camera control with adjustable motion strength.arXiv preprint arXiv:2411.06525, 2024. 2

    arXiv 2024

  8. [8]

    Mo- tion prompting: Controlling video generation with motion trajectories.arXiv preprint arXiv:2412.02700, 2024

    Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. Mo- tion prompting: Controlling video generation with motion trajectories.arXiv preprint arXiv:2412.02700, 2024. 3

    arXiv 2024

Show all 44 references
  1. [9]

    I2v-adapter: A general image-to-video adapter for diffusion models

    Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, et al. I2v-adapter: A general image-to-video adapter for diffusion models. InACM SIGGRAPH 2024 Conference Papers, pages 1–12, 2024. 1

    2024

  2. [10]

    Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024. 2

    2024 arXiv

  3. [11]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022. 1

    2022 arXiv

  4. [12]

    Training-free camera control for video generation.arXiv preprint arXiv:2406.10126, 2024

    Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-free camera control for video generation.arXiv preprint arXiv:2406.10126, 2024. 2

    2024 arXiv

  5. [13]

    Peekaboo: Interactive video generation via masked- diffusion

    Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked- diffusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8079– 8088, 2024. 3

    2024

  6. [14]

    Hunyuanvideo: A systematic framework for large video generative models,

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, and et.al. Hunyuanvideo: A systematic framework for large video generative models,

  7. [15]

    Collab- orative video diffusion: Consistent multi-video generation with camera control.Advances in Neural Information Pro- cessing Systems, 37:16240–16271, 2024

    Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hong- sheng Li, Leonidas J Guibas, and Gordon Wetzstein. Collab- orative video diffusion: Consistent multi-video generation with camera control.Advances in Neural Information Pro- cessing Systems, 37:16240–16271, 2024. 2

    2024

  8. [16]

    Image conductor: Precision control for interactive video syn- thesis

    Yaowei Li, Xintao Wang, Zhaoyang Zhang, Zhouxia Wang, Ziyang Yuan, Liangbin Xie, Ying Shan, and Yuexian Zou. Image conductor: Precision control for interactive video syn- thesis. InProceedings of the AAAI Conference on Artificial Intelligence, pages 5031–5038, 2025. 2

    2025

  9. [17]

    Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Li- uhan Chen, et al. Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024. 1

    2024 arXiv

  10. [18]

    Step-video-t2v technical report: The practice, challenges, and future of video founda- tion model, 2025

    Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, , and et.al. Step-video-t2v technical report: The practice, challenges, and future of video founda- tion model, 2025. 1

    2025

  11. [19]

    Trailblazer: Trajectory control for diffusion-based video generation

    Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video generation. InSIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024. 3

    2024

  12. [20]

    Revideo: Remake a video with motion and content control.Advances in Neural Information Processing Systems, 37:18481–18505, 2024

    Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Jian Zhang. Revideo: Remake a video with motion and content control.Advances in Neural Information Processing Systems, 37:18481–18505, 2024. 3

    2024

  13. [21]

    Mofa-video: Control- lable image animation via generative motion field adaptions in frozen image-to-video diffusion model.arXiv preprint arXiv:2405.20222, 2024

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Control- lable image animation via generative motion field adaptions in frozen image-to-video diffusion model.arXiv preprint arXiv:2405.20222, 2024. 2

    2024 arXiv

  14. [22]

    Scalable diffusion models with transformers.arXiv preprint arXiv:2212.09748, 2022

    William Peebles and Saining Xie. Scalable diffusion models with transformers.arXiv preprint arXiv:2212.09748, 2022. 3

    2022 arXiv

  15. [23]

    Movie gen: A cast of media foundation models, 2025

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, , and et.al. Movie gen: A cast of media foundation models, 2025. 1

    2025

  16. [24]

    Seaweed-7b: Cost-effective train- ing of video generation foundation model.arXiv preprint arXiv:2504.08685, 2025

    Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. Seaweed-7b: Cost-effective train- ing of video generation foundation model.arXiv preprint arXiv:2504.08685, 2025. 1, 2, 3, 4, 7

    2025 arXiv

  17. [25]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. InACM SIGGRAPH 2024 Conference Papers, pag...

    2024

  18. [26]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling.SIGGRAPH 2024, 2024

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, and Dasong andand et.al Li. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling.SIGGRAPH 2024, 2024. 2

    2024

  19. [27]

    Motionbridge: Dynamic video inbetweening with flexible controls.arXiv preprint arXiv:2412.13190, 2024

    Maham Tanveer, Yang Zhou, Simon Niklaus, Ali Mahdavi Amiri, Hao Zhang, Krishna Kumar Singh, and Nanxuan Zhao. Motionbridge: Dynamic video inbetweening with flexible controls.arXiv preprint arXiv:2412.13190, 2024. 2

    2024 arXiv

  20. [28]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, and et.al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    arXiv

  21. [29]

    Levitor: 3d trajectory oriented image-to-video syn- thesis.arXiv preprint arXiv:2412.15214, 2024

    Hanlin Wang, Hao Ouyang, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Qifeng Chen, Yujun Shen, and Limin Wang. Levitor: 3d trajectory oriented image-to-video syn- thesis.arXiv preprint arXiv:2412.15214, 2024. 3

    2024 arXiv

  22. [30]

    Boximator: Generat- ing rich and controllable motions for video synthesis.arXiv preprint arXiv:2402.01566, 2024

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guo- qiang Wei, Liping Yuan, and Hang Li. Boximator: Generat- ing rich and controllable motions for video synthesis.arXiv preprint arXiv:2402.01566, 2024. 3

    2024 arXiv

  23. [31]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint, 2024. 2, 3

    2024

  24. [32]

    Mo- tionbooth: Motion-aware customized text-to-video genera- tion.arXiv preprint, 2024

    Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Mo- tionbooth: Motion-aware customized text-to-video genera- tion.arXiv preprint, 2024. 2

    2024

  25. [33]

    Draganything: Motion control for any- thing using entity representation

    Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for any- thing using entity representation. InEuropean Conference on Computer Vision, pages 331–348. Springer, 2024. 2

    2024

  26. [34]

    Progressive autoregres- sive video diffusion models

    Desai Xie, Zhan Xu, Yicong Hong, Hao Tan, Difan Liu, Feng Liu, Arie Kaufman, and Yang Zhou. Progressive autoregres- sive video diffusion models. 2024. 1

    2024

  27. [35]

    Camco: Camera- controllable 3d-consistent image-to-video generation.arXiv preprint arXiv:2406.02509, 2024

    Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera- controllable 3d-consistent image-to-video generation.arXiv preprint arXiv:2406.02509, 2024. 2

    2024 arXiv

  28. [36]

    Direct-a-video: Customized video generation with user- directed camera movement and object motion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user- directed camera movement and object motion. InACM SIG- GRAPH 2024 Conference Papers, pages 1–12, 2024. 3

    2024

  29. [37]

    Cogvideox: Text-to- video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, and et.al Yang. Cogvideox: Text-to- video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024. 1

    2024 arXiv

  30. [38]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.arXiv preprint arXiv:2308.08089, 2023

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.arXiv preprint arXiv:2308.08089, 2023. 2

    2023 arXiv

  31. [39]

    Zero-shot controllable image-to-video animation via motion decomposition

    Shoubin Yu, Jacob Zhiyuan Fang, Jian Zheng, Gunnar Sig- urdsson, Vicente Ordonez, Robinson Piramuthu, and Mohit Bansal. Zero-shot controllable image-to-video animation via motion decomposition. InProceedings of the 32nd ACM International Conference on Multimedia, pages 3332–3341,

  32. [40]

    Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024

    Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024. 2

    2024 arXiv

  33. [41]

    Motiondirector: Motion customization of text-to- video diffusion models.arXiv preprint, 2023

    Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to- video diffusion models.arXiv preprint, 2023. 2

    2023

  34. [42]

    Cami2v: Camera-controlled image-to-video dif- fusion model.arXiv preprint arXiv:2410.15957, 2024

    Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera-controlled image-to-video dif- fusion model.arXiv preprint arXiv:2410.15957, 2024. 2

    2024 arXiv

  35. [43]

    Vidcraft3: Camera, object, and lighting control for image-to-video generation

    Sixiao Zheng, Zimian Peng, Yanpeng Zhou, Yi Zhu, Hang Xu, Xiangru Huang, and Yanwei Fu. Vidcraft3: Camera, object, and lighting control for image-to-video generation. arXiv preprint arXiv:2502.07531, 2025. 3

    2025 arXiv

  36. [44]

    Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024. 1

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.