Pith. sign in

REVIEW 2 major objections 5 minor 53 references

UniCaMo controls object and camera motion together by building the diffusion model’s input noise in a shared 3D space.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:58 UTC pith:BPZX6H4V

load-bearing objection Solid systems paper: 3D-track noise warping plus a sphere for disocclusions gives real joint object+camera control without adapters; MoveBench gains look real, but evaluation partly rides the same estimators used to build the noise. the 2 major comments →

arxiv 2607.02798 v1 pith:BPZX6H4V submitted 2026-07-02 cs.CV cs.GR

Track the Noise, Move the World:3D-Grounded Motion-Consistent Noise for Controllable Video Generation

Long Vu , Tan Ngo , Animesh Karnewar , Amir Habibian , Binh-Son Hua , Hung Bui , Minh Hoai Nguyen , Phong Nguyen-Ha This is my paper
classification cs.CV cs.GR
keywords controllable video generationimage-to-video diffusion3D point tracksnoise warpingcamera motion controlobject motion controlLoRA fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Image-and-text-to-video diffusion models make realistic clips by denoising random noise, yet they still struggle to obey both object paths and camera paths at once. UniCaMo solves that by constructing the starting noise itself so that it already encodes the desired 3D dynamics. Sparse 3D point tracks copy local patches of the first-frame noise along object trajectories (with a depth test that resolves occlusions), while a virtual sphere coated with Gaussian noise supplies consistent values for newly revealed background as the camera moves. The two sources are simply composited into one motion-consistent latent that replaces ordinary random noise. Because only the input changes, the method needs no extra adapters or architectural surgery; a light LoRA fine-tune of a large pretrained model is enough. On the MoveBench benchmark it reports better visual quality and lower trajectory error than prior controllers that treat object and camera motion separately or that rely on 2D tracks alone.

Core claim

A shared 3D-grounded motion-consistent noise space—local track-guided warping of first-frame Gaussian noise plus global sphere-based sampling for disocclusions—lets a single diffusion generation process obey both object trajectories and camera viewpoints at once, without auxiliary control modules, and yields state-of-the-art quality and controllability after lightweight LoRA fine-tuning.

What carries the argument

3D-grounded motion-consistent noise: Track-the-Noise warps first-frame Gaussian patches along sparse 3D point tracks with depth-aware collision resolution; Move-the-World samples a virtual Gaussian-textured sphere for view-consistent fill of newly visible regions; the two are composited into the diffusion model’s initial latent.

Load-bearing premise

The method assumes that off-the-shelf monocular depth, pose, and sparse 3D track estimators remain accurate enough, both when built from real training videos and when inferred from a single image plus user edits, for the warping and depth-ordering rules to stay geometrically valid.

What would settle it

On MoveBench (or an equivalent joint-control set), replace the estimated 3D tracks and poses with deliberately noisy or depth-inconsistent versions; if end-point error and visual coherence then collapse to the level of 2D-track baselines, the claim that the 3D noise construction is what carries controllability is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. UniCaMo claims that joint object-and-camera motion control for image-and-text-to-video diffusion can be achieved by constructing a 3D-grounded motion-consistent input noise tensor rather than by adding adapters or control branches. Sparse 3D point tracks warp first-frame Gaussian noise along object trajectories with a depth-aware collision rule (Track-the-Noise), while a virtual sphere with a Gaussian texture supplies view-consistent noise for disocclusions under camera motion (Move-the-World). The combined noise initializes a LoRA-fine-tuned Wan 2.1 (14B) I2V model. On MoveBench single- and multi-object splits the method reports SOTA FID/FVD/PSNR/SSIM/EPE, with ablations on sphere noise, degradation rate θ, track count, and patch radius supporting the design.

Significance. If the result holds under independent 3D guidance, the paper offers a clean, architecture-preserving route to unified motion control that is practically attractive: only input noise is modified, overhead is ~0.2 s, and large pretrained backbones remain intact. The combination of sparse 3D tracks with sphere-based disocclusion handling is a concrete advance over 2D-track and pure flow-warping priors, and the MoveBench gains (including a matched-track Wan-Move variant) plus design ablations are useful empirical evidence. The work is therefore of clear interest for controllable video generation and embodied-AI data synthesis, provided evaluation circularity and estimator dependence are addressed.

major comments (2)
  1. §4.1 and Table 1: MoveBench supplies only 2D tracks and first-frame masks; the 3D tracks and camera poses required by UniCaMo are recovered with the same TAPIP3D+ViPE stack used to build training noise and the depth-aware z-buffer (§3.1). The matched Wan-Move+TAPIP3D variant still receives only 2D projections of those tracks, so UniCaMo alone enjoys full 3D depth ordering and sphere sampling that are definitionally consistent with the recovered signals. Reported EPE/FID/FVD gains and occlusion-aware qualitative results (Figs. 1, 4–5) may therefore largely measure exploitation of estimator-consistent noise rather than geometric validity under independent, user-specified, or noisy 3D trajectories. A load-bearing experiment is needed: evaluate with independently annotated or deliberately perturbed 3D tracks/poses (or a held-out estimator), and report EPE/quality under controlled track/pose
  2. §3.2 and Sec. C: Move-the-World rests on a single bounding sphere with latitude–longitude Gaussian texture as a proxy for all newly revealed background. The paper itself notes that this coarse proxy may fail for complex geometry and that tracking/calibration errors degrade controllability. No quantitative stress test of sphere radius (τ), environment-map resolution, or non-spherical backgrounds appears in the main results; the sphere ablation (Table 2a, Fig. 6) only contrasts sphere vs. random fill. Without such isolation, the claim that the construction remains geometrically coherent under large viewpoint change is only partially supported.
minor comments (5)
  1. Title/section headers use both UniCaMo and UniCamMo; standardize the acronym throughout.
  2. §3.1: patch radius is written R=2 (3×3) while the appendix uses r; unify notation and state the latent-scale ratio (Wlat/W) once with a concrete example for Wan 2.1.
  3. Table 1: Go-with-the-Flow is said not to produce meaningful videos under the joint-control protocol; a short failure-mode description or supplementary figure would make the comparison more informative.
  4. Fig. 1 caption and body text contain minor grammar issues (e.g., "A women walk"); a light copy-edit pass would help.
  5. Inference GUI and export format (Sec. B.3) are useful; a brief note on how user-edited tracks are validated against monocular depth would strengthen reproducibility claims.

Circularity Check

0 steps flagged

No derivation circularity: UniCaMo is a noise-construction + LoRA method evaluated on external quality and trajectory metrics, not a self-defining prediction.

full rationale

This is an engineering method paper, not a first-principles derivation. The claimed chain is: (i) extract 3D tracks/poses from video (or user GUI), (ii) warp first-frame Gaussian noise along tracks with depth-aware collision and fill disocclusions via sphere sampling, (iii) LoRA-fine-tune a pretrained I2V diffusion model on that structured noise, (iv) report FID/FVD/PSNR/SSIM/EPE on MoveBench. None of these steps reduces a “prediction” to its inputs by construction. Success is not defined as equality to a fitted parameter; EPE measures whether the generated video follows the control tracks, which the model can fail to do. Shared use of TAPIP3D/ViPE for training noise and for recovering MoveBench 3D signals is a possible experimental confound (estimator-consistent inputs favor the method that consumes full 3D), but that is evaluation design risk, not circular derivation under the stated patterns. There is no self-definitional loop, no fitted quantity renamed as prediction, no load-bearing uniqueness theorem from overlapping authors, and no ansatz smuggled in as a theorem. The paper is self-contained against external benchmarks; score 0 is appropriate.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard diffusion and 3D-vision tooling plus several hand-chosen construction hyperparameters and two invented noise representations. No free parameters are fit to the final MoveBench metrics in a closed loop; the free parameters are design choices validated by ablation. Invented entities are engineering constructs (structured noise fields), not new physical particles, and have no independent evidence outside generation quality.

free parameters (6)
  • patch radius R = 2
    Controls size of noise footprint copied along each track; default R=2 (3×3) chosen by ablation on PSNR/SSIM/EPE/CLIP.
  • degradation rate θ = 0.2
    Alpha-blends constructed motion noise with pure Gaussian noise to restore temporal Gaussianity; default 0.2 from ablation table.
  • sphere radius scale τ = 1.2
    Multiplies max camera-center displacement when setting sphere radius; fixed at 1.2 with d0_max depth bound.
  • number of 3D tracks N = 512–1024
    Training randomly samples 512–1024 tracks; inference uses N=1024 with mask-proportional allocation.
  • LoRA rank and α = rank 64, α=1
    Adapter capacity on DiT denoiser; rank 64, α=1, trained 10K steps.
  • environment map scale k = 4
    Sphere texture resolution relative to video resolution; set to 4.
axioms (5)
  • domain assumption Structured initialization noise that remains approximately per-frame Gaussian can steer a pretrained video diffusion model toward desired motion without changing architecture.
    Inherited from noise-warping literature (Go-with-the-Flow, HIWYN) and used throughout Sec. 3.
  • domain assumption Sparse 3D point tracks plus camera poses recovered by off-the-shelf estimators (TAPIP3D, ViPE) are accurate enough to encode object and camera motion for noise warping.
    Training data construction and inference GUI both depend on these tools (Sec. 3, 3.3, A.1).
  • ad hoc to paper A single bounding sphere with latitude-longitude Gaussian texture supplies view-consistent noise for newly revealed regions under camera motion.
    Core of Move-the-World (Sec. 3.2); paper later admits the sphere is a coarse proxy.
  • domain assumption Depth-aware z-buffer selection among colliding tracks yields physically plausible occlusion ordering in noise space.
    Sec. 3.1 Depth-aware Collision Tracking; standard graphics compositing assumption applied to noise.
  • domain assumption Flow-matching / diffusion training with LoRA on the DiT is sufficient for the model to exploit the structured noise prior.
    Training protocol in Sec. 3.3 on Wan2.1-I2V-A14B.
invented entities (2)
  • 3D-grounded motion-consistent noise space no independent evidence
    purpose: Unified latent initializer that jointly encodes object trajectories and camera viewpoints.
    Name and construction introduced in abstract and Sec. 3; evaluated only via generation metrics, no external physical measurement.
  • virtual spherical noise representation (Move-the-World sphere) no independent evidence
    purpose: Provide globally consistent Gaussian values for disoccluded regions as the camera moves.
    Sec. 3.2 parameterization and ray-sphere sampling; ablated in Fig. 6 / Table 2 but not independently measured outside the generator.

pith-pipeline@v1.1.0-grok45 · 21526 in / 3917 out tokens · 36911 ms · 2026-07-12T06:58:38.050569+00:00 · methodology

0 comments
read the original abstract

Modern image-and-text-to-video diffusion models can synthesize highly realistic videos by iteratively denoising an initial Gaussian noise tensor conditioned on reference image and text inputs. However, existing approaches still lack precise and unified controllability over both object motion and camera motion within a single generation process. We present UniCaMo, a unified framework that enables simultaneous control of object trajectories and camera viewpoints by directly constructing the input noise of the diffusion model. Specifically, UniCaMo builds a shared 3D-grounded motion-consistent noise space across latent video frames. Sparse 3D point tracks are used to warp the Gaussian noise of the reference frame along desired object trajectories, while a virtual spherical noise representation provides globally consistent noise values for newly revealed scene regions under camera motion. By combining local track-guided noise warping with global sphere-based noise sampling, UniCaMo maintains geometric and temporal consistency under both object movement and viewpoint changes. Because UniCaMo modifies only the input noise, it requires no auxiliary adapters, control branches, or architectural changes to the underlying video diffusion model. With lightweight LoRA fine-tuning on large pretrained video diffusion models, including Wan 2.1 (14B), UniCaMo achieves state-of-the-art results in both video quality and motion controllability on standard controllable video generation benchmarks.

Figures

Figures reproduced from arXiv: 2607.02798 by Amir Habibian, Animesh Karnewar, Binh-Son Hua, Hung Bui, Long Vu, Minh Hoai Nguyen, Phong Nguyen-Ha, Tan Ngo.

Figure 1
Figure 1. Figure 1: UniCaMo jointly controls object + camera motion with 3D tracks and spherical noise, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Our framework trains a video diffusion model by warping initialization noise using 3D [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Noise representation in our two key steps, Track the Noise (a) and Move the World (b). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results on MoveBench between UniCaMo and recent approaches [ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative results on complex in-the-wild samples between our proposed UniCaMo and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ablation results on the 3D sphere noise. Without 3D-consistent sphere noise sampling, the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: illustrates our inference pipeline. Given a single input image, we first reconstruct the scene using ViPE [22] to obtain a 3D point cloud, depth map, and camera intrinsics. The user then specifies desired object and camera motions through our interactive GUI (Sec. B.3), which exports guidance signals in the form of 3D point tracks and camera poses. These signals are fed into our Guidance￾based Noise Warpin… view at source ↗
Figure 8
Figure 8. Figure 8: We demonstrate control of the camera motion, single-object movement, multiple-object [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Additional qualitative comparison on MoveBench (1/3). Our method produces more [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Additional qualitative comparison on MoveBench (2/3). Under complex object interactions, [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Additional qualitative comparison on MoveBench (3/3). [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Effect of degradation rate on generated samples. A lower degradation rate ( [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative results of our proposed UniCaMo method using different number of 3D point [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative results of our proposed UniCaMo method using different configuration of path [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Visualization of our interactive GUI for unified camera and object motion control. Users [PITH_FULL_IMAGE:figures/full_fig_p018_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 20 linked inside Pith

  1. [1]

    Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Lyra: Generative 3d scene reconstruction via video diffusion model self-distillation.arXiv preprint arXiv:2509.19296, 2025

    Sherwin Bahmani, Tianchang Shen, Jiawei Ren, Jiahui Huang, Yifeng Jiang, Haithem Turki, Andrea Tagliasacchi, David B Lindell, Zan Gojcic, Sanja Fidler, et al. Lyra: Generative 3d scene reconstruction via video diffusion model self-distillation.arXiv preprint arXiv:2509.19296, 2025

    arXiv 2025

  3. [3]

    Ac3d: Analyzing and improving 3d camera control in video diffusion transformers

    Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera control in video diffusion transformers. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 22875–22889, 2025

    2025

  4. [4]

    Recammaster: Camera-controlled generative rendering from a single video

    Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controlled generative rendering from a single video. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 14834–14844, 2025

    2025

  5. [5]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Lumiere: A space-time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space-time diffusion model for video generation. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024

    2024

  7. [7]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Go-with-the-flow: Motion-controllable video diffusion models using real-time warped noise

    Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, et al. Go-with-the-flow: Motion-controllable video diffusion models using real-time warped noise. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 13–23, 2025

    2025

  9. [9]

    Sam 3: Segment anything with concepts, 2025

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane ...

    2025

  10. [10]

    How i warped your noise: a temporally-correlated noise prior for diffusion models.arXiv preprint arXiv:2504.03072, 2025

    Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C Azevedo. How i warped your noise: a temporally-correlated noise prior for diffusion models.arXiv preprint arXiv:2504.03072, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Videocrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    4dnex: Feed-forward 4d generative modeling made easy.arXiv preprint arXiv:2508.13154, 2025

    Zhaoxi Chen, Tianqi Liu, Long Zhuo, Jiawei Ren, Zeng Tao, He Zhu, Fangzhou Hong, Liang Pan, and Ziwei Liu. 4dnex: Feed-forward 4d generative modeling made easy.arXiv preprint arXiv:2508.13154, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Wan-move: Motion-controllable video generation via latent trajectory guidance.arXiv preprint arXiv:2512.08765, 2025

    Ruihang Chu, Yefei He, Zhekai Chen, Shiwei Zhang, Xiaogang Xu, Bin Xia, Dingdong Wang, Hongwei Yi, Xihui Liu, Hengshuang Zhao, et al. Wan-move: Motion-controllable video generation via latent trajectory guidance.arXiv preprint arXiv:2512.08765, 2025

    arXiv 2025

  14. [14]

    Motion prompting: Controlling video generation with motion trajectories

    Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez- Guevara, Yusuf Aytar, Michael Rubinstein, Chen Sun, et al. Motion prompting: Controlling video generation with motion trajectories. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 1–12, 2025. 10

    2025

  15. [15]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    2017

  18. [18]

    Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    Video diffusion models.Advances in neural information processing systems, 35:8633–8646, 2022

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models.Advances in neural information processing systems, 35:8633–8646, 2022

    2022

  20. [20]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    Training-free camera control for video generation.arXiv preprint arXiv:2406.10126, 2024

    Chen Hou and Zhibo Chen. Training-free camera control for video generation.arXiv preprint arXiv:2406.10126, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Vipe: Video pose engine for 3d geometric perception

    Jiahui Huang, Qunjie Zhou, Hesam Rabeti, Aleksandr Korovko, Huan Ling, Xuanchi Ren, Tianchang Shen, Jun Gao, Dmitry Slepichev, Chen-Hsuan Lin, et al. Vipe: Video pose engine for 3d geometric perception. arXiv preprint arXiv:2508.10934, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    VBench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  24. [24]

    Peekaboo: Interactive video generation via masked-diffusion

    Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked-diffusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8079–8088, 2024

    2024

  25. [25]

    Dreamgen: Unlocking generalization in robot learning through video world models.arXiv preprint arXiv:2505.12705, 2025

    Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, et al. Dreamgen: Unlocking generalization in robot learning through video world models.arXiv preprint arXiv:2505.12705, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Pyramidal flow matching for efficient video generative modeling.arXiv preprint arXiv:2410.05954, 2024

    Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling.arXiv preprint arXiv:2410.05954, 2024

    arXiv 2024

  27. [27]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Onlyflow: Optical flow based motion conditioning for video diffusion models

    Mathis Koroglu, Hugo Caselles-Dupré, Guillaume Jeanneret, and Matthieu Cord. Onlyflow: Optical flow based motion conditioning for video diffusion models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 6226–6236, 2025

    2025

  29. [29]

    Collaborative video diffusion: Consistent multi-video generation with camera control.Advances in Neural Information Processing Systems, 37:16240–16271, 2024

    Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas J Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control.Advances in Neural Information Processing Systems, 37:16240–16271, 2024

    2024

  30. [30]

    Generative video motion editing with 3d point tracks.arXiv preprint arXiv:2512.02015, 2025

    Yao-Chih Lee, Zhoutong Zhang, Jiahui Huang, Jui-Hsien Wang, Joon-Young Lee, Jia-Bin Huang, Eli Shechtman, and Zhengqi Li. Generative video motion editing with 3d point tracks.arXiv preprint arXiv:2512.02015, 2025

    arXiv 2025

  31. [31]

    Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance

    Quanhao Li, Zhen Xing, Rui Wang, Hui Zhang, Qi Dai, and Zuxuan Wu. Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 12112–12123, 2025

    2025

  32. [32]

    Lightx2v: Light video generation inference framework, 2025

    LightX2V . Lightx2v: Light video generation inference framework, 2025

    2025

  33. [33]

    Equivdm: Equivariant video diffusion models with temporally consistent noise.arXiv e-prints, pages arXiv–2504, 2025

    Chao Liu and Arash Vahdat. Equivdm: Equivariant video diffusion models with temporally consistent noise.arXiv e-prints, pages arXiv–2504, 2025

    2025

  34. [34]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Gen3c: 3d-informed world-consistent video generation with precise camera control

    Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3c: 3d-informed world-consistent video generation with precise camera control. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6121–6132, 2025. 11

    2025

  36. [36]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. InACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

    2024

  37. [37]

    Time-to-move: Training-free motion controlled video generation via dual-clock denoising.arXiv preprint arXiv:2511.08633, 2025

    Assaf Singer, Noam Rotstein, Amir Mann, Ron Kimmel, and Or Litany. Time-to-move: Training-free motion controlled video generation via dual-clock denoising.arXiv preprint arXiv:2511.08633, 2025

    arXiv 2025

  38. [38]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. InECCV, 2020

    2020

  39. [39]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019

    2019

  40. [40]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Robovip: Multi-view video generation with visual identity prompting augments robot manipulation.arXiv preprint arXiv:2601.05241, 2026

    Boyang Wang, Haoran Zhang, Shujie Zhang, Jinkun Hao, Mingda Jia, Qi Lv, Yucheng Mao, Zhaoyang Lyu, Jia Zeng, Xudong Xu, et al. Robovip: Multi-view video generation with visual identity prompting augments robot manipulation.arXiv preprint arXiv:2601.05241, 2026

    arXiv 2026

  42. [42]

    Levitor: 3d trajectory oriented image-to-video synthesis

    Hanlin Wang, Hao Liu, Fanyi Liu, Linjiang Li, Lin Pan, Wenhao Zhao, Junjie Zhao, and Limin Wang. Levitor: 3d trajectory oriented image-to-video synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    2025

  43. [43]

    Boximator: Generating rich and controllable motions for video synthesis.arXiv preprint arXiv:2402.01566, 2024

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis.arXiv preprint arXiv:2402.01566, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Image conductor: Precision control for interactive video synthesis

    Yaowei Wang, Xintao Chen, Xian Ma, Shiwei Zhou, Ziyang Huang, Yi Wang, Ceyuan Yang, Han Yu, Jiashi Wang, and Ying Shan. Image conductor: Precision control for interactive video synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    2025

  45. [45]

    Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    2004

  46. [46]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. InACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

    2024

  47. [47]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Dynamic view synthesis as an inverse problem.arXiv preprint arXiv:2506.08004, 2025

    Hidir Yesiltepe and Pinar Yanardag. Dynamic view synthesis as an inverse problem.arXiv preprint arXiv:2506.08004, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory.arXiv preprint arXiv:2308.08089, 2023

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory.arXiv preprint arXiv:2308.08089, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Tapip3d: Tracking any point in persistent 3d geometry.arXiv preprint arXiv:2504.14717, 2025

    Bowei Zhang, Lei Ke, Adam W Harley, and Katerina Fragkiadaki. Tapip3d: Tracking any point in persistent 3d geometry.arXiv preprint arXiv:2504.14717, 2025

    arXiv 2025

  51. [51]

    Tora: Trajectory-oriented diffusion transformer for video generation

    Zhenghao Zhang, Junchao Liao, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented diffusion transformer for video generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 2063–2073, 2025

    2063

  52. [52]

    Versecrafter: Dynamic realistic video world model with 4d geometric control.arXiv preprint arXiv:2601.05138, 2026

    Sixiao Zheng, Minghao Yin, Wenbo Hu, Xiaoyu Li, Ying Shan, and Yanwei Fu. Versecrafter: Dynamic realistic video world model with 4d geometric control.arXiv preprint arXiv:2601.05138, 2026

    arXiv 2026

  53. [53]

    Posecrafter: One-shot personalized video synthesis following flexible pose control

    Yong Zhong, Min Zhao, Zebin You, Xiaofeng Yu, Changwang Zhang, and Chongxuan Li. Posecrafter: One-shot personalized video synthesis following flexible pose control. InEuropean conference on computer vision, pages 243–260. Springer, 2024. 12 Technical appendices and supplementary material A Implementation Details A.1 Inference Pipeline Fig. 7 illustrates o...

    2024