Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Optical flow can replace camera-parameter labels for camera-controlled video synthesis, letting models train on arbitrary videos.

desk verdict A useful step: optical flow replaces ground-truth camera parameters as the training signal for camera-controlled video diffusion, with a physically approximate flow fusion that the paper acknowledges but under-stresses. read the letter →

arxiv 2502.08244 v2 pith:BUVWN4LL submitted 2025-02-12 cs.CV

Wonjoon Jin , Qi Dai , Chong Luo , Seung-Hwan Baek , Sunghyun Cho This is my paper · ORCID
classification cs.CV
keywords camera-controllablevideosynthesisopticalflowdiffusionmodelobjectmotionflow-conditionedlatent3Dwarpingcameratrajectorycontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that optical flow, the per-pixel motion between video frames, can serve as both the training signal and the control interface for camera-controlled video generation. Because optical flow is directly estimable from ordinary videos, FloVD trains on footage with moving objects without needing ground-truth camera parameters. This breaks the dependence on restricted, static-scene datasets and enables detailed camera trajectories through background-flow 3D cues while preserving natural object motion.

What carries the argument

The mechanism is a two-stage flow-then-video pipeline. Stage one generates camera flow maps by 3D-warping an estimated depth map under the input camera trajectory, and object flow maps with an object-motion synthesis model (OMSM) whose VAE encoder and decoder are borrowed and repurposed for flow data; the flows are combined through a segmentation mask into camera-object flow maps. Stage two is a flow-conditioned video synthesis model (FVSM), a latent video diffusion model with a T2I-Adapter-style flow encoder that injects multi-level flow embeddings into the denoising U-Net. Crucially, both models are trained on optical flow estimated from arbitrary videos, so the system learns to interpret flow as a control signal without ever seeing camera parameters during training.

What would settle it

Render or capture scenes where a large foreground object moves toward or away from the camera under strong camera translation, feed the paper's approximate integrated flow maps to the trained FVSM, and compare object-boundary distortion and camera-trajectory error against variants using physically correct 3D-aware flow maps. If the approximate inputs degrade synthesis measurably, the central assumption fails.

Watch

Extended reading notes

Core claim

FloVD's central proposal is to condition a video diffusion model on optical-flow maps that separate camera-induced motion from object motion. Camera flow is produced by unprojecting an estimated depth map and re-projecting it under the input camera trajectory; object flow is synthesized by a dedicated latent video diffusion model. The two flow sets are composited using an object mask into camera-object flow maps, which a second flow-conditioned diffusion module turns into video frames. The paper shows that this flow-based conditioning reaches camera-control accuracy on RealEstate10K comparable to or better than baselines trained with ground-truth camera parameters, and produces more natural object motion on benchmarks with large object displacements.

Load-bearing premise

The flow-conditioned video synthesis model, trained on flow maps estimated from real videos, will produce natural video even when the flow maps it receives at inference are physically inconsistent because object motion lacks depth (z-axis) information.

Editorial extensions

If this is right

  • Any video collection can become training data for camera-controllable synthesis, so the method can scale with uncurated web video rather than limited camera-annotated datasets.
  • Camera movement and object movement are both expressed as flow, letting a user specify a camera path while the object motion model freely synthesizes foreground dynamics.
  • The same flow-conditioned module can perform temporally-consistent video editing by re-synthesizing frames from an edited first frame and the video's own optical flow.
  • Compound shots such as dolly zoom are expressible without special handling of camera intrinsics, as the flow representation absorbs the combined geometric effect.
  • Because FVSM is trained on flow maps that may contain warping artifacts, it can repair holes and distortions from 3D warping, turning a crude geometric proxy into clean video frames.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the approach scales, a natural extension is text- or prompt-conditioned object flow generation, letting a user specify what moves while the camera trajectory stays exact.
  • The model's tolerance for physically approximate flow suggests that other approximate 3D priors, such as monocular depth sequences or point maps, could feed the same flow-conditioning interface without retraining the backbone.
  • A regime the paper does not quantify is large object depth variation combined with strong camera translation; targeted evaluations there could map the boundary of the correction assumption.
  • The flow-based representation might transfer across domains more easily than camera-parameter embeddings because flow is a modality-agnostic, pixel-level quantity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. FloVD is a two-stage image-to-video synthesis framework in which stage 1 generates optical flow maps for camera motion (by unprojecting an estimated depth map and warping with the input camera parameters) and for object motion (via a latent video diffusion model, OMSM), then integrates them using a segmentation mask; stage 2 feeds the integrated camera-object flow maps into a flow-conditioned video synthesis model (FVSM) built on Stable Video Diffusion with a T2I-Adapter-style flow encoder. The paper claims two main benefits: training does not require ground-truth camera parameters, and background optical flow encodes enough 3D structure for detailed camera control. The evaluation measures camera controllability on RealEstate10K and video quality on Pexels-based benchmarks, with ablations of OMSM and large-scale training.

Significance. If the central claim holds, FloVD would be a meaningful step toward camera-controllable video synthesis trained on arbitrary videos rather than pose-annotated static scenes, and the paper provides several concrete pieces of evidence: training a variant on RealEstate10K without using its camera parameters matches or slightly outperforms CameraCtrl on all three camera metrics in Table 1; the Pexels benchmarks in Table 2 show consistent gains in FVD, FID, and IS over MotionCtrl and CameraCtrl; and the ablations in Table 3 and Figure 7 isolate the contributions of OMSM and large-scale data. The main weakness is that the physical inaccuracy of the Eq. (1) integration, and the model's ability to correct it, is only qualitatively supported; no quantitative experiment isolates the combined regime of significant object motion and significant camera motion. The paper does not mention code or data release, which limits reproducibility.

major comments (3)
  1. [Section 3.1, Eq. (1)] The integration of camera and object flow is explicitly acknowledged to be physically inaccurate because object flow lacks z-axis motion and the warping uses the depth at the original pixel x rather than at the displaced position x'. This is load-bearing for the paper's headline claim of simultaneous accurate camera control and natural object motion. The quantitative evaluations do not isolate this combined regime: Table 1 uses RealEstate10K, which has static scenes with no object motion, and the Pexels-small/medium/large benchmarks in Section 5.2 are selected for minimal camera motions. There is no experiment that jointly varies object-motion magnitude and camera-motion magnitude, so the paper does not show whether FVSM absorbs the geometric inconsistency or whether camera drift or object artifacts emerge. I recommend adding an evaluation on sequences with known depth and controlled object z-motion (e.g., rendered or synthetic scenes) or, at minimum, a systematic study of camera-control error and video quality as a function of object-flow magnitude under the same camera trajectory, plus a failure analysis of the combined regime.
  2. [Section 4.3 and Section 5.4] FVSM is trained on optical flow estimated by RAFT from real videos, but at inference it receives clean flow maps produced by depth warping and OMSM. Section 4.3 states that both kinds of flow 'contain camera and object motions in the form of flow vectors,' but this does not establish distributional similarity: estimated flow contains noise, occlusions, and estimation errors, while the synthesized flow is smooth in background regions and geometrically inconsistent in object regions by construction. Figure 8 demonstrates repair of holes and distortions from 3D warping, but holes are not the systematic geometric error in Eq. (1). A direct test of the correction assumption would be to train or fine-tune FVSM on the same type of clean flow maps used at inference and compare downstream metrics, or to measure camera estimation error when the object flow magnitude is varied while holding the camera trajectory fixed.
  3. [Table 1 and Section 5.3] The available evidence already shows a quantitative tradeoff between object motion and camera accuracy: adding OMSM to the RealEstate10K-trained model increases all three camera errors (mRotErr 1.52 vs. 1.43, mTransErr 0.0971 vs. 0.0869, mCamMC 0.0989 vs. 0.0887), and the internal-data version with OMSM has larger errors still (1.88, 0.1042, 0.1066). The paper attributes this to domain differences and object motions, but it does not quantify or analyze the tradeoff. Given the central claim, this degradation should be characterized explicitly, and ideally reported with variance or significance information, since the differences between the best variant and CameraCtrl are small.
minor comments (4)
  1. [Section 5.2] The camera-controllability protocol paragraph is repeated verbatim twice in the same subsection; the duplicate should be removed.
  2. [Table 2] The column headers 'Pexels-med. (<40)' and 'Pexels-large (≥40)' are ambiguous because the threshold refers to average optical-flow magnitude of moving objects, and the 'Pexels-random' column has no threshold; a clearer caption with the exact definitions would help readers interpret the three benchmarks.
  3. [Tables 1 and 3] No confidence intervals, multiple seeds, or significance tests are reported; given the small differences in Table 1 (e.g., mRotErr 1.43 vs. 1.44), reporting variance across seeds or evaluation subsets would strengthen the claims.
  4. [Section 2 and Section 5.3] The Related Work cites several recent camera-control methods (CamCo, CamI2V, VD3D) but the experimental comparison includes only MotionCtrl and CameraCtrl; if a comparison is not feasible, the paper should state why these methods are omitted from the quantitative evaluation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is validated against external benchmarks and the flow-conditioned synthesis is a learned mapping, not a fitted constant or self-citation chain.

full rationale

The paper's load-bearing claim is that optical flow, estimated directly from videos, can replace ground-truth camera parameters as the training signal for camera-controllable video synthesis. This claim is implemented by training OMSM and FVSM with RAFT-estimated flow from real videos; both models are learned via denoising score matching and evaluated on held-out RealEstate10K and Pexels datasets with standard external metrics (mRotErr, mTransErr, mCamMC, FVD, FID, IS). At inference, the camera-object flow maps are synthesized from the input camera parameters, estimated depth, and OMSM, but the output video is not constructed from the evaluation target: camera parameters are re-estimated with GLOMAP, an external SfM pipeline, and the Pexels quality metrics compare against real videos. Eq. (1) explicitly states that the flow integration is not physically accurate and delegates the residual correction to FVSM; this is an openly acknowledged assumption and limitation, not a circular reduction. The curated-dataset threshold and flow-normalization statistics are training-set preprocessing choices, not quantities later presented as predictions. There is one self-citation in the dataset list ([31]), but it is not load-bearing for any central argument. No fitted parameter is renamed as a prediction, and no 'uniqueness theorem' or self-citation is invoked to force the method's design. The central derivation is therefore self-contained and externally checkable.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method is an empirical ML system; the only free parameters are training hyperparameters and normalization statistics. The key domain assumptions are about the reliability of off-the-shelf optical flow, depth, and segmentation modules, plus the VAE's ability to represent flow maps. No new physical entities are introduced.

free parameters (2)
  • Curated dataset threshold = not reported
    A hand-chosen threshold on the average background optical flow magnitude in Section 4.1, used to decide whether a clip has no camera motion for OMSM fine-tuning.
  • Flow normalization statistics = not reported
    Mean and standard deviation of optical flow computed from the training dataset (Section 5.1, following Li et al.), used to scale input flow maps; the paper does not report the numerical values.
assumptions (5)
  • domain assumption Optical flow can be reliably estimated from arbitrary videos using RAFT.
    Section 4.1: RAFT-estimated flow maps are treated as ground truth for training OMSM and FVSM.
  • domain assumption Single-image depth estimation provides sufficiently accurate 3D structure for unprojection and camera-flow computation.
    Section 3.1: depth from an off-the-shelf network is used to generate camera flow maps without verification of per-pixel accuracy.
  • domain assumption The latent video diffusion VAE can encode and decode optical flow maps with negligible error using two RGB channels plus their average.
    Section 3.1: the authors state they verified this but provide no numeric error.
  • domain assumption SAM2 binary segmentation reliably separates moving objects from static background.
    Section 3.1: the mask is used in flow integration; the Limitations section acknowledges estimation errors.
  • standard math Denoising score matching is a valid training objective for the two diffusion models.
    Sections 4.2 and 4.3 train via denoising score matching following Karras et al.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis." pith.science (2026). https://pith.science/paper/BUVWN4LL

@misc{pith2026250208244,
  author       = {Pith},
  title        = {Pith review of: FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BUVWN4LL}},
  note         = {Machine review of arXiv:2502.08244}
}
read the original abstract

We present FloVD, a novel video diffusion model for camera-controllable video generation. FloVD leverages optical flow to represent the motions of the camera and moving objects. This approach offers two key benefits. Since optical flow can be directly estimated from videos, our approach allows for the use of arbitrary training videos without ground-truth camera parameters. Moreover, as background optical flow encodes 3D correlation across different viewpoints, our method enables detailed camera control by leveraging the background motion. To synthesize natural object motion while supporting detailed camera control, our framework adopts a two-stage video synthesis pipeline consisting of optical flow generation and flow-conditioned video synthesis. Extensive experiments demonstrate the superiority of our method over previous approaches in terms of accurate camera control and natural object motion synthesis.

Figures

Figures reproduced from arXiv: 2502.08244 by the authors.

Figure 1
Figure 1. (Left) Our method using optical flow enables video synthesis with complex camera movements (dolly zoom). (Right) Synthesized [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of FloVD. Given an image and camera parameters, our framework synthesizes video frames following the input camera [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Network architectures of OMSM and FVSM. as input, and iteratively denoises the latent feature volume to synthesize latent object motion flow maps. Finally, the VAE decoder decodes the synthesized result and produces object flow maps F o . Inspired by Marigold [15], we utilize the VAE decoder of the latent video diffusion model, which is trained on RGB images, for decoding object flow maps without any archi￾tectural … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Object flow maps synthesized by OMSM, which is [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison of camera control using the RealEstate10K test dataset [ [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison of video synthesis quality. Video frames are synthesized with ’stop’ camera motion. X-t slice reveals how [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Explicit camera control. Our model can follow the [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance

    cs.CV 2025-05 conditional novelty 6.0 of 10

    EPiC trains a 30M-parameter visibility-aware ControlNet on mask-based anchor videos from 5,000 in-the-wild videos and 500 steps, reaching SOTA camera accuracy on RealEstate10K and MiraData.

Reference graph

Works this paper leans on

43 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    https:// www.pexels.com

    Pexels, royalty-free stock footage website. https:// www.pexels.com. Accessed: 2024-09-30. 4, 5, 7

    work page 2024

  2. [2]

    Vd3d: Taming large video diffu- sion transformers for 3d camera control

    Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siaro- hin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffu- sion transformers for 3d camera control. arXiv preprint arXiv:2407.12781, 2024. 1, 2

    arXiv 2024

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023. 1, 3, 4, 5

    arXiv 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In CVPR, pages 22563–22575, 2023

    work page 2023

  5. [5]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  6. [6]

    Motion-conditioned diffu- sion model for controllable video synthesis

    Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung- Yi Lin, and Ming-Hsuan Yang. Motion-conditioned diffu- sion model for controllable video synthesis. arXiv preprint arXiv:2304.14404, 2023. 1, 2

    arXiv 2023

  7. [7]

    Animating Landscape: Self-Supervised Learning of Decoupled Motion and Appearance for Single-Image Video Synthesis

    Yuki Endo, Yoshihiro Kanamori, and Shigeru Kuriyama. Animating landscape: self-supervised learning of decoupled motion and appearance for single-image video synthesis. arXiv preprint arXiv:1910.07192, 2019. 2

    work page Pith review arXiv 1910

  8. [8]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2

    arXiv 2023

Show all 43 references
  1. [9]

    Cameractrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 1, 2, 5, 6, 7

    2024 arXiv

  2. [10]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  3. [11]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 6

    2017

  4. [12]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 1

    2022

  5. [13]

    Animating pictures with eulerian mo- tion fields

    Aleksander Holynski, Brian L Curless, Steven M Seitz, and Richard Szeliski. Animating pictures with eulerian mo- tion fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5810– 5819, 2021. 2

    2021

  6. [14]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. 5, 7

    2022

  7. [15]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9492...

    2024

  8. [16]

    Col- laborative video diffusion: Consistent multi-video genera- tion with camera control

    Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hong- sheng Li, Leonidas Guibas, and Gordon Wetzstein. Col- laborative video diffusion: Consistent multi-video genera- tion with camera control. arXiv preprint arXiv:2405.17414,

    arXiv

  9. [17]

    Generative image dynamics

    Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24142–24153, 2024. 2, 5

    2024

  10. [18]

    Movideo: Motion-aware video generation with diffusion model

    Jingyun Liang, Yuchen Fan, Kai Zhang, Radu Timofte, Luc Van Gool, and Rakesh Ranjan. Movideo: Motion-aware video generation with diffusion model. In European Con- ference on Computer Vision, pages 56–74. Springer, 2024

    2024

  11. [19]

    Controllable animation of fluid elements in still images

    Aniruddha Mahapatra and Kuldeep Kulkarni. Controllable animation of fluid elements in still images. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3667–3676, 2022. 2

    2022

  12. [20]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 2024. 4, 5

    2024

  13. [21]

    Conditional image-to-video gener- ation with latent flow diffusion models

    Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video gener- ation with latent flow diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18444–18455, 2023. 2

    2023

  14. [22]

    Global structure-from-motion revisited

    Linfei Pan, D ´aniel Bar ´ath, Marc Pollefeys, and Johannes L Sch¨onberger. Global structure-from-motion revisited. In Eu- ropean Conference on Computer Vision (ECCV), 2024. 5 9

    2024

  15. [23]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  16. [24]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 4

    2024 arXiv

  17. [25]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. 6

    2016

  18. [26]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers , p...

    2024

  19. [27]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  20. [28]

    Light field networks: Neu- ral scene representations with single-evaluation rendering

    Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neu- ral scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems , 34: 19313–19325, 2021. 1, 2

    2021

  21. [29]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  22. [30]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  23. [31]

    Microcinema: A divide-and- conquer approach for text-to-video generation

    Yanhui Wang, Jianmin Bao, Wenming Weng, Ruoyu Feng, Dacheng Yin, Tao Yang, Jingxu Zhang, Qi Dai, Zhiyuan Zhao, Chunyu Wang, et al. Microcinema: A divide-and- conquer approach for text-to-video generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  24. [32]

    Humanvid: Demystifying training data for camera-controllable human image animation

    Zhenzhi Wang, Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan, Kai Chen, Tianfan Xue, Bo Dai, et al. Humanvid: Demystifying training data for camera-controllable human image animation. arXiv preprint arXiv:2407.17438, 2024. 1

    2024 arXiv

  25. [33]

    Mo- tionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tian- shui Chen, Menghan Xia, Ping Luo, and Ying Shan. Mo- tionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers , pages 1–11, 2024. 1, 2, 6, 7

    2024

  26. [34]

    Cavia: Camera-controllable multi-view video diffusion with view-integrated attention

    Dejia Xu, Yifan Jiang, Chen Huang, Liangchen Song, Thorsten Gernoth, Liangliang Cao, Zhangyang Wang, and Hao Tang. Cavia: Camera-controllable multi-view video diffusion with view-integrated attention. arXiv preprint arXiv:2410.10774, 2024

    arXiv

  27. [35]

    Camco: Camera- controllable 3d-consistent image-to-video generation

    Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera- controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509, 2024. 1, 2

    2024 arXiv

  28. [36]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv preprint arXiv:2406.09414, 2024. 3

    2024 arXiv

  29. [37]

    Direct-a-video: Customized video generation with user- directed camera movement and object motion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user- directed camera movement and object motion. In ACM SIG- GRAPH 2024 Conference Papers, pages 1–12, 2024. 1, 2

    2024

  30. [38]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 1

    2024 arXiv

  31. [39]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089, 2023. 1, 2

    2023 arXiv

  32. [40]

    Recapture: Gener- ative video camera controls for user-provided videos using masked video fine-tuning

    David Junhao Zhang, Roni Paiss, Shiran Zada, Nikhil Kar- nad, David E Jacobs, Yael Pritch, Inbar Mosseri, Mike Zheng Shou, Neal Wadhwa, and Nataniel Ruiz. Recapture: Gener- ative video camera controls for user-provided videos using masked video fine-tuning. arXiv preprint arXi...

    arXiv

  33. [41]

    Thin-plate spline motion model for image animation

    Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3657–3666, 2022. 2

    2022

  34. [42]

    Cami2v: Camera-controlled image-to-video dif- fusion model

    Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera-controlled image-to-video dif- fusion model. arXiv preprint arXiv:2410.15957, 2024. 1, 2, 5

    2024 arXiv

  35. [43]

    Stereo magnification: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817, 2018. 2, 5, 6, 7 10

    2018 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.