Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MotionCanvas claims that a user-drawn 3D camera path and object motions can be translated into 2D point tracks and bounding boxes that a video diffusion model follows — no 3D training data required.

desk verdict Solid empirical systems paper for joint camera-object control in I2V; conditional accept, with the depth-translation gap needing a real response from the authors. read the letter →

arxiv 2502.04299 v1 pith:AIUDRILN submitted 2025-02-06 cs.CV

Jinbo Xing , Long Mai , Cusuh Ham , Jiahui Huang , Aniruddha Mahapatra , Chi-Wing Fu , Tien-Tsin Wong , Feng Liu This is my paper · ORCID
classification cs.CV
keywords image-to-videogenerationcinematicshotdesigncameramotioncontrolobjectscene-spacepoint-trajectoryconditioningbounding-boxmonoculardepth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MotionCanvas sets out to prove that cinematic shot design can be done directly on a single photograph: the user plans a 3D camera path and decides where objects move in scene space, and the system turns that plan into a generated video that follows it. The authors' central claim is that this scene-space intent can be translated into the 2D screen-space signals a video diffusion model is trained to obey — point trajectories for the camera, bounding-box sequences for object global motion, and points for local motion — using a monocular depth estimate and classical projective geometry, so no 3D-labeled video is needed for training. If the claim holds, a creator gets joint control of camera and object motion that prior methods deliver separately or not at all, in a zero-shot setting that beats camera-control models trained on 3D-annotated data. The payoff is practical: filmmakers and editors could storyboard shots by drawing, rather than by writing prompts or reconstructing scenes in 3D.

What carries the argument

The load-bearing mechanism is the Motion Signal Translation module, which connects scene-space design to screen-space conditioning by treating camera motion, object global motion, and object local motion as a hierarchy of transformations applied on top of each other. It uses a monocular depth estimate (with the camera intrinsics it implies) to lift the design to 2.5D, warps along the specified camera path, and reprojects to screen space via the two projection equations (Eq. (1) for boxes, Eq. (2) for points). The complementary machinery is the conditioning format of the video model: point trajectories are compressed into a small set of Discrete Cosine Transform coefficients ($K=10$) and injected as in-context tokens, while bounding-box sequences are rasterized into unique color-coded masks, passed through the same 3D video autoencoder as the image latents, and added to the noisy tokens. This design keeps camera signals and object signals separate, avoids the ambiguity of raw Euclidean coordinate tokens, and adds only about 1.1 percent more tokens over the base video model.

What would settle it

Run only the Motion Signal Translation module on inputs whose true depth and intrinsics are known (for example, rendered scenes or benchmark frames with ground-truth poses), specify a known camera path and object motion, and compare the module's warped point tracks and projected boxes against exact projective ground truth; if the translation error on those inputs equals the end-to-end camera-motion error on the same inputs, then depth estimation — not the video model — is the bottleneck. A cheaper version of the same test: corrupt the estimated depth by a controlled amount and check whether the reported rotation and translation errors degrade by a corresponding amount.

Watch

Extended reading notes

Core claim

MotionCanvas claims that user motion intent for a video shot can be specified in 3D scene space and then translated into the 2D conditioning format of a video diffusion model, eliminating the need for 3D-supervised training. The translation module decomposes the design hierarchically: camera motion is represented as 2D point tracks obtained by sampling static background points and warping them along the camera path with estimated depth; object global motion is represented as scene-anchored bounding boxes lifted to 2.5D and reprojected through the camera transformation (Eq. (1)); object local motion is represented as point trajectories transformed by both camera and object-global motion (Eq. (2)). The resulting signals condition a Diffusion Transformer — a token-based generative video model — fine-tuned with a flow-matching objective on roughly 600K videos annotated only by automatically extracted bounding boxes and point tracks, with trajectories compressed to $K=10$ Discrete Cosine Transform coefficients and boxes rasterized into color-coded masks added to the latents. The authors report that on a real-estate video benchmark this achieves lower camera rotation and translation error than camera-control baselines that were trained on that same benchmark, while on an object-motion benchmark it lowers object-motion error below drag-based and tracklet-based methods; a user study favors its motion adherence, motion quality, and frame fidelity. The paper presents these results as evidence that 3D-aware joint control is achievable in image-to-video synthesis without costly 3D training data.

Load-bearing premise

The entire translation from 3D intent to 2D control signals assumes that a single-image depth estimator returns accurate, scale-consistent depth for the user's input photo; if that depth is wrong, the warped trajectories and reprojected boxes will not match the motion the user drew, and the video model has no way to detect or correct the mismatch.

Editorial extensions

If this is right

  • Camera and object control are decoupled: the paper shows videos that share one camera path while the object follows different specified motions, so a creator can lock the shot and explore options, or freeze the object and vary the camera.
  • Because training needs only bounding boxes and point tracks that can be extracted automatically from ordinary video, the method is not confined to the static-scene datasets that limited 3D-supervised camera control — the paper argues this is what lets it generalize to open-domain photos.
  • The same screen-space signal format transfers across tasks: motion can be copied from a source video onto a structurally similar still image, and edits applied to a video's first frame can be propagated through the whole clip.
  • Camera control includes intrinsics, so cinematic effects like dolly zoom — changing focal length while moving the camera — become expressible, something the camera-control baselines cannot do.
  • Long shots are reachable by splitting the design into segments and generating autoregressively with overlapping 16-frame conditioning, with the user re-specifying or refining motion per segment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the authors do not draw: because camera motion is carried entirely by point tracks on static background, the same trajectory-conditioning format should drive camera-only re-animation of an existing video, not just generation from a single frame — the paper's motion-transfer experiments point that way but stop at still images.
  • The paper leaves depth as the unmeasured frontier: I would expect the method's control accuracy to track the accuracy of the single-image depth estimate, so swapping the depth estimator or restricting inputs to near-planar scenes should measurably change rotation and translation error, giving an external check on where the pipeline's error budget actually sits.
  • If the conditioning format is the key contribution, it should transplant: the paper fine-tunes only the transformer blocks plus a few new linear layers while freezing everything else, so the same DCT-token and color-mask conditioning could in principle be added to other Diffusion Transformer video generators as a motion layer.
  • The collision between text and motion is a foreseeable design problem: the paper acknowledges that explicit motion control overrides a contradictory prompt, so a harmonization step — for instance, rewriting the prompt to match the drawn motion plan before generation — is the natural next control to build.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents MotionCanvas, an image-to-video (I2V) synthesis system that lets a user jointly specify camera motion and object global/local motion in a scene-aware manner using camera paths, scene-anchored bounding boxes, and point trajectories. A Motion Signal Translation module converts these scene-space intents into screen-space conditioning signals by warping depth-estimated points and reprojecting bounding boxes via Eq. (1) and (2), and a DiT-based video diffusion model is fine-tuned on these 2D signals (DCT-coded point trajectories and color-coded bbox masks). Quantitative experiments compare camera control against MotionCtrl and CameraCtrl on RealEstate10K, object motion control against DragAnything, MOFA-Video, and TrackDiffusion on VIPSeg, and report a 35-participant user study plus ablations. The main claim is that users can reliably translate intuitive 3D motion designs into videos that follow both camera and object motion, without requiring 3D-annotated training videos.

Significance. If the central claim holds, MotionCanvas is a practically valuable step for controllable I2V generation: it provides a unified interface for camera and object motion, avoids costly 3D supervision by training only on 2D point tracks and bounding boxes, and shows measurable gains over several published baselines on external benchmarks. Strengths include the use of RealEstate10K and VIPSeg rather than only qualitative examples, a formal user study, explicit ablations for the conditioning representations, and a compact DCT-based trajectory encoding that lowers token cost. No circularity is present: the projection equations are standard, and the evaluations use external data and baselines. The main caveat is that the '3D-aware' reliability is inherited from an unvalidated monocular depth translation step, so the current evidence is strongest for RealEstate10K-like static scenes and weaker for open-domain joint control.

major comments (3)
  1. [Section 3.2, Eqs. (1)-(2); Tables 1-2] The Motion Signal Translation module is the load-bearing component of the '3D-aware' claim, but it is never validated in isolation or on open-domain data. The only quantitative camera-control results (Table 1) are on RealEstate10K, where monocular depth estimation is likely strongest and scenes are static; Table 2 evaluates full systems on VIPSeg, so translation errors are entangled with generation errors. I ask for a direct evaluation of the translation step: compare depth-warped point tracks against optical-flow/point-track ground truth on a benchmark with known camera poses, compare the projected bbox sequences against tracked boxes on a video segmentation benchmark, and report how the final metrics change when MoGe is replaced by an alternative depth estimator or when the depth scale is perturbed. Without such measurements, the central claim that user scene-space intents are 'accurately interpreted' (Section 1) is not supported outside the RealEstate10K domain.
  2. [Supplement B and Section 3.2] The camera-path specification uses unitless speed values (e.g., 'dolly in by 2'), while the depth map from MoGe has an unstated scale; the ratio between camera translation magnitude and scene depth determines the screen-space displacement. If the depth scale is not normalized or calibrated, the same user input will produce different screen-space trajectories for different images, contradicting the claim of reliable, consistent scene-space control. The paper should state the normalization/calibration procedure for translation magnitudes relative to depth, or provide a sensitivity analysis showing that the final generated motion is robust to this choice.
  3. [Section 5.4, Tables 1-3] The joint camera-and-object control claim is the paper's central contribution, but it is never evaluated quantitatively. Table 1 measures camera control only (RealEstate10K static scenes), Table 2 measures object control only (VIPSeg), and the only joint evaluation is a subjective user study (Table 3). A user study is valuable, but it does not quantify control accuracy. I request a joint quantitative evaluation, for example measuring camera rotation/translation error and object bounding-box or point-tracking error on the same generated clips, or using a small set of designed shots with ground-truth motion, so that the headline claim is supported by an objective metric.
minor comments (6)
  1. [Section 5.5, Table 4] The text says the trajectory-coefficient representation delivers 'robust performance,' but its RotErr (0.6334) is higher than that of the Plucker variant (0.5965); please qualify the conclusion or report a metric that combines rotation and translation accuracy.
  2. [Sections 3.3 and 5.1] The choices K=10, the point-sampling schedule (80% random N~U(0,100), 10% moving, 10% non-moving), and the data-filtering thresholds are not ablated; please add a sensitivity analysis or at least a sentence explaining why these choices are stable.
  3. [Tables 1-3] No confidence intervals or significance tests are reported; given that some differences (e.g., RotErr 0.6334 vs 0.6355 in Table 1) are very small, please report error bars or a paired test.
  4. [Sections 3.3 and 5.1] The base video model and the 1.1M-video dataset are internal; to support reproducibility, please provide the model weights or a more detailed description of the architecture and filtering thresholds.
  5. [Fig. 9 caption] The word 'resulatant' should be 'resultant'.
  6. [Supplement G] The limitations list does not mention the dependency of the translation module on monocular depth accuracy or depth-scale calibration; adding this would help readers calibrate the scope of the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core pipeline is an empirical translation and conditioning system validated against external benchmarks, and its equations are standard projections rather than restatements of the inputs.

full rationale

MotionCanvas's central derivation is the Motion Signal Translation module, which converts user-specified camera paths and scene-space bounding boxes into screen-space point tracks and bbox sequences. This is implemented by explicit projection operations, Eq. (1) 'bl_screen = T^l_camera(bl_scene)' and Eq. (2) 'pl_screen = T^l_camera(T^l_global(pl_scene))'. These are ordinary camera and rigid-motion transformations computed from user inputs and an off-the-shelf monocular depth estimator (MoGe); they are not restatements of the output video or of the conditioning signals used for evaluation. The video generation model is trained on 2D signals (point trajectories from chained optical flow and bbox sequences from panoptic segmentation) and evaluated against external benchmarks: RealEstate10K for camera control (Table 1) and VIPSeg for object-motion control (Table 2), with external baselines MotionCtrl, CameraCtrl, DragAnything, MOFA-Video, and TrackDiffusion. No 'prediction' in the paper is a fitted parameter renamed as a result; the model is fine-tuned once and then tested zero-shot on held-out test sets. The paper does contain self-citations (e.g., DynamiCrafter [52], ToonCrafter [50], ViewCrafter [56], VideoCrafter [7]), but these appear in related-work or application contexts and are not load-bearing for the central derivation. Supplement G honestly discloses a limitation: object local motion is approximated by assuming a frontal parallel depth plane, which 'may not be suitable for extreme close-up or macro shots.' This is a correctness/robustness caveat, not a circularity. The internal DiT base model is not open-sourced, which is a reproducibility concern, but it does not make the argument circular. The derivation therefore stands independently of its conclusions, and no circular step can be exhibited from the paper's own equations, definitions, or citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The core method is an empirical system; the 'free parameters' are training and representation hyperparameters, not constants in a physical derivation. The most load-bearing assumption is the accuracy of monocular depth for the scene-to-screen translation, and the frontal-parallel approximation for local motion is explicitly acknowledged as a limitation.

free parameters (4)
  • DCT coefficient count K = 10
    Number of DCT coefficients used to encode each point trajectory; chosen by the authors as a compact representation (Section 3.3). If too small, fine motion is lost; too large, conditioning is inefficient.
  • Training point sampling schedule = 80% N~U(0,100), 10% moving-object-only, 10% non-moving
    Probability distribution for sampling point trajectories during training (Section 5.1); shapes how the model learns to attend to object vs background points.
  • Data filtering thresholds = tracking length, adjacent-frame IoU, size change ratio, position change, moving-object keyword list
    Hand-chosen thresholds that reduce ~1.1M videos to ~600K training videos (Section 5.1); the quality and coverage of the training set depends on these choices.
  • Depth assignment heuristics for subsequent boxes = reference depth from a scene point or perspective consistency
    When the user draws multiple key bounding boxes, later boxes get depth either from a clicked reference point or from a growing-box heuristic (Section 3.2).
assumptions (4)
  • domain assumption Monocular depth estimator (MoGe) provides sufficiently accurate depth and intrinsics for warping points and bounding boxes.
    Used in Section 3.2 to convert scene-space trajectories to screen space; no quantitative validation of depth accuracy is given.
  • standard math Sparse point tracks on static background regions encode camera motion.
    Borrowed from human perception and SLAM literature (Section 3.2); well-established in computer vision.
  • domain assumption A DiT-based video diffusion model can learn to follow DCT-coded point trajectories and color-coded bbox sequences after fine-tuning.
    The whole approach depends on this empirical ability; shown by results, but not theoretically guaranteed.
  • ad hoc to paper Object local motion lies on a frontal-parallel depth plane relative to the camera.
    Stated in Section 3.2 and Supplement G; used to assign all local-motion points the same depth as their initial positions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation." pith.science (2026). https://pith.science/paper/AIUDRILN

@misc{pith2026250204299,
  author       = {Pith},
  title        = {Pith review of: MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AIUDRILN}},
  note         = {Machine review of arXiv:2502.04299}
}
read the original abstract

This paper presents a method that allows users to design cinematic video shots in the context of image-to-video generation. Shot design, a critical aspect of filmmaking, involves meticulously planning both camera movements and object motions in a scene. However, enabling intuitive shot design in modern image-to-video generation systems presents two main challenges: first, effectively capturing user intentions on the motion design, where both camera movements and scene-space object motions must be specified jointly; and second, representing motion information that can be effectively utilized by a video diffusion model to synthesize the image animations. To address these challenges, we introduce MotionCanvas, a method that integrates user-driven controls into image-to-video (I2V) generation models, allowing users to control both object and camera motions in a scene-aware manner. By connecting insights from classical computer graphics and contemporary video generation techniques, we demonstrate the ability to achieve 3D-aware motion control in I2V synthesis without requiring costly 3D-related training data. MotionCanvas enables users to intuitively depict scene-space motion intentions, and translates them into spatiotemporal motion-conditioning signals for video diffusion models. We demonstrate the effectiveness of our method on a wide range of real-world image content and shot-design scenarios, highlighting its potential to enhance the creative workflows in digital content creation and adapt to various image and video editing applications.

Figures

Figures reproduced from arXiv: 2502.04299 by the authors.

Figure 1
Figure 1. MotionCanvas offers comprehensive motion controls to animate a static image (the “Inputs” column) with various types of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of MotionCanvas. Given an input image and high-level scene-space motion intent, MotionCanvas decomposes and translates the motion (camera and object motion with their timing) into screen space by leveraging the depth-based synthesis and hierar￾chical transformation with the Motion Signal Translation module. These screen-space motion signals are subsequently passed to a video generation model to produce the … view at source ↗
Figure 3
Figure 3. Illustration of our motion-conditioned video generation [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Shot design generated by our MotionCanvas under various types of joint camera and object motion controls. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Long videos with the same complex sequences of camera motion while different object motion controls in each case generated [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Generated videos with diverse and fine-grained local motion controls (upper), and in coordination with camera motion control [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Results when our method is applied for: (upper) motion transfer, and (bottom) video editing for changing objects, adding and [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Camera motion control comparison. Compared to existing baselines, our method performs better at following the intended [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Visual comparison of the resulatant videos from DragAnything, MOFA-Video, TrackDiffusion, Ours [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Visual comparison of joint object and camera motion [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Visual comparison of the results generated by different variants of our method. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: A sample of the designed user interface for our MotionCanvas. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 14
Figure 14. Figure 14: Illustration of the recomputation process for input mo [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Legend of camera motions used in the main paper. [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Designed user study interface. Each participant is required to evaluate 15 video comparisons and respond to three corresponding [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A diffusion model animates a character into arbitrary dynamic backgrounds by conditioning on a rendered 3D-avatar video, reframing open-domain animation as a restoration problem.

  2. Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A retrieval-initialized symbolic regression method discovers equations of motion from video trajectories and uses them to guide image-to-video generation, improving physical alignment on classical mechanics scenes.

Reference graph

Works this paper leans on

62 extracted references · 41 canonical work pages · cited by 2 Pith papers

  1. [1]

    Lindell, and Sergey Tulyakov

    Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Ali- aksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B. Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera control in video diffusion trans- formers. arXiv preprint arXiv:2411.18673, 2024. 3

    arXiv 2024

  2. [2]

    VD3D: Taming large video diffu- sion transformers for 3D camera control

    Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siaro- hin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. VD3D: Taming large video diffu- sion transformers for 3D camera control. arXiv preprint arXiv:2407.12781, 2024. 10

    arXiv 2024

  3. [3]

    Lumiere: A space-time diffu- sion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Her- rmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space-time diffu- sion model for video generation. In ACM SIGGRAPH Asia Conference Proceedings, 2024. 3

    work page 2024

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 3

    arXiv 2023

  5. [5]

    Tim Brooks, Aleksander Holynski, and Alexei A. Efros. In- structPix2Pix: Learning to follow image editing instructions. In CVPR, 2023. 8

    work page 2023

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  7. [7]

    VideoCrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. VideoCrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 3

    arXiv

  8. [8]

    Tracking anything with de- coupled video segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. Tracking anything with de- coupled video segmentation. In ICCV, 2023. 8

    work page 2023

Show all 62 references
  1. [9]

    Local all-pair correspon- dence for point tracking

    Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seun- gryong Kim, and Joon-Young Lee. Local all-pair correspon- dence for point tracking. In ECCV, 2024. 4

    2024

  2. [10]

    Animat- ing pictures with stochastic motion textures

    Yung-Yu Chuang, Dan B Goldman, Ke Colin Zheng, Brian Curless, David H Salesin, and Richard Szeliski. Animat- ing pictures with stochastic motion textures. In ACM SIG- GRAPH Papers, 2005. 3

    2005

  3. [11]

    Epstein and S

    W. Epstein and S. Rogers. Perception of Space and Motion. Academic Press, 1995. 4

    1995

  4. [12]

    Simulating fluids in real-world still images

    Siming Fan, Jingtan Piao, Chen Qian, Hongsheng Li, and Kwan-Yee Lin. Simulating fluids in real-world still images. In ICCV, 2023. 3

    2023

  5. [13]

    Ltx-video: Realtime video latent diffusion

    Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103 ,

    arXiv

  6. [14]

    Multiple View Ge- ometry in Computer Vision

    Richard Hartley and Andrew Zisserman. Multiple View Ge- ometry in Computer Vision . Cambridge University Press, USA, 2 edition, 2003. 4

    2003

  7. [15]

    CameraCtrl: En- abling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. CameraCtrl: En- abling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 2, 3, 9, 10

    2024 arXiv

  8. [16]

    GANs trained by a two time-scale update rule converge to a local Nash equi- librium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equi- librium. In NeurIPS, 2017. 9

    2017

  9. [17]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 9

    2022 arXiv

  10. [18]

    Curless, Steven M

    Aleksander Holynski, Brian L. Curless, Steven M. Seitz, and 11 Richard Szeliski. Animating pictures with Eulerian motion fields. In CVPR, 2021. 2, 3

    2021

  11. [19]

    Tour into the picture: using a spidery mesh interface to make an- imation from a single image

    Youichi Horry, Ken-Ichi Anjyo, and Kiyoshi Arai. Tour into the picture: using a spidery mesh interface to make an- imation from a single image. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, page 225–232, 1997. 2

    1997

  12. [20]

    Fine-grained controllable video generation via object appearance and con- text

    Hsin-Ping Huang, Yu-Chuan Su, Deqing Sun, Lu Jiang, Xuhui Jia, Yukun Zhu, and Ming-Hsuan Yang. Fine-grained controllable video generation via object appearance and con- text. In WACV, 2025. 3

    2025

  13. [21]

    YOLOv11: An overview of the key architectural enhancements

    Rahima Khanam and Muhammad Hussain. YOLOv11: An overview of the key architectural enhancements. arXiv preprint arXiv:2410.17725, 2024. 4

    2024 arXiv

  14. [22]

    TrackDiffusion: Tracklet-conditioned video generation via diffusion models

    Pengxiang Li, Kai Chen, Zhili Liu, Ruiyuan Gao, Lan- qing Hong, Guo Zhou, Hua Yao, Dit-Yan Yeung, Huchuan Lu, and Xu Jia. TrackDiffusion: Tracklet-conditioned video generation via diffusion models. arXiv preprint arXiv:2312.00651, 2023. 3, 10

    2023 arXiv

  15. [23]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023. 5

    2023

  16. [24]

    Infinite nature: Perpetual view generation of natural scenes from a single image

    Andrew Liu, Ameesh Makadia, Richard Tucker, Noah Snavely, Varun Jampani, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In ICCV, 2021. 2

    2021

  17. [25]

    Generative video propagation

    Shaoteng Liu, Tianyu Wang, Jui-Hsien Wang, Qing Liu, Zhifei Zhang, Joon-Young Lee, Yijun Li, Bei Yu, Zhe Lin, Soo Ye Kim, et al. Generative video propagation. arXiv preprint arXiv:2412.19761, 2024. 8

    2024 arXiv

  18. [26]

    InstaFlow: One step is enough for high-quality diffusion- based text-to-image generation

    Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. InstaFlow: One step is enough for high-quality diffusion- based text-to-image generation. In ICLR, 2024. 5

    2024

  19. [27]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 9

    2017 arXiv

  20. [28]

    Trailblazer: Trajectory control for diffusion-based video generation

    Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video generation. In ACM SIGGRAPH Asia Conference Proceed- ings, 2024. 3

    2024

  21. [29]

    Text-guided synthesis of eulerian cinemagraphs

    Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin-Ying Lee, Sergey Tulyakov, and Jun-Yan Zhu. Text-guided synthesis of eulerian cinemagraphs. ACM TOG, 42(6):1–13, 2023. 3

    2023

  22. [30]

    Large-scale video panoptic segmen- tation in the wild: A benchmark

    Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yun- chao Wei, and Yi Yang. Large-scale video panoptic segmen- tation in the wild: A benchmark. In CVPR, 2022. 10

    2022

  23. [31]

    ReVideo: Remake a video with motion and content control

    Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Jian Zhang. ReVideo: Remake a video with motion and content control. In NeurIPS, 2024. 3

    2024

  24. [32]

    3D ken burns effect from a single image

    Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu. 3D ken burns effect from a single image. ACM TOG, 38(6):1– 15, 2019. 2

    2019

  25. [33]

    MOFA-video: Control- lable image animation via generative motion field adaptions in frozen image-to-video diffusion model

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. MOFA-video: Control- lable image animation via generative motion field adaptions in frozen image-to-video diffusion model. In ECCV, 2024. 3, 10, 16

    2024

  26. [34]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 2, 5

    2023

  27. [35]

    Movie Gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie Gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  28. [36]

    Freetraj: Tuning-free tra- jectory control in video diffusion models

    Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free tra- jectory control in video diffusion models. arXiv preprint arXiv:2406.16863, 2024. 3

    2024 arXiv

  29. [37]

    SAM 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 4

    2024 arXiv

  30. [38]

    Vi- sual slam algorithms: A survey from 2010 to 2016

    Takafumi Taketomi, Hideaki Uchiyama, and Sei Ikeda. Vi- sual slam algorithms: A survey from 2010 to 2016. IPSJ trans. on computer vision and app., 9:1–11, 2017. 4

    2010

  31. [39]

    Generative AI for cel-animation: A survey

    Yunlong Tang, Junjia Guo, Pinxin Liu, Zhiyuan Wang, Hang Hua, Jia-Xing Zhong, Yunzhong Xiao, Chao Huang, Luchuan Song, Susan Liang, Yizhi Song, Liu He, Jing Bi, Mingqian Feng, Xinyang Li, Zeliang Zhang, and Chenliang Xu. Generative AI for cel-animation: A survey. arXiv preprin...

    2025

  32. [40]

    RAFT: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020. 8

    2020

  33. [41]

    William Thompson, Roland Fleming, Sarah Creem-Regehr, and Jeanine Kelly Stefanucci.Visual Perception from a Com- puter Graphics Perspective . A. K. Peters, Ltd., USA, 1st edition, 2011. 4

    2011

  34. [42]

    FVD: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. FVD: A new metric for video generation. In ICLR work- shop, 2019. 9

    2019

  35. [43]

    Boximator: Gener- ating rich and controllable motions for video synthesis

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guo- qiang Wei, Liping Yuan, and Hang Li. Boximator: Gener- ating rich and controllable motions for video synthesis. In ICML, 2024. 3, 10

    2024

  36. [44]

    MoGe: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision

    Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. MoGe: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. arXiv preprint arXiv:2410.19115, 2024. 4

    arXiv

  37. [45]

    Akira: Augmentation kit on rays for optical video generation

    Xi Wang, Robin Courant, Marc Christie, and Vicky Kalo- geiton. Akira: Augmentation kit on rays for optical video generation. arXiv preprint arXiv:2412.14158, 2024. 3

    2024 arXiv

  38. [46]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH Conference Proceed- ings, 2024. 2, 3, 9

    2024

  39. [47]

    Synsin: End-to-end view synthesis from a single image

    Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In CVPR, 2020. 2

    2020

  40. [48]

    Mo- tionbooth: Motion-aware customized text-to-video genera- tion

    Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Mo- tionbooth: Motion-aware customized text-to-video genera- tion. In NeurIPS, 2024. 3 12

    2024

  41. [49]

    DragAnything: Motion control for any- thing using entity representation

    Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. DragAnything: Motion control for any- thing using entity representation. In ECCV, 2024. 2, 3, 10, 16

    2024

  42. [50]

    ToonCrafter: Generative cartoon interpolation

    Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xin- tao Wang, Ying Shan, and Tien-Tsin Wong. ToonCrafter: Generative cartoon interpolation. ACM TOG, 43(6):1–11,

  43. [51]

    Make-your-video: Customized video generation using textual and structural guidance.IEEE TVCG, 31(2):1526–1541, 2024

    Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, et al. Make-your-video: Customized video generation using textual and structural guidance.IEEE TVCG, 31(2):1526–1541, 2024. 3

    2024

  44. [52]

    DynamiCrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. DynamiCrafter: Animating open-domain images with video diffusion priors. In ECCV,

  45. [53]

    CamCo: Camera- controllable 3D-consistent image-to-video generation

    Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. CamCo: Camera- controllable 3D-consistent image-to-video generation. arXiv preprint arXiv:2406.02509, 2024. 10

    2024 arXiv

  46. [54]

    Animating animal motion from still

    Xuemiao Xu, Liang Wan, Xiaopei Liu, Tien-Tsin Wong, Liansheng Wang, and Chi-Sing Leung. Animating animal motion from still. In SIGGRAPH Asia, 2008. 2

    2008

  47. [55]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 3, 5

    2024 arXiv

  48. [56]

    Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024

    Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024. 3

    2024 arXiv

  49. [57]

    Open-Sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all, 2024. 5

    2024

  50. [58]

    Stereo magnification: Learning view syn- thesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view syn- thesis using multiplane images. ACM TOG, 37(4), 2018. 4, 9 13 MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation Supplementary Materi...

    2018

  51. [59]

    Motion Design Module – Capturing User In- tents

    MotionCanvas 3 3.1. Motion Design Module – Capturing User In- tents . . . . . . . . . . . . . . . . . . . . . 3 3.2. Motion Signal Translation Module . . . . . . 4 3.3. Motion-conditioned Video Generation . . . . 5 3.4. Generating Variable-length Videos via Auto- regression . ...

  52. [60]

    Shot Design with Joint Camera and Object Control

    Applications 6 4.1. Shot Design with Joint Camera and Object Control . . . . . . . . . . . . . . . . . . . 7 4.2. Object Local Motion Control . . . . . . . . . 7 4.3. Additional Applications: Simple Video- based Editing . . . . . . . . . . . . . . . . 7

  53. [61]

    Implementation Details

    Experiments 8 5.1. Implementation Details . . . . . . . . . . . . 8 5.2. Camera Motion Control Quality . . . . . . . 9 5.3. 3D-Aware Object Motion Control Quality . . 10 5.4. Joint Camera and Object Control . . . . . . 10 5.5. Ablation Studies . . . . . . . . . . . . . . . 10

  54. [62]

    Camera Motion Control

    Conclusion 11 A . Additional Details 14 B . User Interface 14 C . Essentiality of Camera-aware and Camera- object-aware Transformations 15 D . More Details of MotionCanvasAR 16 E . User Study 16 F . Additional Analysis 17 F.1. Effect of Point Track Density on Camera Motion Con...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.