Pith. sign in

REVIEW 4 major objections 3 minor 5 cited by

CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CineMaster turns 3D box layouts into camera-controlled video.

desk verdict A promising 3D-box-and-camera control framework for text-to-video generation, with an ambitious but unvalidated auto-labeling pipeline and thin quantitative evidence; worth refereeing, not ready as is. read the letter →

arxiv 2502.08639 v1 pith:WIHGLFLG submitted 2025-02-12 cs.CV

Qinghe Wang , Yawen Luo , Xiaoyu Shi , Xu Jia , Huchuan Lu , Tianfan Xue , Xintao Wang , Pengfei Wan
show 2 more authors
Di Zhang Kun Gai
This is my paper · ORCID
classification cs.CV
keywords text-to-videogeneration3D-awarevideocameramotioncontrolobjectdiffusiontransformer3Dboundingboxesdepthconditioningdatasetannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CineMaster claims that a text-to-video diffusion model can be steered like a film shoot by conditioning it on 3D bounding boxes, camera trajectories, per-frame projected depth maps, and object class labels. Users first stage objects and camera in a 3D scene editor, then the model renders the planned video from the exported control signals. Because real videos rarely carry 3D box and camera annotations, the paper adds an automated labeling pipeline that extracts those signals from in-the-wild footage, and it trains the layout controller and camera controller jointly so object motion and camera motion do not conflate. In direct comparisons, CineMaster reports better object-box alignment, lower trajectory deviation, and better video-quality and text-alignment scores than prior joint controllers. If the claim holds, this is a practical route from 3D shot planning to generated video for users who want director-level placement control.

What carries the argument

The load-bearing machinery is a Semantic Layout ControlNet paired with a Camera Adapter, built on a pretrained text-to-video transformer. The ControlNet copies half the transformer blocks; a semantic injector takes per-entity class-label embeddings and splices them into the hidden states at the positions given by downsampled entity masks, then concatenates the fused semantic map with depth latents and projects it with MLPs. That gives every subject an explicit 3D location. The Camera Adapter adds a sequence of 6-DoF camera poses (rotation plus translation) as residual features between the self-attention and temporal-attention layers of each block, so the model can tell apart 'the balloon rose' from 'the camera dropped.' The rendered depth maps serve as the common visual ground that ties the 3D scene layout to each output frame. Around this sits a data-labeling pipeline that turns raw video into the training conditions, plus a three-stage training schedule that first learns dense depth control, then adapts to 3D boxes, then jointly trains both controllers.

What would settle it

Take synthetic or LiDAR-captured videos with known 3D object boxes and camera poses, run the labeling pipeline on them, and compare the extracted boxes and camera paths to the ground truth; if the errors are comparable to or larger than the motions the user wants to control, or if retraining with perfect labels fails to change the outputs, the central claim is disproved.

Watch

Extended reading notes

Core claim

The paper's central claim is that joint 3D object and camera control in text-to-video generation is achievable by combining three conditions: rendered depth maps that encode the desired 3D layout, camera trajectories that disambiguate camera motion from object motion, and per-entity class labels that bind each prompt subject to its 3D box. The authors show this through a two-stage design in which an interactive 3D workflow produces the conditions, and a fine-tuned diffusion transformer consumes them. They further claim that an automated pipeline—instance segmentation, metric depth, monocular camera pose, and 3D point tracking—can generate reliable 3D box and trajectory annotations at scale, yielding the largest training set of its kind and enabling joint training that outperforms existing 2D-trajectory and box-based controllers on every reported metric.

Load-bearing premise

The load-bearing premise is that the pipeline's estimated depths, camera positions, object masks, and 3D point tracks are accurate enough to teach the model what user-drawn 3D boxes should look like in video, and the paper does not verify these estimates against true 3D measurements.

Editorial extensions

If this is right

  • Users can specify object placement and camera motion in 3D, and the generated video is claimed to follow both simultaneously without the object-camera coupling seen in prior controllers.
  • The automated labeling pipeline removes the need for synthetic or manually annotated 3D video datasets, so training data can be scaled up from in-the-wild footage.
  • Joint training of the layout controller and camera adapter is required: training them separately or freezing one while training the other degrades alignment, quality, and depth accuracy.
  • Because depth maps carry the layout, the method can use rendered depth from any 3D scene editor, not just footage-derived conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method deliberately leaves out object orientation: rotating a 3D box to turn a person around is named as future work, so current control is translation-and-placement level rather than full 6-DoF object control.
  • Reported metrics measure alignment using automated detectors and monocular depth, so the true ceiling may be coupled to the same estimators that build the training labels; a human or synthetic-ground-truth evaluation would give a cleaner read.
  • If the monocular estimates in the labeling pipeline drift, the error is likely correlated across depth, pose, and tracking, which could bias the learned controller in ways that per-video examples do not reveal.
  • The same condition set could be transferred to image-to-video models or to user-provided 3D keyframe animations, since the control signals are renderer-agnostic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This paper proposes CineMaster, a two-stage framework for 3D-aware controllable text-to-video generation. In the first stage, users construct control signals by placing 3D bounding boxes with class labels and defining camera trajectories in Blender; the system renders per-frame depth maps, camera poses, and class-label maps. In the second stage, a latent diffusion T2V model is fine-tuned with a Semantic Layout ControlNet (fusing depth latents with class-label embeddings placed via a scatter mask) and a Camera Adapter that injects 6-DoF camera poses into each DiT block. To obtain training data, the authors build an automated pipeline that extracts instance masks (Grounding DINO + SAM 2), metric depth (DepthAnything V2), camera poses (MonST3R), and per-object 3D boxes tracked by SpatialTracker displacements, producing 156K videos with 3D box and camera annotations. Experiments compare CineMaster to MotionCtrl and Direct-A-Video on mIoU, Traj-D, FVD, FID, CLIP-T, and an ablation study over training paradigms.

Significance. If the claims hold, CineMaster provides a practical 3D-native control interface for T2V generation and a scalable way to obtain 3D annotations from in-the-wild video. The paper's strengths include a clear system architecture, a substantial data pipeline, and qualitative demonstrations of joint object/camera control. The authors also make a good-faith attempt to evaluate with multiple metrics and baselines. However, the central quantitative claims rest on unvalidated labels and single-run numbers, so the paper's main contribution is currently not fully supported. The ideas are sufficiently novel and promising that the concerns can be addressed with additional validation and experiments.

major comments (4)
  1. [3.3] The automatic labeling pipeline is the central source of training supervision, but it is never validated against ground-truth 3D data. The pipeline (Fig. 4) composites DepthAnything V2 metric depth, MonST3R camera poses, and SpatialTracker 3D displacements without a stated coordinate or scale alignment. These estimators are known to have different scale conventions; a box computed via inverse projection in the depth frame and moved with tracker displacements may not be consistent with the MonST3R camera frame. If the errors are systematic, the learned Semantic Layout ControlNet and Camera Adapter inherit the bias. In Section 3.3 and the Introduction, the outputs are called 'ground-truth 3D bounding box and 3D camera trajectory annotations,' which overstates an unvalidated estimator. Please compare the extracted boxes and trajectories against ground-truth annotations (e.g., synthetic sequences or scanned scenes) and report errors.
  2. [4.1, Evaluation Metrics] Depth-D is computed with SAM 2 and DepthAnything V2 on generated frames, and the reference depths are derived from the given 3D boxes. Since DepthAnything V2 is also the model used to generate the metric depth labels for training (Sec 3.3 and Sec 4.1), Depth-D effectively measures how well the model reproduces DepthAnything V2's output rather than true geometric accuracy. This circularity weakens the claim of improved depth control. Please either replace Depth-D with an independent depth source or explicitly acknowledge this limitation and supplement with a non-circular validation (e.g., on synthetic scenes with known depth).
  3. [Tables 1 and 2] The quantitative results are reported as single-run point estimates with no standard deviations, confidence intervals, or significance tests. The differences between CineMaster and Direct-A-Video on FID (175.9 vs 183.5) and CLIP-T (0.321 vs 0.273) are considerable, but without variance we cannot assess reliability. Additionally, only two baselines are used, and their conditions are converted from the 3D box condition (object trajectories for MotionCtrl, 2D boxes for Direct-A-Video), which may put them at a disadvantage. The abstract's claim of 'significantly outperforms' is not established. Please report statistics across multiple seeds, include more recent 3D-aware baselines (e.g., 3DTrajMaster, SynFMC, MotionCanvas), and use a consistent condition-conversion protocol with an analysis of its effect.
  4. [4.3, Table 2] The ablation study varies training paradigms but does not include a 'without Camera Adapter' condition. The paper's contribution is joint object and camera control, and Section 3.2 argues that the camera adapter resolves ambiguity between object and camera motion. Without an ablation that removes or freezes the camera adapter, the benefit of joint training cannot be separated from the benefit of the Semantic Layout ControlNet alone. Please add such an ablation (and optionally a 'without depth maps' ablation) to support the architectural claims.
minor comments (3)
  1. [Figure 2] The label '3D Depth Box' in the figure is unclear; consider simplifying it to '3D Box' or 'Projected Depth Maps' to match the text.
  2. [4.1] The sentence 'We make a trade-off between object motion and camera motion by injecting semantic layout information and camera poses with 25 and 15 steps respectively' is vague; please specify whether these are training steps, inference steps, or guidance scales for the two adapters.
  3. [1 and 3.3] The claim of building 'the largest video datasets with both the ground-truth 3D bounding box and 3D camera trajectory annotations' is difficult to verify without a dataset release or comparison with existing annotated datasets (e.g., SynFMC); please clarify that the annotations are automatically estimated and not human-verified.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild metric circularity in Depth-D; otherwise the empirical derivation is self-contained.

  1. other [Sec. 4.1 Evaluation Metrics and Sec. 3.3 Dataset Labeling Pipeline]
    "we calculate the average depth of the object regions in each generated frame using SAM 2 [29] and DepthAnything V2 [44], and measure the depth deviation (Depth-D) by the Root Mean Squared Error (RMSE) with the depth values of the given 3D boxes. ... We employ DepthAnything V2 [44] to generate metric depth maps for the entire video sequence, which are essential for the subsequent inverse projection process."

    The same DepthAnything V2 estimator supplies both the training-time depth labels (via inverse projection into 3D boxes) and the inference-time Depth-D evaluation. The model is trained to make rendered depth maps consistent with DepthAnything V2 geometry, so the depth-control metric largely rewards reproducing this one estimator's scale and biases rather than independently verified 3D placement. This is a partial self-referentiality of the depth metric, not a collapse of the whole system: the other quantitative metrics and qualitative comparisons remain independent evidence for the central claim.

full rationale

CineMaster is an empirical, pipeline-based system rather than a formal derivation, so most circularity patterns (self-definition, imported uniqueness theorems, ansatz-smuggling via self-citation) do not apply. The central claim—joint 3D box and camera control for T2V—is supported by a trained diffusion model and compared against external baselines; no fitted parameter is renamed as a prediction, and no load-bearing argument reduces to a self-citation. The only noticeable circular element is the Depth-D evaluation: both the training labels and this metric use DepthAnything V2, so the depth-accuracy number is partly a measure of how well the model imitates that particular depth estimator. The same observation applies in a weaker form to mIoU, which uses Grounding DINO for evaluation while that detector also participates in label generation; however, the relative comparison across baselines is still informative. The annotation pipeline outputs are called 'ground-truth 3D bounding box and camera trajectory annotations' without independent validation, but that is a validity/correctness risk, not circularity. We therefore assign a low score reflecting the self-referential depth metric while judging the overall framework's derivation chain to be essentially independent.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The system relies mainly on standard diffusion training plus a set of external pretrained models and annotation heuristics. The heaviest assumptions are that metric depth and camera pose estimates are reliable, and that a constant-volume 3D box tracked by average point displacements is a valid representation of object motion. No new conceptual entities are postulated.

free parameters (4)
  • Training step counts per stage = 12000, 7000, 6000
    Hand-chosen schedule for the three training stages; affects how well the depth ControlNet, semantic injector, and camera adapter settle, but no criterion for choosing these numbers is given.
  • Classifier-free guidance scale = 12.5
    Set during inference; high guidance is known to trade sample diversity for prompt adherence, and the value is not justified beyond being chosen empirically.
  • Inference step budget split = 25 steps semantic layout, 15 steps camera, 50 DDIM total
    Used to balance object and camera control; the paper states only that this trade-off exists, not how the split was selected.
  • Data mixing ratio = 3:1 (own dataset to RealEstate10K)
    Controls the balance between in-the-wild diversity and larger camera motion during joint training; no sensitivity analysis is reported.
assumptions (5)
  • standard math Rectified flow and conditional flow matching define the training objective (Eqs. 1-3).
    Standard diffusion and flow framework used as background; not in question.
  • domain assumption The internal pretrained T2V diffusion model can be finetuned with additional control modules and retains its generation quality.
    The base model is not public; all results depend on this proprietary model and the assumption that ControlNet-style injection does not degrade it.
  • ad hoc to paper Each entity has constant volume in 3D and its 3D bounding box can be tracked by averaged inter-frame displacements of feature points.
    Sec 3.3 assumes constant volume and uses average SpatialTracker displacements to move boxes; object rotation, non-rigidity, and occlusion are ignored.
  • domain assumption DepthAnything V2 produces metric depth accurate enough for inverse projection into 3D boxes.
    The whole annotation pipeline rests on metric depth from this model, which the paper does not validate for its target domain.
  • domain assumption MonST3R camera poses are accurate enough to supervise camera control.
    Camera trajectories are taken directly from MonST3R with no error analysis; errors would propagate to the camera adapter training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation." pith.science (2026). https://pith.science/paper/WIHGLFLG

@misc{pith2026250208639,
  author       = {Pith},
  title        = {Pith review of: CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WIHGLFLG}},
  note         = {Machine review of arXiv:2502.08639}
}
read the original abstract

In this work, we present CineMaster, a novel framework for 3D-aware and controllable text-to-video generation. Our goal is to empower users with comparable controllability as professional film directors: precise placement of objects within the scene, flexible manipulation of both objects and camera in 3D space, and intuitive layout control over the rendered frames. To achieve this, CineMaster operates in two stages. In the first stage, we design an interactive workflow that allows users to intuitively construct 3D-aware conditional signals by positioning object bounding boxes and defining camera movements within the 3D space. In the second stage, these control signals--comprising rendered depth maps, camera trajectories and object class labels--serve as the guidance for a text-to-video diffusion model, ensuring to generate the user-intended video content. Furthermore, to overcome the scarcity of in-the-wild datasets with 3D object motion and camera pose annotations, we carefully establish an automated data annotation pipeline that extracts 3D bounding boxes and camera trajectories from large-scale video data. Extensive qualitative and quantitative experiments demonstrate that CineMaster significantly outperforms existing methods and implements prominent 3D-aware text-to-video generation. Project page: https://cinemaster-dev.github.io/.

Figures

Figures reproduced from arXiv: 2502.08639 by the authors.

Figure 1
Figure 1. CineMaster targets at granting users 3D-aware and intuitive control over the text-to-video generation process. We first design [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of CineMaster. CineMaster consists of two [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the network architecture. We design a Semantic Layout ControlNet which consists of a semantic injector and a DiT [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Dataset Labeling Pipeline. We propose a data labeling pipeline to extract 3D bounding boxes, class labels and camera poses from [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: We present three different feature comparisons: moving object [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ToonComposer: Streamlining Cartoon Production with Generative Post-Keyframing

    cs.CV 2025-08 conditional novelty 6.0 of 10

    ToonComposer generates cartoon videos from a colored reference frame and sparse keyframe sketches, merging inbetweening and colorization in one diffusion model.

  2. UNIC: Unified In-Context Video Editing

    cs.CV 2025-06 conditional novelty 6.0 of 10

    One diffusion transformer handles ID insert, swap, delete, stylization, propagation, and re-camera control in a single model using in-context token concatenation with task-aware positional encoding and bias.

  3. T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generation

    cs.CV 2025-07 reject novelty 4.0 of 10

    A 1,200-prompt benchmark across six world-knowledge domains reports that ten state-of-the-art text-to-video models average below 0.70 on a 0 to 1 scale for producing videos consistent with real-world knowledge.

  4. CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step

    cs.CV 2025-07 conditional novelty 4.0 of 10

    CoT-Diff couples a multimodal LLM's step-by-step 3D layout reasoning into the diffusion denoising loop, claiming large gains in spatial alignment for text-to-image generation.

  5. Fuel Consumption in Platoons: A Literature Review

    eess.SY 2025-08 unverdicted

    A literature review compiling factors that affect fuel consumption in vehicle platoons, including drag reduction, coordination, and instability.

Reference graph

Works this paper leans on

50 extracted references · 21 canonical work pages · cited by 5 Pith papers

  1. [1]

    Loosecontrol: Lifting controlnet for generalized depth conditioning

    Shariq Farooq Bhat, Niloy Mitra, and Peter Wonka. Loosecontrol: Lifting controlnet for generalized depth conditioning. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 2, 4, 6

    work page 2024

  2. [2]

    Gs-dit: Ad- vancing video generation with pseudo 4d gaussian fields through efficient dense 3d point tracking

    Weikang Bian, Zhaoyang Huang, Xiaoyu Shi, Yijin Li, Fu-Yun Wang, and Hongsheng Li. Gs-dit: Ad- vancing video generation with pseudo 4d gaussian fields through efficient dense 3d point tracking. arXiv preprint arXiv:2501.02690, 2025. 2

    arXiv 2025

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    arXiv 2023

  4. [4]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512 ,

    arXiv

  5. [5]

    Pixart- alpha: Fast training of diffusion transformer for pho- torealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- alpha: Fast training of diffusion transformer for pho- torealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 4

    arXiv

  6. [6]

    Control-a-video: Controllable text-to-video generation with diffusion models

    Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840, 2023. 2

    arXiv 2023

  7. [7]

    Perception-as-control: Fine-grained controllable image animation with 3d-aware motion representation

    Yingjie Chen, Yifang Men, Yuan Yao, Miaomiao Cui, and Liefeng Bo. Perception-as-control: Fine-grained controllable image animation with 3d-aware motion representation. arXiv preprint arXiv:2501.05020 ,

    arXiv

  8. [8]

    Patch n’pack: Navit, a vision transformer for any aspect ratio and resolu- tion

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolu- tion. Advances in Neural Information Processing Sys- tems, 36, 2024. 6

    work page 2024

Show all 50 references
  1. [9]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Confer- ence on Machi...

    2024

  2. [10]

    3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation

    Xiao Fu, Xian Liu, Xintao Wang, Sida Peng, Meng- han Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di Zhang, and Dahua Lin. 3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation. arXiv preprint arXiv:2412.07759, 2024. 2, 3

    2024 arXiv

  3. [11]

    Motion prompting: Controlling video generation with motion trajectories

    Daniel Geng, Charles Herrmann, Junhwa Hur, For- rester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. Motion prompting: Controlling video generation with motion trajectories. arXiv preprint arXiv:2412.02700, 2024. 3

    2024 arXiv

  4. [12]

    Diffusion as shader: 3d-aware video diffusion for versatile video generation control

    Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Zi- wei Liu, et al. Diffusion as shader: 3d-aware video diffusion for versatile video generation control. arXiv preprint arXiv:2501.03847, 2025. 2, 3

    2025 arXiv

  5. [13]

    Sparsectrl: Adding sparse controls to text-to-video diffusion mod- els, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion mod- els, 2023. 2

    2023

  6. [14]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 ,

    arXiv

  7. [15]

    Cameractrl: Enabling camera control for text-to-video generation,

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation,

  8. [16]

    Classifier-free dif- fusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free dif- fusion guidance. arXiv preprint arXiv:2207.12598 ,

    arXiv

  9. [17]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural in- formation processing systems, 33:6840–6851, 2020. 2

    2020

  10. [18]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 8153– 8163, 2024. 2

    2024

  11. [19]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 4

    2013 arXiv

  12. [20]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 6

    2014 arXiv

  13. [21]

    Onlyflow: Optical flow based motion condition- ing for video diffusion models

    Mathis Koroglu, Hugo Caselles-Dupr ´e, Guil- laume Jeanneret Sanmiguel, and Matthieu Cord. Onlyflow: Optical flow based motion condition- ing for video diffusion models. arXiv preprint arXiv:2411.10501, 2024. 2

    2024

  14. [22]

    Microsoft coco: Com- mon objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proce...

    2014

  15. [23]

    Flow match- ing for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow match- ing for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 4

    2022 arXiv

  16. [24]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 5, 7

    2023 arXiv

  17. [25]

    Are gans cre- ated equal? a large-scale study

    Mario Lucic, Karol Kurach, Marcin Michalski, Syl- vain Gelly, and Olivier Bousquet. Are gans cre- ated equal? a large-scale study. arXiv preprint arXiv:1711.10337, 2017. 7

    2017 arXiv

  18. [26]

    T2i- adapter: Learning adapters to dig out more control- lable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more control- lable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial In- telligence, pages 4296–430...

    2024

  19. [27]

    Scalable diffu- sion models with transformers

    William Peebles and Saining Xie. Scalable diffu- sion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 4195–4205, 2023. 4

    2023

  20. [28]

    Exploring the limits of trans- fer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of trans- fer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67,

  21. [29]

    Sam 2: Segment anything in images and videos, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R ¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Gir- shick, Piotr Doll´ar, and Christoph Feic...

    2024

  22. [30]

    High- resolution image synthesis with latent diffusion mod- els

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High- resolution image synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684–10695, 2022. 2

    2022

  23. [31]

    Ob- jects365: A large-scale, high-quality dataset for ob- ject detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Ob- jects365: A large-scale, high-quality dataset for ob- ject detection. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 8430– 8439, 2019. 6

    2019

  24. [32]

    Motion-i2v: Consistent and controllable image-to- video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to- video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, p...

    2024

  25. [33]

    Free-form mo- tion control: A synthetic video generation dataset with controllable camera and object motions

    Xincheng Shuai, Henghui Ding, Zhenyuan Qin, Hao Luo, Xingjun Ma, and Dacheng Tao. Free-form mo- tion control: A synthetic video generation dataset with controllable camera and object motions. arXiv preprint arXiv:2501.01425, 2025. 2, 3

    2025 arXiv

  26. [35]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 6

    2010 arXiv

  27. [36]

    Towards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 7

    2018 arXiv

  28. [37]

    Easycontrol: Transfer controlnet to video dif- fusion for controllable generation and interpolation

    Cong Wang, Jiaxi Gu, Panwen Hu, Haoyu Zhao, Yuanfan Guo, Jianhua Han, Hang Xu, and Xiaodan Liang. Easycontrol: Transfer controlnet to video dif- fusion for controllable generation and interpolation. arXiv preprint arXiv:2408.13005, 2024. 2

    2024 arXiv

  29. [38]

    Boxi- mator: Generating rich and controllable motions for video synthesis, 2024

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boxi- mator: Generating rich and controllable motions for video synthesis, 2024. 3

    2024

  30. [39]

    VideoComposer: Compo- sitional Video Synthesis with Motion Controllability,

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. VideoComposer: Compo- sitional Video Synthesis with Motion Controllability,

  31. [40]

    Mo- tionCtrl: A Unified and Flexible Motion Controller for Video Generation, 2024

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Mo- tionCtrl: A Unified and Flexible Motion Controller for Video Generation, 2024. arXiv:2312.03641 [cs]. 2, 3, 6, 7

    2024 arXiv

  32. [41]

    Spa- tialtracker: Tracking any 2d pixels in 3d space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spa- tialtracker: Tracking any 2d pixels in 3d space. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3, 5

    2024

  33. [42]

    Motioncanvas: Cinematic shot design with controllable image-to-video generation,

    Jinbo Xing, Long Mai, Cusuh Ham, Jiahui Huang, Aniruddha Mahapatra, Chi-Wing Fu, Tien-Tsin Wong, and Feng Liu. Motioncanvas: Cinematic shot design with controllable image-to-video generation,

  34. [43]

    Qwen2 technical re- port

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao- ran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhe...

    2024 arXiv

  35. [44]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.09414, 2024. 5, 6, 7

    2024 arXiv

  36. [45]

    Direct-a-video: Customized video generation with user-directed camera movement and object mo- tion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object mo- tion. In ACM SIGGRAPH 2024 Conference Papers , pages 1–12, 2024. 2, 3, 6, 7

    2024

  37. [46]

    Dragnuwa: Fine-grained control in video generation by integrat- ing text, image, and trajectory, 2023

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrat- ing text, image, and trajectory, 2023. 2

    2023

  38. [47]

    Monst3r: A simple approach for estimating geometry in the presence of motion

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arxiv:2410.03825, 2024. 5

    arXiv

  39. [48]

    Adding conditional control to text-to-image diffusion models, 2023

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 2

    2023

  40. [49]

    Tora: Trajectory-oriented Diffusion Transformer for Video Generation, 2024

    Zhenghao Zhang, Junchao Liao, Menghao Li, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented Diffusion Transformer for Video Generation, 2024. arXiv:2407.21705 [cs]. 2

    2024 arXiv

  41. [50]

    Stereo magnifica- tion: Learning view synthesis using multiplane im- ages, 2018

    Tinghui Zhou, Richard Tucker, John Flynn, Gra- ham Fyffe, and Noah Snavely. Stereo magnifica- tion: Learning view synthesis using multiplane im- ages, 2018. 3, 6

    2018

  42. [2023]

    arXiv:2306.02018 [cs]. 2, 3

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.