Pith. sign in

REVIEW 5 major objections 5 minor 4 cited by

CPA: Camera-pose-awareness Diffusion Transformer for Video Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a plug-in camera-pose module can give a diffusion-transformer video generator precise, long-horizon camera control without retraining the backbone.

desk verdict A sensible plug-in for camera-aware DiT video generation, but the SOTA claim rides on a 20-video, resolution-mismatched comparison that doesn't support it. read the letter →

arxiv 2412.01429 v1 pith:4A6Y5LOT submitted 2024-12-02 cs.CV

Yuelei Wang , Jian Zhang , Pengtao Jiang , Hao Zhang , Jinwei Chen , Bo Li This is my paper · ORCID
classification cs.CV
keywords cameraposecontrolvideogenerationdiffusiontransformertext-to-videoPlückercoordinatestemporalattentioninjectionsparsemotionencodinglong
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents CPA, a camera-pose-awareness method for diffusion-transformer (DiT) text-to-video generation. It claims that camera trajectories can be converted into a sparse motion field, compressed into a compact pose latent, and injected directly into the temporal attention layers of an existing DiT model, so the model follows the requested camera motion over long clips while keeping its original text-to-video quality. On 72-frame clips the method reports a camera-motion error of 0.042, lower than the previous best 0.044 among the compared methods, together with better video-consistency scores. If the result holds, camera pose becomes a practical first-class condition for long video generation, useful for pre-visualization, virtual cinematography, and controllable world-model applications.

What carries the argument

Two plug-in modules carry the argument. The Sparse Motion Encoding (SME) module samples a camera-derived Plücker-coordinate motion field at regular grid intervals in the image, building a sparse field $F_s \in \mathbb{R}^{L\times M\times N}$ that is then encoded by a VAE into a pose latent $z_p \in \mathbb{R}^{l\times m\times n\times 4}$. The Temporal Attention Injection (TAI) module aligns this pose latent with the temporal attention features of each spatial-temporal DiT block by normalizing both, concatenating them, and applying a learned shift-and-scale through an MLP; the authors argue this unifies the pose and temporal latents into a similar distribution, which is what makes the injection effective.

What would settle it

Run the same 72-frame benchmark at one matched resolution for all methods (for instance everything at 640x360) with a larger test set of 100 or more clips and uncertainty estimates on CamMC and FVD; if the camera-motion error no longer beats the leading compared DiT baseline, or if the latent-diffusion baselines improve substantially with matched resolution, the state-of-the-art claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that camera-pose control for DiT-based video generators can be achieved by encoding the inter-frame camera extrinsics as a sparse pixel-wise motion field based on Plücker coordinates, where each sampled pixel's back-projected ray is moved by the rotation and translation between frames, then training a small VAE to compress that field into a pose latent, and injecting the latent into temporal attention through layer normalization and an MLP. The author argues that this preserves the pretrained DiT's capabilities because only the temporal attention layers are fine-tuned, with LoRA used in self-attention, while all other parameters stay frozen. On the long-video (72-frame) evaluation, CPA reports CamMC, a camera-motion error metric, of 0.042 and FVD 1387, improving on the 0.044 CamMC and 1400 FVD of the leading compared DiT camera-control baseline, while also scoring highest on CLIPSIM. The claim is therefore that CPA achieves state-of-the-art camera-motion consistency for long video generation.

Load-bearing premise

The long-video state-of-the-art claim assumes that the metrics are comparable across the different resolutions used for different methods (256x256 or 384x256 for the latent-diffusion baselines, 640x360 for the transformer-based ones) and that the 20-video test set, reported without error bars, is representative.

Editorial extensions

If this is right

  • If the claims hold, camera-pose conditioning can be layered onto existing DiT video generators as a plug-in, without retraining the full generative model.
  • Per-frame pose injection makes long clips follow a prescribed trajectory in one pass, avoiding the drift that comes from chaining short segments.
  • A sampling density around 40x, about 16 by 9 control points at 640 by 360, appears to be a practical sweet spot; the paper reports that both denser and sparser sampling degrade consistency.
  • Because the backbone is mostly frozen, adapting the plug-in to a new DiT base may be inexpensive: the reported fine-tune takes about 100k steps on four GPUs.
  • The paper's own limitation discussion notes the method is conservative about object motion, so large object movement remains an open problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We suspect the same SME-plus-TAI recipe could transfer to other DiT video backbones, since it does not modify the backbone's weights and only touches temporal attention, so a head-to-head test on a second base model would show whether the plug-in generalizes.
  • The sparse Plücker field could in principle also encode object trajectories as per-point displacements, which would let object motion and camera motion share one conditioning space; the paper does not claim this, but its representation invites the extension.
  • The reported numbers leave a resolution confound: the LDM baselines are evaluated at 256x256 or 384x256 while the DiT methods run at 640x360, so a matched-resolution evaluation would be a sharper test of the state-of-the-art claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CPA, a plug-in for DiT-based text-to-video generation (built on OpenSora-v1.2) that adds camera-pose control. It introduces a Sparse Motion Encoding (SME) module that converts camera rotation/translation sequences into a sparse Plücker-coordinate motion field, a VAE that compresses this field into a pose latent, and a Temporal Attention Injection (TAI) module that fuses the pose latent into the temporal attention layers of the ST-DiT blocks. The backbone is kept largely frozen; only temporal-attention and LoRA parameters are updated. Experiments on 20 RealEstate10K test videos compare CPA with LDM- and DiT-based baselines for both 16-frame and 72-frame generation, reporting the best long-video CamMC, FVD, and CLIPSIM in Table 1. The paper's own discussion (Sec. 4.6) acknowledges that object consistency is relatively weak and that pose support is currently limited to 16 frames.

Significance. The core idea is timely and potentially useful: a camera-pose conditioner that works with DiT video generators, preserves the base model's parameters, and uses a compact VAE-compressed pose latent. If the evaluation were properly controlled, the method could be a meaningful step toward practical camera control in long-form video generation. The paper's claim to state-of-the-art long-video camera motion control is not yet established, however, because the comparison in Table 1 is confounded by resolution mismatch, a very small test set, no error bars, and a hyperparameter selected on the same test set. The manuscript would also benefit from releasing the pose VAE and checkpoints, since the contribution is largely architectural.

major comments (5)
  1. [§4.4, Table 1] The long-video SOTA claim is not supported by the presented evaluation because the comparison is not controlled. LDM-based baselines are evaluated at 256×256 or 384×256 resolution and are extended to 72 frames by chaining 16-frame segments with the previous final frame as reference, while CPA natively generates 72 frames at 640×360. All four metrics (FID, FVD, CLIPSIM, CamMC) are resolution- and protocol-sensitive. The reported long-video CamMC gaps (0.042 vs 0.044 for VD3D, 0.042 vs 0.068 for EasyAnimate) are small enough that they may reflect the evaluation setup rather than a genuine method advantage. Please report baselines at matched resolution with the same chaining or native-generation protocol, and include confidence intervals or multiple seeds.
  2. [§4.2, §4.4] The evaluation uses only 20 test videos, with no error bars, no multiple seeds, and no per-video breakdown. Without these, a CamMC difference of 0.002 (0.042 vs 0.044) or an FVD difference of 13 (1387 vs 1400) is within plausible sampling noise. Additionally, Table 1 reports no CamMC for OpenSora, the base model of CPA; without this number, the camera-control improvement contributed by the proposed modules over the backbone is not quantified. Please evaluate OpenSora on the same videos and report the CamMC, along with a bootstrap or per-video analysis.
  3. [§4.5, Table 2] The sparse-motion sampling ratio (40×) is selected using the same 20-video test set that produces the headline results of Table 1. This is test-set tuning and can inflate the reported performance. The ratio should be selected on a validation split, or the main results should be re-reported with the chosen ratio applied to a held-out test set. The same concern applies to the injection-module ablation in Table 3.
  4. [§3.2, Eqs. (3)–(6)] The Plücker-coordinate derivation is not mathematically complete. Eq. (3) defines x = PX = [R|t]X, but Eq. (4) then introduces the intrinsic matrix K, so the notation is inconsistent. Eq. (5) back-projects a pixel to a camera-coordinate point at an implicit depth of 1, and Eq. (6) forms an inner product of homogeneous optical-center coordinates with that point, which is not the standard Plücker-coordinate representation of a line. Please provide a complete derivation with clearly defined R, t, K, o_c, depth conventions, and the exact construction of the sparse RGB motion field shown in Fig. 5, so that the SME module is reproducible.
  5. [§3.3, §4.1] Several training details are underspecified or inconsistent. The text says 'freeze all layers except temporal attention layers', but then says 'introduce LoRA during the update of the self-attention'; it is not clear which parameters are actually updated. The VAE training is described only as reconstruction loss plus KL loss on the sparse motion field, with no architecture, frame count, optimizer, or reconstruction-quality numbers. Since the pose latent is a core input to the whole pipeline, please report these details and a reconstruction metric for the sparse-motion VAE.
minor comments (5)
  1. [Abstract and §4.6] The Abstract and Conclusion claim 'optimal performance in trajectory consistency and object consistency', but Sec. 4.6 states that object consistency is relatively weak and that large-scale object motion is limited. Please reconcile this wording with the stated limitations.
  2. [§3.2, Eqs. (8)–(10)] The TAI equations do not specify the dimensions of z^(k), z_p^(k), or the MLP output. Please include the exact tensor shapes, and state how the spatial patch count of the pose latent is aligned with the temporal-attention patch count when the two are inconsistent.
  3. [§4.2] The selection of the 20 test videos is described only as 'randomly select', with no seed or list. For reproducibility, please provide the specific video identifiers or a documented sampling procedure.
  4. [§4.5, Table 3] The 'Concat' baseline is described as 'channel-dimension concatation ... used in MotionCtrl', but it is not clear that this faithfully reproduces MotionCtrl's conditioning mechanism. Please clarify or cite the exact MotionCtrl configuration used.
  5. [General] The paper does not state whether code or checkpoints will be released. Given the plug-in design and the custom pose VAE, releasing these artifacts would substantially aid reproducibility and fair comparison.

Circularity Check

1 steps flagged · score 1.0 of 10

No derivation-level circularity; one minor test-set-selection self-reference in the 40× ablation, not load-bearing for the camera-conditioning pipeline.

  1. other [Section 4.2 (test set) and Section 4.5 / Table 2 (ablation)]
    "We randomly select 20 videos from the test set... We find that the 40× achieves the best results across all metrics, indicating that the camera pose motion sampling quantity at 40× is relatively optimal."

    The 40× sampling ratio is selected by evaluating 20×, 40×, and 80× on the same 20-video test set that is later used for the headline Table 1 comparisons. The claim '40× is optimal' is therefore a restatement of the selection criterion (argmax over ratios of the Table 2 metrics), not an independent validation. This is a mild self-reference: the evidence for the hyperparameter choice is the same measurement used to report the final SOTA numbers. However, it does not infect the core derivation, because the camera pose conditioning, VAE reconstruction, and diffusion training are not defined in terms of CamMC or FVD, and the evaluation metric is external to the training loss.

full rationale

The central derivation chain is self-contained. Camera pose matrices are converted to Plücker coordinates (Eq. 7), sparsely sampled into a motion field, compressed by a VAE trained with reconstruction and KL losses, injected into frozen ST-DiT temporal attention via LayerNorm and MLP (Eqs. 8-10), and optimized with the standard noise-prediction objective (Eq. 11). The evaluation metric CamMC is an external camera-consistency measure on generated videos, not a term in the training loss, so the target is not baked into the inputs by construction. No load-bearing self-citation or imported uniqueness theorem appears; cited works such as MotionCtrl, CameraCtrl, and VD3D provide architectural context or baselines, not justification for CPA's superiority. The only mild self-referential element is the ablation in Section 4.5, where the 40× sampling ratio is selected using the same 20-video test set later used for the headline comparison, making the '40× is optimal' statement a restatement of the selection criterion rather than an independent confirmation. This is an evaluation-validity concern (no error bars, resolution mismatches between LDM and DiT baselines, small test set) rather than a definitional circularity of the method itself.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim relies on several choices: the sparse sampling ratio (40x) is tuned on the test set; the VAE that converts motion fields into latents is trained only on RealEstate10K; and the evaluation assumes metric comparability across resolutions. No new physical entities are introduced.

free parameters (3)
  • Sparse motion sampling ratio = 40x (16x9 extraction points for 640x360)
    Chosen via ablation in Table 2; the 40x setting gives the best CamMC/FVD on the test set. The choice is made on the evaluation set, risking selection bias.
  • Guidance scale = 7.0
    Set in Section 4.1; standard diffusion sampling hyperparameter, not tied to data.
  • VAE latent compression factors = l=L/4, m=M/8, n=N/8
    Architecture choice in Section 3.2; affects temporal alignment but is not fitted to the target metric.
assumptions (4)
  • standard math Plucker coordinates are a valid line representation: P_{x,y} = [o_c, 1] * (R * K^{-1} * [x, y, 1]^T + t).
    Eq 6 in Section 3.2 assumes standard projective geometry but does not show the full derivation of Plucker coordinates.
  • domain assumption Camera intrinsic matrix K is known and constant for all frames.
    Eq 5 back-projects pixels using K^{-1}; inference-time K is not specified for user-provided camera poses.
  • ad hoc to paper A VAE trained on RealEstate10K sparse motion fields generalizes to OpenSora's temporal latent space.
    Section 3.3 trains the VAE only on RealEstate10K with reconstruction loss; no cross-domain validation is reported.
  • domain assumption Rendering the sparse motion field as an RGB image preserves camera-control information.
    Section 3.2 converts the field to RGB before VAE encoding; Table 2 ablations support this indirectly but only on the 20-video test set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CPA: Camera-pose-awareness Diffusion Transformer for Video Generation." pith.science (2026). https://pith.science/paper/4A6Y5LOT

@misc{pith2026241201429,
  author       = {Pith},
  title        = {Pith review of: CPA: Camera-pose-awareness Diffusion Transformer for Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4A6Y5LOT}},
  note         = {Machine review of arXiv:2412.01429}
}
read the original abstract

Despite the significant advancements made by Diffusion Transformer (DiT)-based methods in video generation, there remains a notable gap with controllable camera pose perspectives. Existing works such as OpenSora do NOT adhere precisely to anticipated trajectories and physical interactions, thereby limiting the flexibility in downstream applications. To alleviate this issue, we introduce CPA, a unified camera-pose-awareness text-to-video generation approach that elaborates the camera movement and integrates the textual, visual, and spatial conditions. Specifically, we deploy the Sparse Motion Encoding (SME) module to transform camera pose information into a spatial-temporal embedding and activate the Temporal Attention Injection (TAI) module to inject motion patches into each ST-DiT block. Our plug-in architecture accommodates the original DiT parameters, facilitating diverse types of camera poses and flexible object movement. Extensive qualitative and quantitative experiments demonstrate that our method outperforms LDM-based methods for long video generation while achieving optimal performance in trajectory consistency and object consistency.

Figures

Figures reproduced from arXiv: 2412.01429 by the authors.

Figure 1
Figure 1. The relevance of this work to video generation models. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of CPA. CPA includes the Sparse Motion Encoding (SME) Module and the Temporal Attention Injection (TAI) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of camera pose sequences encoding. The matrix parameters between adjacent frames are calculated to obtain the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Temporal Attention Injection Module. Layer normal [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: A visualization for camera pose series. We visualize image sequence after sparse motion sampling, with each row representing [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The performance of “zoom in” on three video generation methods, MotionCtrl, CameraCtrl and CPA. The text prompt is: “The [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The performance of “roundabout” on three video generation methods, MotionCtrl, CameraCtrl and CPA. The text prompt is: [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: The performance of “shaking” camera pose on three video generation methods, MotionCtrl, CameraCtrl and CPA. The text [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: The performance of MotionCtrl with object motion and CPA without object motion. Two cases are used, one with simple object [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Understanding Camera Motions in Any Video

    cs.CV 2025-04 conditional novelty 7.0 of 10

    CameraBench is a new expert-annotated benchmark with a 50-primitive camera motion taxonomy, revealing that SfM and VLMs fail in complementary ways and that SFT on the benchmark boosts VLM performance.

  2. Wonderland: Navigating 3D Scenes from a Single Image

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A feed-forward pipeline reconstructs 3D Gaussian scenes from single images by regressing 3DGS directly from camera-conditioned video diffusion latents.

  3. PostCam: Camera-Controllable Novel-View Video Generation with Query-Shared Cross-Attention

    cs.CV 2025-11 conditional novelty 6.0 of 10

    PostCam generates new videos from a reference video along user-specified camera trajectories using a query-shared cross-attention that fuses pose data and rendered frames, improving control precision and detail preservation.

  4. MoWorld: A Flash World Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    MoWorld achieves up to 50 FPS real-time interactive world simulation on NPUs by combining a 3D-native data engine, curriculum cross-frame pretraining, autoregressive distillation, and mixed-precision parallel inference.

Reference graph

Works this paper leans on

44 extracted references · 5 canonical work pages · cited by 4 Pith papers

  1. [1]

    Vd3d: Taming large video diffu- sion transformers for 3d camera control

    Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siaro- hin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffu- sion transformers for 3d camera control. arXiv preprint arXiv:2407.12781, 2024. 4, 5, 8

    arXiv 2024

  2. [2]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models

    Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024. 2

    arXiv 2024

  3. [3]

    Lumiere: A space- time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Her- rmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space- time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024. 1

    arXiv 2024

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1, 5, 8

    arXiv 2023

  5. [5]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 2

    2023

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  7. [7]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 1, 5, 8

    arXiv 2023

  8. [8]

    Cameractrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 1, 4, 5, 8

    arXiv 2024

Show all 44 references
  1. [9]

    Latent video diffusion models for high-fidelity video generation with arbitrary lengths

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2(3):4, 2022. 3

    2022 arXiv

  2. [10]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 5

    2017

  3. [11]

    Viewdiff: 3d-consistent image generation with text-to-image models

    Lukas H ¨ollein, Aljaˇz Boˇziˇc, Norman M¨uller, David Novotny, Hung-Yu Tseng, Christian Richardt, Michael Zollh ¨ofer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  4. [12]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 2

    2013 arXiv

  5. [13]

    Open-sora-plan, Apr

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, Apr

  6. [14]

    Image conductor: Precision control for interactive video syn- thesis

    Yaowei Li, Xintao Wang, Zhaoyang Zhang, Zhouxia Wang, Ziyang Yuan, Liangbin Xie, Yuexian Zou, and Ying Shan. Image conductor: Precision control for interactive video syn- thesis. arXiv preprint arXiv:2406.15339, 2024. 2

    2024 arXiv

  7. [15]

    Motionclone: Training-free motion cloning for controllable video generation

    Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. arXiv preprint arXiv:2406.05338, 2024. 2

    2024 arXiv

  8. [16]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 2

    2024 arXiv

  9. [17]

    Snap video: Scaled spatiotemporal transformers for text-to-video synthesis

    Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. In Proceedings of the IEEE/CVF Conference...

    2024

  10. [18]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  11. [19]

    Controlnext: Powerful and effi- cient control for image and video generation

    Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming- Chang Yang, and Jiaya Jia. Controlnext: Powerful and effi- cient control for image and video generation. arXiv preprint arXiv:2408.06070, 2024. 2

    2024 arXiv

  12. [20]

    Camera pose estimation from lines using pl ¨ucker coordinates

    Bronislav Pr ˇribyl, Pavel Zemˇc´ık, and Martin ˇCad´ık. Camera pose estimation from lines using pl ¨ucker coordinates. arXiv preprint arXiv:1608.02824, 2016. 4

    2016 arXiv

  13. [21]

    Freetraj: Tuning-free tra- jectory control in video diffusion models

    Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free tra- jectory control in video diffusion models. arXiv preprint arXiv:2406.16863, 2024. 2

    2024 arXiv

  14. [22]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  15. [23]

    Consisti2v: Enhanc- ing visual consistency for image-to-video generation

    Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhanc- ing visual consistency for image-to-video generation. arXiv preprint arXiv:2402.04324, 2024. 2

    2024 arXiv

  16. [24]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  17. [25]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  18. [26]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers , p...

    2024

  19. [27]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  20. [28]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 5

    2018 arXiv

  21. [29]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 2

    2017

  22. [30]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  23. [31]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Pa- pers, pages 1–11, 2024. 1, 4, 5, 8

    2024

  24. [32]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  25. [33]

    Easyanimate: A high-performance long video generation method based on transformer architecture

    Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture. arXiv preprint arXiv:2405.18991,

  26. [34]

    Direct-a-video: Customized video generation with user- directed camera movement and object motion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user- directed camera movement and object motion. In ACM SIG- GRAPH 2024 Conference Papers, pages 1–12, 2024. 2

    2024

  27. [35]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2

    2024 arXiv

  28. [36]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 4

    arXiv

  29. [37]

    Make pixels dance: High- dynamic video generation

    Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High- dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8850–8860, 2024. 2

    2024

  30. [38]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2

    2023

  31. [39]

    Videoelevator: Elevating video generation quality with versatile text-to-image diffusion models

    Yabo Zhang, Yuxiang Wei, Xianhui Lin, Zheng Hui, Peiran Ren, Xuansong Xie, Xiangyang Ji, and Wangmeng Zuo. Videoelevator: Elevating video generation quality with versatile text-to-image diffusion models. arXiv preprint arXiv:2403.05438, 2024. 2

    2024 arXiv

  32. [40]

    Tora: Trajectory-oriented diffu- sion transformer for video generation

    Zhenghao Zhang, Junchao Liao, Menghao Li, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented diffu- sion transformer for video generation. arXiv preprint arXiv:2407.21705, 2024. 1

    2024 arXiv

  33. [41]

    Open-sora: Democratizing efficient video production for all, March 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. 1, 5, 8

    2024

  34. [42]

    Magicvideo: Efficient video generation with latent diffusion models

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 2

    2022 arXiv

  35. [43]

    Trackgo: A flexible and efficient method for controllable video generation.arXiv preprint arXiv:2408.11475, 2024

    Haitao Zhou, Chuang Wang, Rui Nie, Jinxiao Lin, Dong- dong Yu, Qian Yu, and Changhu Wang. Trackgo: A flexible and efficient method for controllable video generation.arXiv preprint arXiv:2408.11475, 2024. 2

    2024 arXiv

  36. [44]

    Stereo magnification: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817, 2018. 2

    2018 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.