Pith. sign in

REVIEW 4 major objections 3 minor 121 references

LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusion

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LoRA fusion for video diffusion can be made linear, orthogonal, and norm-consistent, so a single scaling token linearly controls camera trajectory and object motion with far less training data.

desk verdict Useful scaling-token idea for controllable video generation, but the paper's headline claim of decoupled linear multi-LoRA control is untested. read the letter →

arxiv 2507.05678 v1 pith:PKXK2HQO submitted 2025-07-08 cs.CV

Yisu Zhang , Chenjie Cao , Chaohui Yu , Jianke Zhu This is my paper
classification cs.CV
keywords LiON-LoRAvideodiffusionmodelsLofusioncameratrajectorycontrolmotionstrengthscalingtokennormconsistencyorthogonality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LiON-LoRA is proposed to establish that low-level camera and object-motion control in a video diffusion model can be handled by rethinking how LoRA adapters are fused, rather than by injecting explicit camera poses. It rests on three principles: camera LoRAs trained from basic motion primitives are nearly orthogonal in the shallow transformer blocks; normalizing each layer's LoRA output makes combinations smooth; and a dedicated scaling token gives a linear lever over motion amplitude. The paper demonstrates this with a CogVideoX-based system trained on 100 rendered scenes for camera primitives and static-camera videos for object motion, using about 4,000 fine-tuning steps. If the claims hold, trajectory accuracy and motion-strength adjustment improve over explicit-control baselines while using far less annotated data.

What carries the argument

The engine is the scaling token $E = \mathrm{linear}(\gamma(S))$, a trainable embedding of the scalar amplitude $S$ (Fourier positional features encoding $S$ from a small minimum up to $1.0$) concatenated to the visual and textual token sequence $H$. During fusion, $k$ tokens are concatenated to form $H' = [H; E_1; \dots; E_k]$, and each LoRA computes its own $H_i' = [H; E_i]$ inside its own attention subspace; the $k$ sets of visual and textual tokens are averaged into $H_{\mathrm{out}}$ while the scaling tokens are concatenated. The other two pillars prepare the ground: per-layer norm normalization $\Delta \hat{W}_i = \alpha \|\Delta W_i\|^{-1} \Delta W_i$, and the orthogonality observation that shallow-layer camera LoRA outputs have low cosine similarity, so their controls do not interfere.

What would settle it

Run a fusion of a forward LoRA and an orbit-left LoRA, fix one scaling token, sweep the other through its full range, and recover the resulting camera poses with GLOMAP; if the trajectory does not vary monotonically with the swept token while the fixed motion stays constant, the decoupled linear-scalability claim is false.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the three failure modes of vanilla LoRA fusion for camera control—correlated features, mismatched magnitudes, and non-linear intensity—all have vector-space fixes. In shallow layers, different camera LoRAs have average cosine similarity of $0.06\pm0.06$, so they can be treated as nearly orthogonal and hence decoupled; deep-layer correlations are less relevant because low-frequency camera control is encoded early. Norms vary enough across trajectories and blocks that a single adapter scale cannot balance them, so each LoRA output is renormalized to a common norm. Linear amplitude control comes from a Fourier-embedded scaling token appended to the token sequence, letting the model see the desired scale $S$ explicitly instead of tuning an adapter coefficient. At fusion time, each LoRA operates in its own attention subspace with its own scaling token, and the visual and textual outputs are averaged, which the paper argues keeps controls independent without joint training.

Load-bearing premise

The argument depends on the averaging of the $k$ attention outputs keeping each LoRA's control signal separate; if hidden states mix, the claimed decoupled linear scalability for camera and object motion fails.

Editorial extensions

If this is right

  • Basic camera-trajectory errors drop by roughly a quarter on rotation error and one-fifth on absolute trajectory error relative to the strongest prior controller tested, with lower FVD on the same base model.
  • A single camera primitive can be trained with about 100 rendered scenes and 4,000 fine-tuning steps, and the reported ablations show that increasing to 7,000 samples at 4,000 steps does not improve accuracy, so minimal data suffices.
  • Because fusion is training-free, combining several camera primitives, or camera with object-motion LoRA, requires no joint fine-tuning; the scaling tokens stay separate through attention.
  • The same scaling token used for camera trajectory also linearly controls object-motion strength, measured by Pearson correlation with optical flow, so spatial and temporal controllability share one mechanism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shallow-layer orthogonality observed here is a general property of DiT video models rather than an artifact of CogVideoX, the same three fixes could be dropped into other transformer-based video generators without retraining the base model.
  • Because the scaling token is just an extra sequence element, one natural extension is to use several tokens per LoRA, such as one for trajectory shape and one for speed, to obtain even richer continuous controls; the paper does not explore that.
  • Norm-consistency normalization might also stabilize high-level LoRA composition, such as style plus subject, where the paper only tests camera and motion LoRAs; an explicit transfer study would be a cheap check.
  • A practical product consequence is a two-slider interface for camera path and motion speed, since the reported Pearson correlation suggests the mapping is near-linear; this is an extension, not a result the paper proves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The manuscript proposes LiON-LoRA, a parameter-efficient framework for controlling camera trajectories and object-motion strength in video diffusion models. It trains separate LoRA modules for discrete camera primitives on rendered DL3DV videos, normalizes per-layer LoRA output norms, and injects a dedicated scaling token (with Fourier positional embedding) to encode the desired motion amplitude. For fusing multiple LoRAs, it runs each LoRA in an attention subspace attending only to its own scaling token and averages the resulting visual/textual hidden states. The same scaling-token mechanism is applied to motion-strength control using static-camera videos, claiming unified spatial and temporal control. Experiments report lower RotErr, TransErr, ATE, and FVD than several baselines, plus a Pearson-correlation analysis for motion strength.

Significance. If the central claims held, LiON-LoRA would be a useful and practical contribution: it achieves strong camera-control metrics with only 100 training scenes and 4,000 fine-tuning steps, keeps the base VDM unchanged at inference, and proposes a clean, training-free fusion scheme. The paper deserves credit for including an ablation on dataset size and training iterations (Table 3), for re-implementing DimensionX-S on the same CogVideoX backbone as a controlled baseline, and for explicitly separating scaling-token control from adapter-scale control in Figure 10. However, the evidence as presented does not yet establish the headline property of decoupled linear scalability for multi-LoRA fusion, and several evaluation choices (train/eval scene separation, error bars, baseline comparability) need to be addressed before the quantitative gains can be taken at face value.

major comments (4)
  1. [Section 3.5, Figure 8] The central claim of decoupled linear scalability is not quantitatively supported. The fusion averages k sets of visual/textual tokens into H_out while concatenating the scaling tokens; however, all subsequent transformer blocks operate on this common averaged H together with all scaling tokens, and softmax attention and LoRA updates do not commute with averaging. Hence there is no reason, without further argument, that the fused output equals a linear combination of the individual LoRA-controlled outputs. The paper provides no multi-LoRA experiment that sweeps the scaling values S1...Sk and checks whether camera trajectory and motion strength respond linearly and independently. Table 2 evaluates fixed complex poses, Figure 10 tests only a single camera primitive, and Figure 6 tests motion strength without fusion. Please add a controlled multi-LoRA scalability experiment (e.g., independent sweeps of camera S and motion S, with the ideal linear prediction overlaid) and, if possible, a diagnostic that measures cross-LoRA coupling.
  2. [Section 4.1] The evaluation split between training and test scenes is unclear. The paper states that 100 DL3DV scenes are reconstructed and rendered for each primitive, and later that 'we selected 100 samples from DL3DV for each motion primitive' for quantitative evaluation, with no statement that these are held-out scenes. If the evaluation scenes overlap with the 100 training scenes, the trajectory metrics in Tables 1-2 may reflect memorization rather than generalization. Please specify the exact train/eval scene split and report metrics separately for seen and unseen scenes.
  3. [Tables 1-3, Section 4.3] The numerical comparisons lack error bars and are partly across uncontrolled settings. Tables 1 and 2 report a single run per method, so differences such as TransErr 0.197 vs. 0.201 and ATE 0.345 vs. 0.398 in Table 2 may be within sampling noise; multiple seeds and confidence intervals are needed. In addition, FVD is compared across different base models and sequence lengths (e.g., CamI2V is a different architecture and uses shorter sequences), which makes the reported 37.8% FVD improvement not directly attributable to LiON-LoRA. Only DimensionX-S* is a same-backbone re-implementation; please either re-run the other baselines under CogVideoX/49-frame evaluation or clearly restrict the FVD comparison to matched settings.
  4. [Section 3.4, Figure 6] The Pearson-correlation result is partially self-consistent by construction. For camera control, S directly defines the number of frames sampled from a rendered trajectory (first 600*S frames), so larger S corresponds to greater motion by design; the analogous construction holds for the 240-frame object-motion videos. Demonstrating that the generated outputs correlate with S is therefore evidence that the model follows its training signal, but it is not independent evidence of linear scalability. Please include held-out, continuously varied S values that are not derived from the frame-subsampling protocol, and compare against adapter scaling with identical normalization, reporting the fitted slope and correlation with error bars.
minor comments (3)
  1. [Equation (4)] The definition of alpha is ambiguous: 'alpha = P(||Delta W_i||)/k' does not state what P denotes. Please specify whether P is the product, mean, or another aggregation, and clarify that the norm is computed per layer and per sample.
  2. [Section 3.4] The phrase 'we use specific project linear to encode S for different camera LoRAs' should be rewritten. Also specify the exact Fourier dimension J and the minimal scaling values s used in the experiments.
  3. [Section 4.2] The spelling 'Cami2V' appears in the text while the tables use 'CamI2V'; please unify the notation throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LiON-LoRA's conditioning and fusion are evaluated against independent external benchmarks, and the scaling-token supervision is not a fitted prediction.

full rationale

The paper's central derivation chain is self-contained. The scaling token S is defined as a conditioning input, not as a parameter fitted to the evaluated outputs; the training pairs are constructed by sampling 49 frames from the first 600·S frames, and the Pearson-correlation evaluation in Section 3.4/Figure 6 is an empirical check of whether the model learned the intended S-to-motion mapping. This is a self-consistency evaluation, not a reduction of the predicted quantity to a fitted input. The norm-consistency formula (Eq. 4) is a fixed layer-wise normalization with a balancing constant, not a fitted parameter renamed as a result. The orthogonality analysis is a measured feature-statistic claim, and the low-frequency control claim relies on external citations ([1], [77]) rather than a self-citation chain. The main quantitative claims are validated against independent baselines (CameraCtrl, MotionCtrl, CamI2V, DimensionX) using standard metrics (RotErr, TransErr, ATE, FVD) in Tables 1 and 2, providing external support. No load-bearing step reduces, by the paper's own equations or by self-citation, to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a handful of design choices: the LoRA rank, the norm-normalization recipe, the Fourier embedding dimension, and the fusion protocol. The scaling token is the primary invented mechanism, and its independence is testable. The domain assumptions about rendered trajectories and shallow-layer orthogonality are not formally derived.

free parameters (4)
  • LoRA rank r = 256
    Chosen rank for all LoRA adapters; larger than typical LoRA ranks, close to low-rank fine-tuning scale.
  • norm normalization scale alpha = alpha = P(||dWi||)/k
    Computed from the trained LoRA weights; the balancing constant central to norm consistency. It is derived from the weights, not fitted to the evaluation metric, but it is a hand-set recipe.
  • Fourier embedding dimension J = not stated in main text
    Number of frequency indices in the scaling-token Fourier embedding; deferred to supplementary.
  • min scaling s = 49/600 (camera), 49/240 (motion)
    Determined by frame counts; not really free, but it bounds the controllable range.
assumptions (4)
  • domain assumption Rendered DL3DV trajectories from 3DGS are a valid proxy for real camera motion and transfer to real videos
    The training and quantitative evaluation rely on synthetic rendered trajectories; generalization to web images is only shown qualitatively.
  • domain assumption Camera control is encoded primarily in shallow VDM layers (cited [1,77])
    Used to argue shallow-layer orthogonality is sufficient for decoupled low-level control; no formal proof given.
  • ad hoc to paper Averaging output tokens across LoRA-specific attention subspaces preserves independent control
    The fusion design in Section 3.5 and Figure 8 assumes averaging does not couple the control signals; no theoretical or ablative support.
  • domain assumption The diffusion model's denoising process remains valid when a scaling token is appended to the token sequence
    The architecture change assumes the transformer can absorb an extra token without destabilizing generation; empirically shown, not proved.
invented entities (1)
  • scaling token E independent evidence
    purpose: Conditioning token encoding scalar control value S for camera or motion amplitude
    Its effect is observable in generated videos via optical flow and pose estimation; it is a learned embedding, not a physical entity, but it is a new architectural element.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusion." pith.science (2026). https://pith.science/paper/PKXK2HQO

@misc{pith2026250705678,
  author       = {Pith},
  title        = {Pith review of: LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PKXK2HQO}},
  note         = {Machine review of arXiv:2507.05678}
}
read the original abstract

Video Diffusion Models (VDMs) have demonstrated remarkable capabilities in synthesizing realistic videos by learning from large-scale data. Although vanilla Low-Rank Adaptation (LoRA) can learn specific spatial or temporal movement to driven VDMs with constrained data, achieving precise control over both camera trajectories and object motion remains challenging due to the unstable fusion and non-linear scalability. To address these issues, we propose LiON-LoRA, a novel framework that rethinks LoRA fusion through three core principles: Linear scalability, Orthogonality, and Norm consistency. First, we analyze the orthogonality of LoRA features in shallow VDM layers, enabling decoupled low-level controllability. Second, norm consistency is enforced across layers to stabilize fusion during complex camera motion combinations. Third, a controllable token is integrated into the diffusion transformer (DiT) to linearly adjust motion amplitudes for both cameras and objects with a modified self-attention mechanism to ensure decoupled control. Additionally, we extend LiON-LoRA to temporal generation by leveraging static-camera videos, unifying spatial and temporal controllability. Experiments demonstrate that LiON-LoRA outperforms state-of-the-art methods in trajectory control accuracy and motion strength adjustment, achieving superior generalization with minimal training data. Project Page: https://fuchengsu.github.io/lionlora.github.io/

Figures

Figures reproduced from arXiv: 2507.05678 by the authors.

Figure 1
Figure 1. Camera and motion control results of our LiON-LoRA. LiON-LoRA can linearly control both camera trajectory and object motion in videos generated by the video diffusion model. Furthermore, based on LoRA fine-tuning, LiON-LoRA achieves excellent generalization with minimal training data. the conventional approach of adjusting the multiplying co￾efficients of LoRA features is adequate for modifying high￾level properties… view at source ↗
Figure 3
Figure 3. Illustration of the dilemma for LoRA fusion in cam￾era control. We show the 2D toy case to demonstrate the impor￾tance of orthogonality and norm consistency. Non-orthogonality results in undesired enhancements or conflicts of fusions (b, c), while significantly different norms lead to unstable fusion (d). is an implicit way to control camera trajectories, explicit lin￾ear scalability is essential to achieve accurate… view at source ↗
Figure 2
Figure 2. Comparison between the vanilla LoRA fusion and ours. The vanilla fusion of “forward” and “orbit left” LoRA con￾trols with adjusted adapter scales suffers from abrupt changes. While our LiON-LoRA enhanced with norm consistency enjoys a much smoother combination without parameter adjusting. ing distance according to the medium depth of each scene. More details are discussed in the supplementary. 3.2. Rethinking LoRA F… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Cosine similarity of different camera LoRAs. In the shallow layers, LoRAs maintain orthogonality, while in the later layers, coupling effects begin to emerge. Case1 Case2 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Norm Consistency of different LoRAs. The norms of outputs from different camera control LoRAs vary significantly for different samples, which occur in both early and later layers, reflecting that LoRA weights fail to ensure norm consistency. Norm Consistency. As illust…
Figure 6
Figure 6. Figure 6: The results demonstrate that our method achieves rapid convergence during the early stages of fine-tuning LiON-LoRA. In contrast, the vanilla adapter scaling fails to establish a meaningful correlation for effective control. Moreover, we show the qualitative results in…
Figure 7
Figure 7. Figure 7: Illustration of motion strength scaling. Our LiON￾LoRA enables the control of object motion amplitude through the scaling token without complex bounding box annotations or drag￾based interactions. Furthermore, this parameter-driven approach ensures that the generated v…
Figure 9
Figure 9. Figure 9: Qualitative comparisons on camera pose control. (a) Our model achieves diverse styles with strong generalization and adapt￾ability. Existing methods (MotionCtrl [91], CameraCtrl [23], CamI2V [119]) struggle with inconsistent scene geometry and hallucinations across sty…
Figure 10
Figure 10. Figure 10: Linear scalability of LoRA. Vanilla LoRA adjusts in￾jection strength via the global “adapter scale” λ, which lacks preci￾sion in camera motion control. Specifically, an insufficient adapter scale fails to control the camera of VDM, while a large adapter scale may lead…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

121 extracted references · 26 canonical work pages

  1. [1]

    Ac3d: Analyzing and improving 3d camera control in video diffusion trans- formers

    Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Ali- aksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera control in video diffusion trans- formers. arXiv preprint arXiv:2411.18673, 2024. 1, 4

    arXiv 2024

  2. [2]

    Scaling learning algo- rithms towards AI

    Yoshua Bengio and Yann LeCun. Scaling learning algo- rithms towards AI. In Large Scale Kernel Machines. MIT Press, 2007

    2007

  3. [3]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1, 2, 3

    arXiv 2023

  5. [5]

    Align your latents: High-resolution video synthe- sis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthe- sis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023

    2023

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  7. [7]

    Mvsformer: Multi-view stereo by learning robust image features and temperature-based depth

    Chenjie Cao, Xinlin Ren, and Yanwei Fu. Mvsformer: Multi-view stereo by learning robust image features and temperature-based depth. Transactions of Machine Learn- ing Research, 2023

    2023

  8. [8]

    Still-moving: Customized video generation without customized video data

    Hila Chefer, Shiran Zada, Roni Paiss, Ariel Ephrat, Omer Tov, Michael Rubinstein, Lior Wolf, Tali Dekel, Tomer Michaeli, and Inbar Mosseri. Still-moving: Customized video generation without customized video data. arXiv preprint arXiv:2407.08674, 2024. 3

    arXiv 2024

Show all 121 references
  1. [9]

    Motion-zero: Zero-shot mov- ing object control framework for diffusion-based video gen- eration

    Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He, Changbo Wang, and Yang Li. Motion-zero: Zero-shot mov- ing object control framework for diffusion-based video gen- eration. arXiv preprint arXiv:2401.10150, 2024. 3

    2024 arXiv

  2. [10]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 1, 2

    arXiv

  3. [11]

    Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7310– 73...

    2024

  4. [12]

    Seine: Short-to-long video diffu- sion model for generative transition and prediction

    Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffu- sion model for generative transition and prediction. In The Twelfth International Conference on Learning Representa- ti...

    2023

  5. [13]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021

    2021

  6. [14]

    Parameter-efficient fine-tuning of large-scale pre-trained language models

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220–235, 2023. 1

    2023

  7. [15]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023

    2023

  8. [16]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first international conference on mach...

  9. [17]

    I2vcontrol-camera: Precise video camera control with adjustable motion strength

    Wanquan Feng, Jiawei Liu, Pengqi Tu, Tianhao Qi, Mingzhen Sun, Tianxiang Ma, Songtao Zhao, Siyu Zhou, and Qian He. I2vcontrol-camera: Precise video camera control with adjustable motion strength. arXiv preprint arXiv:2411.06525, 2024. 3

    2024 arXiv

  10. [18]

    Preserve your own cor- relation: A noise prior for video diffusion models

    Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, An- drew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji. Preserve your own cor- relation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on C...

    2023

  11. [19]

    Emu video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factoriz- ing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023. 2

    2023 arXiv

  12. [20]

    Deep learning

    Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT Press, 2016

    2016

  13. [21]

    Sparsectrl: Adding sparse con- trols to text-to-video diffusion models

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse con- trols to text-to-video diffusion models. arXiv preprint arXiv:2311.16933, 2023. 3

    2023 arXiv

  14. [22]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 3

    2023 arXiv

  15. [23]

    Cameractrl: En- abling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: En- abling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 1, 3, 7, 8

    2024 arXiv

  16. [24]

    Co-speech gesture video genera- tion via motion-decoupled diffusion model

    Xu He, Qiaochu Huang, Zhensong Zhang, Zhiwei Lin, Zhiyong Wu, Sicheng Yang, Minglei Li, Zhiyi Chen, Song- cen Xu, and Xiaofei Wu. Co-speech gesture video genera- tion via motion-decoupled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- te...

    2024

  17. [25]

    Epipolar transformers

    Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. In Proceedings of the ieee/cvf con- ference on computer vision and pattern recognition , pages 7779–7788, 2020

    2020

  18. [26]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221, 2022. 1, 2

    2022 arXiv

  19. [27]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017

    2017

  20. [28]

    Hinton, Simon Osindero, and Yee Whye Teh

    Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Com- putation, 18:1527–1554, 2006

    2006

  21. [29]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  22. [30]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural informa- tion processing systems, 33:6840–6851, 2020

    2020

  23. [32]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 1, 3

    2021 arXiv

  24. [33]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 1, 2, 3

    2024

  25. [34]

    Mo- tionmaster: Training-free camera motion transfer for video generation

    Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, and Lizhuang Ma. Mo- tionmaster: Training-free camera motion transfer for video generation. arXiv preprint arXiv:2404.15789, 2024. 3

    2024 arXiv

  26. [35]

    Lorahub: Efficient cross- task generalization via dynamic lora composition

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross- task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023. 1

    2023 arXiv

  27. [36]

    Peekaboo: Interactive video generation via masked- diffusion

    Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked- diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8079– 8088, 2024. 1, 3

    2024

  28. [37]

    A survey of multimodal con- trollable diffusion models

    Rui Jiang, Guang-Cong Zheng, Teng Li, Tian-Rui Yang, Jing-Dong Wang, and Xi Li. A survey of multimodal con- trollable diffusion models. Journal of Computer Science and Technology, 39(3):509–541, 2024

    2024

  29. [38]

    Consistent4d: Consistent 360° dynamic object gener- ation from monocular video

    Yanqin Jiang, Li Zhang, Jin Gao, Weiming Hu, and Yao Yao. Consistent4d: Consistent 360° dynamic object gener- ation from monocular video. In The Twelfth International Conference on Learning Representations, 2024. 1

    2024

  30. [39]

    Animate3d: Animating any 3d model with multi-view video diffusion

    Yanqin Jiang, Chaohui Yu, Chenjie Cao, Fan Wang, Weim- ing Hu, and Jin Gao. Animate3d: Animating any 3d model with multi-view video diffusion. Advances in Neural Infor- mation Processing Systems, 37:125879–125906, 2025. 1

    2025

  31. [40]

    Spad: Spatially aware multi-view diffusers

    Yash Kant, Aliaksandr Siarohin, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, and Igor Gilitschenski. Spad: Spatially aware multi-view diffusers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- ter...

    2024

  32. [41]

    Co- tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together. In European Confer- ence on Computer Vision, pages 18–35. Springer, 2024. 7

    2024

  33. [42]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1,

  34. [43]

    Hunyuanvideo: A systematic frame- work for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jian- wei Zhang, et al. Hunyuanvideo: A systematic frame- work for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 3, 7

    2024 arXiv

  35. [44]

    Kuaishou. Kling. https://klingai.kuaishou.com, 2024. 2

    2024

  36. [45]

    Realcam-i2v: Real-world image- to-video generation with interactive complex camera con- trol

    Teng Li, Guangcong Zheng, Rui Jiang, Tao Wu, Yehao Lu, Yining Lin, Xi Li, et al. Realcam-i2v: Real-world image- to-video generation with interactive complex camera con- trol. arXiv preprint arXiv:2502.10059, 2025. 3

    2025 arXiv

  37. [46]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023

    2023

  38. [47]

    Generative image dynamics

    Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24142–24153, 2024. 3

    2024

  39. [48]

    Open-sora plan: Open-source large video generation model

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00131,

    arXiv

  40. [49]

    Magic3d: High- resolution text-to-3d content creation

    Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High- resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2023

  41. [50]

    Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision

    Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , p...

    2024

  42. [51]

    Zero-1-to- 3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to- 3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 9298–9309, 2023

    2023

  43. [52]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 3

    2024 arXiv

  44. [53]

    Follow your pose: Pose- guided text-to-video generation using pose-free videos

    Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose- guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 4117–4125, 2024

    2024

  45. [54]

    Sdedit: Guided image synthesis and editing with stochastic differential equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021

    2021 arXiv

  46. [55]

    On distillation of guided diffusion models

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14297–14306, 2023

    2023

  47. [56]

    MiniMax. Hailuo. https://hailuoai.com/video, 2024

    2024

  48. [57]

    T2i-adapter: Learn- ing adapters to dig out more controllable ability for text-to- image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learn- ing adapters to dig out more controllable ability for text-to- image diffusion models. In Proceedings of the AAAI Con- ference on Artificial Intelligence , pages 4296–43...

    2024

  49. [58]

    Sg-i2v: Self- guided trajectory control in image-to-video generation

    Koichi Namekata, Sherwin Bahmani, Ziyi Wu, Yash Kant, Igor Gilitschenski, and David B Lindell. Sg-i2v: Self- guided trajectory control in image-to-video generation. arXiv preprint arXiv:2411.04989, 2024. 3

    2024 arXiv

  50. [59]

    K-lora: Unlock- ing training-free fusion of any subject and style loras.arXiv preprint arXiv:2502.18461, 2025

    Ziheng Ouyang, Zhen Li, and Qibin Hou. K-lora: Unlock- ing training-free fusion of any subject and style loras.arXiv preprint arXiv:2502.18461, 2025. 1, 4

    2025 arXiv

  51. [60]

    Global Structure-from-Motion Revisited

    Linfei Pan, Daniel Barath, Marc Pollefeys, and Jo- hannes Lutz Sch ¨onberger. Global Structure-from-Motion Revisited. In European Conference on Computer Vision (ECCV), 2024. 7

    2024

  52. [61]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 4195– 4205, 2023. 3

    2023

  53. [62]

    Controlnext: Powerful and effi- cient control for image and video generation.arXiv preprint arXiv:2408.06070, 2024

    Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming- Chang Yang, and Jiaya Jia. Controlnext: Powerful and effi- cient control for image and video generation.arXiv preprint arXiv:2408.06070, 2024. 3

    2024 arXiv

  54. [63]

    Analytisch-Geometrische Entwicklungen

    Julius Pl ¨ucker. Analytisch-Geometrische Entwicklungen. GD Baedeker, 1828. 7

  55. [64]

    Dreamfusion: Text-to-3d using 2d diffusion

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Milden- hall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022

    2022 arXiv

  56. [65]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  57. [66]

    Consisti2v: En- hancing visual consistency for image-to-video generation

    Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: En- hancing visual consistency for image-to-video generation. arXiv preprint arXiv:2402.04324, 2024

    2024 arXiv

  58. [67]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. 2022

    2022

  59. [68]

    Mm-diffusion: Learning multi-modal diffusion mod- els for joint audio and video generation

    Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion mod- els for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2023

  60. [69]

    Gen-3 alpha

    RunwayML. Gen-3 alpha. https://runwayml.com/research/ introducing-gen-3-alpha, 2024. 2

    2024

  61. [70]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Sali- mans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural in- forma...

    2022

  62. [71]

    Structure-from-motion revisited

    Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016

    2016

  63. [72]

    Structure-from-Motion Revisited

    Johannes Lutz Sch ¨onberger and Jan-Michael Frahm. Structure-from-Motion Revisited. In Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016

    2016

  64. [73]

    Pixelwise view selection for unstructured multi-view stereo

    Johannes L Sch ¨onberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Nether- lands, October 11-14, 2016, Proceedings, Part III 14, pag...

    2016

  65. [74]

    Ziplora: Any subject in any style by effectively merging loras

    Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svet- lana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. 2023. 1, 4

    2023

  66. [75]

    Moma: Multimodal llm adapter for fast personalized image generation

    Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, and Xiao Yang. Moma: Multimodal llm adapter for fast personalized image generation. arXiv preprint arXiv:2404.05674, 2024. 3

    2024 arXiv

  67. [76]

    Score- based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  68. [77]

    Dimensionx: Create any 3d and 4d scenes from a single image with con- trollable video diffusion

    Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and Yikai Wang. Dimensionx: Create any 3d and 4d scenes from a single image with con- trollable video diffusion. arXiv preprint arXiv:2411.04928,

    arXiv

  69. [78]

    Stable score distillation for high-quality 3d generation

    Boshi Tang, Jianan Wang, Zhiyong Wu, and Lei Zhang. Stable score distillation for high-quality 3d generation. arXiv preprint arXiv:2312.09305, 2023

    2023 arXiv

  70. [79]

    Dreamgaussian: Generative gaussian splat- ting for efficient 3d content creation

    Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splat- ting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023

    2023 arXiv

  71. [80]

    Any-to-any generation via composable diffusion

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. In Thirty-seventh Conference on Neural Informa- tion Processing Systems, 2023. 3

    2023

  72. [81]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485, 2024. 3

    2024 arXiv

  73. [82]

    Vision transformers need registers

    Darcet Timoth ´ee, Oquab Maxime, Mairal Julien, and Bo- janowski Piotr. Vision transformers need registers. In The Twelfth International Conference on Learning Representa- tions, 2024

    2024

  74. [83]

    Consistent view syn- thesis with pose-guided diffusion models

    Hung-Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Consistent view syn- thesis with pose-guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16773–16783, 2023

    2023

  75. [84]

    Towards accurate generative models of video: A new met- ric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new met- ric & challenges. arXiv preprint arXiv:1812.01717, 2018. 7

    2018 arXiv

  76. [85]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 , 2023. 1, 2

    2023 arXiv

  77. [86]

    Disco: Disentangled control for referring hu- man dance generation in real world

    Tan Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Li- juan Wang. Disco: Disentangled control for referring hu- man dance generation in real world. arXiv e-prints, pages arXiv–2307, 2023

    2023

  78. [87]

    Videofactory: Swap attention in spatiotemporal diffusions for text-to-video gen- eration, 2024

    Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video gen- eration, 2024

    2024

  79. [88]

    Videocomposer: Compositional video syn- thesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video syn- thesis with motion controllability. Advances in Neural In- formation Processing Systems, 36, 2024. 1, 3

    2024

  80. [89]

    Lavie: High-quality video genera- tion with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video genera- tion with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 1, 2

    2023 arXiv

  81. [90]

    Customvideo: Customizing text- to-video generation with multiple subjects

    Zhao Wang, Aoxue Li, Enze Xie, Lingting Zhu, Yong Guo, Qi Dou, and Zhenguo Li. Customvideo: Customizing text- to-video generation with multiple subjects. arXiv preprint arXiv:2401.09962, 2024. 3

    2024

  82. [91]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 1, 3, 7, 8

    2024

  83. [92]

    Dreamvideo: Composing your dream videos with customized subject and motion

    Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhi- heng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hong- ming Shan. Dreamvideo: Composing your dream videos with customized subject and motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  84. [93]

    Motionbooth: Motion-aware customized text-to- video generation

    Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Motionbooth: Motion-aware customized text-to- video generation. arXiv preprint arXiv:2406.17758, 2024. 3

    2024 arXiv

  85. [94]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  86. [95]

    Spherediffusion: Spherical geometry-aware distortion resilient diffusion model

    Tao Wu, Xuewei Li, Zhongang Qi, Di Hu, Xintao Wang, Ying Shan, and Xi Li. Spherediffusion: Spherical geometry-aware distortion resilient diffusion model. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 6126–6134, 2024. 3

    2024

  87. [96]

    Customcrafter: Customized video generation with pre- serving motion and concept composition abilities

    Tao Wu, Yong Zhang, Xintao Wang, Xianpan Zhou, Guangcong Zheng, Zhongang Qi, Ying Shan, and Xi Li. Customcrafter: Customized video generation with pre- serving motion and concept composition abilities. arXiv preprint arXiv:2408.13239, 2024. 3

    2024 arXiv

  88. [97]

    Draganything: Motion control for any- thing using entity representation

    Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for any- thing using entity representation. In European Conference on Computer Vision, pages 331–348. Springer, 2024. 3

    2024

  89. [98]

    Ifadapter: Instance feature con- trol for grounded text-to-image generation

    Yinwei Wu, Xianpan Zhou, Bing Ma, Xuefeng Su, Kai Ma, and Xinchao Wang. Ifadapter: Instance feature con- trol for grounded text-to-image generation. arXiv preprint arXiv:2409.08240, 2024. 3

    2024 arXiv

  90. [99]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Eu- ropean Conference on Computer Vision , pages 399–417. Springer, 202...

    2024

  91. [101]

    Camco: Camera-controllable 3d-consistent image-to-video genera- tion

    Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controllable 3d-consistent image-to-video genera- tion. arXiv preprint arXiv:2406.02509, 2024. 1, 3

    2024 arXiv

  92. [102]

    Magicanimate: Temporally consistent human image animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Han- shu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  93. [103]

    Depth anything: Un- leashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Ji- ashi Feng, and Hengshuang Zhao. Depth anything: Un- leashing the power of large-scale unlabeled data. In CVPR,

  94. [104]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv:2406.09414, 2024. 3

    2024 arXiv

  95. [105]

    Direct-a-video: Customized video generation with user- directed camera movement and object motion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user- directed camera movement and object motion. In ACM SIGGRAPH 2024 Conference Papers, pages 1–12, 2024. 1, 3

    2024

  96. [106]

    Lora-composer: Leverag- ing low-rank adaptation for multi-concept customization in training-free diffusion models

    Yang Yang, Wen Wang, Liang Peng, Chaotian Song, Yao Chen, Hengjia Li, Xiaolong Yang, Qinglin Lu, Deng Cai, Boxi Wu, and Wei Liu. Lora-composer: Leverag- ing low-rank adaptation for multi-concept customization in training-free diffusion models. arXiv preprint arXiv: 2403.11627, 2024. 1

    2024 arXiv

  97. [107]

    Cogvideox: Text-to- video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to- video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 1, 3, 7

    2024 arXiv

  98. [108]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  99. [109]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089, 2023. 1, 3

    2023 arXiv

  100. [110]

    4dgen: Grounded 4d content gener- ation with spatial-temporal consistency

    Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content gener- ation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225, 2023. 1

    2023 arXiv

  101. [111]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2023

  102. [112]

    Efficient video diffusion mod- els via content-frame motion-latent decomposition

    Sihyun Yu, Weili Nie, De-An Huang, Boyi Li, Jinwoo Shin, and Anima Anandkumar. Efficient video diffusion mod- els via content-frame motion-latent decomposition. arXiv preprint arXiv:2403.14148, 2024. 3

    2024 arXiv

  103. [113]

    Viewcrafter: Taming video dif- fusion models for high-fidelity novel view synthesis

    Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video dif- fusion models for high-fidelity novel view synthesis. arXiv preprint arXiv:2409.02048, 2024. 1, 3

    2024 arXiv

  104. [114]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 3

    2023

  105. [115]

    I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023. 1

    2023 arXiv

  106. [116]

    Pia: Your personalized image ani- mator via plug-and-play modules in text-to-image models

    Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image ani- mator via plug-and-play modules in text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7747–7756, 2024. 1

    2024

  107. [117]

    Entropy-driven sampling and training scheme for conditional diffusion gen- eration

    Guangcong Zheng, Shengming Li, Hui Wang, Taiping Yao, Yang Chen, Shouhong Ding, and Xi Li. Entropy-driven sampling and training scheme for conditional diffusion gen- eration. In European Conference on Computer Vision , pages 754–769. Springer, 2022

    2022

  108. [118]

    Layoutdiffusion: Controllable diffusion model for layout-to-image generation

    Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22490–22499, 2023

    2023

  109. [119]

    Cami2v: Camera-controlled image-to-video dif- fusion model

    Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera-controlled image-to-video dif- fusion model. arXiv preprint arXiv:2410.15957, 2024. 1, 3, 7, 8

    2024 arXiv

  110. [120]

    Open-sora: Democratizing efficient video production for all

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404, 2024. 3

    2024 arXiv

  111. [121]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, and Yang You. Open-sora: Democratizing efficient video production for all, 2024

    2024

  112. [122]

    Multi-lora composition for image genera- tion

    Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image genera- tion. arXiv preprint arXiv:2402.16843, 2024. 1, 4

    2024 arXiv

  113. [123]

    Stereo magnification: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018

    2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.