Pith. sign in

REVIEW 4 major objections 4 minor 49 references

Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adapting only the Key embeddings of temporal attention and rerouting U-Net skip connections transfers a reference video's motion without copying its appearance.

desk verdict TAP and AH are clean, well-motivated tweaks that consistently reduce appearance leakage over MotionDirector, but the paper never directly measures whether generated motion matches the reference video, so the headline motion claim is under-supported. read the letter →

arxiv 2501.16714 v2 pith:YELOQH3W submitted 2025-01-28 cs.CV cs.AI

Huijie Liu , Jingyun Wang , Shuai Ma , Jie Hu , Xiaoming Wei , Guoliang Kang This is my paper · ORCID
classification cs.CVcs.AI
keywords text-to-videogenerationmotioncustomizationdiffusionmodelsLoRAadaptationtemporalattentionappearanceleakageU-Netskipconnectionsmotion-appearanceseparation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Motion customization usually means showing a text-to-video model a handful of clips with the same motion and expecting it to reproduce that motion under new text prompts. The problem is 'appearance leakage': the adapted model also copies backgrounds, objects, or identity from the reference clips, so prompts about a sea dragon or a sandwich still come out looking like the shark or pizza in the reference videos. This paper claims the leakage can be removed by two surgical changes: adapt only the Key (or Query/Key) projections of the temporal attention with LoRA, and reroute each U-Net skip connection to start from the non-adapted spatial transformer instead of the adapted temporal transformer. A third trick—using the adapted model only for early denoising steps—keeps late-step appearance generation in the vanilla model. If these claims hold, motion concept transfer no longer has to trade away appearance fidelity.

What carries the argument

The load-bearing machinery is a pair of surgical changes inside a spatial-temporal U-Net, plus an inference-time schedule. TAP (Temporal Attention Purification) restricts the motion LoRA to the Query/Key subspace of temporal attention, so weight updates only re-rank how existing Value embeddings are combined; the paper justifies this with a probing table comparing adaptation of Query, Key, Value, and feed-forward modules. AH (Appearance Highway) moves the origin of each encoder-to-decoder skip connection from the adapted temporal transformer's output to the non-adapted spatial transformer's output, leveraging the known role of skip connections as carriers of high-frequency appearance information; scaling that branch by a factor $\beta$ adjusts how strongly the vanilla appearance dominates. PLI (Phased LoRA Integration) toggles between the adapted and vanilla model depending on the denoising step, on the empirical observation that later steps specialize in fine appearance. Together these components ensure the adapted branch is responsible for motion and the unadapted branch for appearance.

What would settle it

Find one motion concept—for example, a deformation or articulation never present in the pretraining distribution—that cannot be reproduced when only Query/Key are adapted while Value and feed-forward are frozen, yet is faithfully reproduced when Value is also adapted; that single counterexample would show TAP's restriction is not universally sufficient. A more direct check is to train the exact TAP/AH pipeline on reference clips whose appearance and motion are deliberately matched, then swap the appearance in the prompt and measure whether the reference video's background object still appears in more than a trivial fraction of generated videos.

Watch

Extended reading notes

Core claim

The central claim is that a text-to-video diffusion model can learn a new motion from reference clips without encoding the clips' appearance, provided the adaptation is shaped by where motion and appearance actually live in the network. In Temporal Attention Purification (TAP), the temporal LoRA is applied only to the attention projections that determine which Value embeddings get combined—the Query and Key—leaving the pretrained Value embeddings and feed-forward layers untouched, on the assumption that those Values already contain all the visual ingredients a new motion needs. In Appearance Highway (AH), each U-Net skip connection is re-pointed so that the decoder receives its high-frequency appearance shortcut from the non-adapted spatial transformer rather than from the LoRA-adapted temporal transformer, cutting off the path by which leaked appearance would re-enter. A phased LoRA integration schedule then runs the adapted model only during early denoising, when motion is decided, and the vanilla model later, when fine appearance is decided. The paper demonstrates through quantitative metrics and human evaluation that this combination yields videos whose appearance matches the text prompt while motion matches the reference videos, on both one-shot and few-shot setups.

Load-bearing premise

The method rests on the assumption that the pretrained Value embeddings in the temporal attention already contain all the building blocks needed to compose any new motion, so only the Query and Key projections that combine them need to be adapted; the paper supports this with a small probing study rather than a general proof.

Editorial extensions

If this is right

  • A user with only one reference clip can transfer its motion to an arbitrary new subject and scene described by text, without unwanted background objects from the reference video reappearing.
  • Few-shot motion customization (three to five clips per motion category) improves in appearance diversity and motion fidelity, so the approach is useful when a motion concept is defined by several examples.
  • Because only the Key projection of temporal attention is adapted, the learned motion representation is smaller and carries less appearance information, making it easier to share and reuse across prompts.
  • The separation strategy transfers across different text-to-video backbones built on spatial-temporal U-Nets, so it is not tied to one base model.
  • The phased schedule suggests that diffusion-based customization generally should confine adapted modules to early denoising steps, since late steps are where appearance overfitting occurs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same 'reroute the appearance shortcut' recipe might apply to any LoRA-adapted diffusion module that contaminates output appearance, such as personalized image generation or subject-driven video editing; the general principle is to locate which architectural pathway carries appearance and keep it unadapted.
  • The authors' assumption that pretrained Value embeddings are a complete motion dictionary is directly testable: probe with motion concepts far outside the training distribution (fluid dynamics, cloth, articulated machinery) and see whether only Query/Key adaptation still suffices.
  • A cleaner test of appearance leakage than CLIP text similarity would use pairs of reference videos with identical motion but maximally different appearance, then measure how much object/background identity transfers under TAP/AH; this would isolate the separation mechanism from language-alignment confounds.
  • The phased integration idea could combine with other conditioning mechanisms, such as structure or pose control, to reduce their appearance side effects, since it only requires a schedule over denoising steps.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper addresses motion customization in text-to-video diffusion models, building on MotionDirector's dual-path LoRA adaptation. It introduces Temporal Attention Purification (TAP), which restricts temporal LoRA updates to the Key projection so that pretrained Value embeddings are recombined rather than changed, and Appearance Highway (AH), which reroutes U-Net skip connections from the outputs of temporal transformers to the outputs of spatial transformers. A phased LoRA integration (PLI) uses the adapted model for early denoising steps and the vanilla model later. Experiments on TGVE and UCF Sports compare against prior customization and controllable generation methods on automatic and human metrics.

Significance. If the central claim holds, the paper offers a simple, training-light recipe that materially reduces appearance leakage in motion customization, and it is among the few works to explicitly tie architectural choices (attention projections, skip connections) to the motion/appearance separation. The method is clearly specified, the code is released, and the ablations isolate each contribution. However, the motion side of the claim is currently under-evidenced: no automatic metric compares generated motion to reference motion, and the human motion-fidelity vote against MotionDirector is a near tie. Because the method's rationale is precisely that it preserves motion while improving appearance, this gap is consequential.

major comments (4)
  1. [§4.3, Table 3] The abstract and Sec. 5 claim that the method generates 'motion more consistent with the reference videos,' but no reported automatic metric measures agreement with the reference motion. Temporal Consist. is frame-to-frame CLIP coherence, and VBench Average dilutes motion submetrics; Tables 2 and 3 contain no reference-based motion comparison (e.g., optical-flow or trajectory similarity between generated and reference videos). In human evaluation, Motion Fidelity against MotionDirector is 51.5/48.5, a near tie with no confidence intervals or significance test. The motion-consistency half of the central claim is therefore not substantiated.
  2. [§3.2, Table 1] The probing experiment that motivates TAP, and specifically the choice to adapt only K rather than Q or V, is a single-concept study using a motion classifier with no error bars or significance testing. It also uses a proxy 'App. Leak' score computed as CLIP similarity to 'a person,' which is confounded when the generated subject is, e.g., a monkey or a robot. Given that the sufficiency of pretrained Value embeddings is assumed rather than derived, this is thin evidence for an architectural design choice that is central to the method.
  3. [§3.3, Fig. 4] The hidden-state analysis meant to show that AH 'does not disturb motion generation' is circular. The motion classifier is trained on TAP hidden states as positives and vanilla T2V-DM hidden states as negatives; showing that AH hidden states are classified as TAP-like only demonstrates that AH resembles TAP, not that either matches the reference motion. An independent, reference-based measure of generated motion is needed to support the claim that AH preserves motion fidelity.
  4. [§4.4, Tables 4 and 6] The appearance-highway scale β and the phased-LoRA threshold τ are tuned on the same benchmark that is later reported as the final result (β is selected via Table 6, and PLI is ablated only on/off in Table 4 without a sweep over τ). This introduces test-set tuning into the headline numbers and should be addressed with held-out validation or at least a sensitivity analysis for τ.
minor comments (4)
  1. [Table 2 caption] Table 2's caption reads 'few-shot' while Sec. 4.2 and the surrounding text describe it as one-shot; Table 3 is the few-shot evaluation. Please correct the caption.
  2. [§3.4, Eq. (2)] In Eq. (2), α_t and the overbarred α are used without definition, and the value of τ is never stated. The text says 'early steps' use the adapted model while the displayed condition t>τ uses the adapted model; this is consistent if the denoising index t decreases from T to 1, but that convention should be made explicit.
  3. [§1 and §4.4] The acronym is introduced as 'PIL' in Sec. 1 but 'PLI' is used in Sec. 4 and Table 4; please use a single acronym consistently.
  4. [Fig. 4] The bars in Fig. 4 have no error bars and no sample-size information; please specify how the similarities are computed and over how many videos.

Circularity Check

2 steps flagged · score 3.0 of 10

Minor circularity in AH validation: motion is defined by TAP hidden states and β is tuned on the same benchmark; central claim still independently supported by human evaluation.

  1. self definitional [Section 3.3, Investigation of AH (Fig. 4)]
    "We extract optical flows from TAP’s hidden states as positive samples and those from T2V-DM’s hidden states as negative samples to train a motion classifier. Then we use this classifier to evaluate the hidden states from AH. Fig. 4 shows most hidden states from AH can be classified into TAP rather than vanilla T2V-DM in a high accuracy and with high confidence. These results show that utilizing AH may not disturb the motion generation ability of TAP."

    The classifier's 'motion' category is defined by TAP's hidden states, not by an independent measure of reference-motion fidelity. AH is TAP with a rerouted skip connection, so its hidden states are expected to resemble TAP's by construction. The test therefore verifies only that AH retains TAP's internal signature; it cannot independently show that reference motion is preserved. This makes the AH motion-preservation claim self-consistent rather than externally evidenced. The central claim of superior motion fidelity nevertheless also relies on human votes, so this step is supporting, not solely load-bearing.

  2. fitted input called prediction [Section 4.4, Different choices of appearance highway's scale (Table 6)]
    "As shown in Tab. 6, we change the β and evaluate the model's capabilities. When β is small, increasing β enhances the appearance information generated by T2V-DM and improves text alignment and aesthetic score."

    β is selected by sweeping the exact metrics (Text Align, Aesthetic, ViCLIP, VBench) that are later reported as evidence of the method's superiority. In particular, the TAP+AH row of Tab. 4 (Text Align 28.32) corresponds to the β=1.1 row of Tab. 6, so the ablation's improvement is the selected value, not an out-of-sample prediction. This is a mild form of fitting a design parameter on the evaluation benchmark; it does not by itself force the central result, which is also supported by human evaluation.

full rationale

The paper's main contributions are empirical architectural modifications to a motion-LoRA baseline. The headline claims (appearance alignment and motion fidelity) are evaluated against external baselines (MotionDirector, VMC, etc.) with automatic metrics and 1,419 human votes, so the central result is not derived from a fitted quantity. The TAP design is motivated by a stated assumption about Value-embedding sufficiency and is probed with an off-the-shelf action classifier and CLIP-based appearance metrics; that is a design study, not a circular derivation. The two circularity concerns are confined to supporting evidence. First, the Fig. 4 diagnostic defines 'motion' as TAP-like hidden states; since AH is TAP plus a skip reroute, the finding that AH resembles TAP is partly a self-consistency check rather than independent proof that reference motion survives. Second, the scale β of AH is swept on the same Text Align/VBench metrics that later support the ablation's improvement; the reported TAP+AH row corresponds to the best β row, so that portion of the gain is a selected value. The PLI threshold τ is not shown to be tuned, so no specific circular step can be exhibited for it. Neither concern invalidates the full comparison: the final ZeroScope/ModelScope numbers are externally benchmarked and the human motion-fidelity votes, while close against MotionDirector, give independent (if modest) support. Overall the paper is not a circular derivation; the score of 3 reflects these two mild self-referential/tuning issues.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on two substantive domain assumptions: that Value embeddings are motion-sufficient, and that skip connections carry appearance. The free parameters beta and tau are tuned on the benchmark, which reduces the strength of the reported gains. No new physical or conceptual entities are introduced.

free parameters (3)
  • appearance highway scale beta = 1.1
    Scales the spatial-transformer hidden states in the skip connection. Selected as best on the evaluation benchmark (Tab. 6), making part of the reported gain a selection effect.
  • phased LoRA integration threshold tau = not reported
    Number of early denoising steps run with the adapted model. The paper says it is set empirically but does not disclose the value.
  • LoRA rank = not reported
    Rank of the temporal LoRA matrices. Affects the capacity of the adapted Key embeddings; not stated in the provided manuscript.
assumptions (4)
  • domain assumption Pretrained Value embeddings in temporal attention are sufficient components to produce any new motion; only attention weights need reshaping.
    Stated in Sec. 3.2. Validated only by a small probing experiment, not derived or tested across diverse motions.
  • domain assumption Skip connections in the U-Net mainly convey high-frequency appearance information.
    Cites FreeU [28] in Sec. 1 to justify rerouting skip connections in AH. Adopted without in-paper verification.
  • domain assumption The dual-path LoRA framework encodes appearance into temporal LoRAs unless explicitly constrained.
    Motivates the whole method. Observed in MotionDirector's outputs rather than proven as a structural property.
  • standard math The standard epsilon-prediction diffusion loss (Eq. 1) is an appropriate objective for learning motion from reference videos.
    Common assumption in diffusion model fine-tuning; the paper follows MotionDirector in using it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models." pith.science (2026). https://pith.science/paper/YELOQH3W

@misc{pith2026250116714,
  author       = {Pith},
  title        = {Pith review of: Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YELOQH3W}},
  note         = {Machine review of arXiv:2501.16714}
}
read the original abstract

Motion customization aims to adapt the diffusion model (DM) to generate videos with the motion specified by a set of video clips with the same motion concept. To realize this goal, the adaptation of DM should be possible to model the specified motion concept, without compromising the ability to generate diverse appearances. Thus, the key to solving this problem lies in how to separate the motion concept from the appearance in the adaptation process of DM. Typical previous works explore different ways to represent and insert a motion concept into large-scale pretrained text-to-video diffusion models, e.g., learning a motion LoRA, using latent noise residuals, etc. While those methods can encode the motion concept, they also inevitably encode the appearance in the reference videos, resulting in weakened appearance generation capability. In this paper, we follow the typical way to learn a motion LoRA to encode the motion concept, but propose two novel strategies to enhance motion-appearance separation, including temporal attention purification (TAP) and appearance highway (AH). Specifically, we assume that in the temporal attention module, the pretrained Value embeddings are sufficient to serve as basic components needed by producing a new motion. Thus, in TAP, we choose only to reshape the temporal attention with motion LoRAs so that Value embeddings can be reorganized to produce a new motion. Further, in AH, we alter the starting point of each skip connection in U-Net from the output of each temporal attention module to the output of each spatial attention module. Extensive experiments demonstrate that compared to previous works, our method can generate videos with appearance more aligned with the text descriptions and motion more consistent with the reference videos.

Figures

Figures reproduced from arXiv: 2501.16714 by the authors.

Figure 1
Figure 1. Limitations of previous methods. Previous meth [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our method. This figure illustrates our main contributions: (a) Overall inference process. (b) Temporal [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The comparison between the Appearance Highway [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Investigation of AH. (a) shows the similarity be [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results on one-shot. Tune-A-Video [ [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results on few-shot. During training, few videos from UCF Sports dataset [ [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 17 canonical work pages

  1. [1]

    Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. 2023. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation.arXiv preprint arXiv:2304.08477(2023)

    arXiv 2023

  2. [2]

    Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. 2019. Conditional GAN with Discriminative Filter Generation for Text-to- Video Synthesis. InProceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, 1995–20...

    work page doi:10.24963/ijcai.2019/276 2019

  3. [3]

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. 2023. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127(2023)

    arXiv 2023

  4. [4]

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. 2023. Align your latents: High-resolution video synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22563–22575

    work page 2023

  5. [5]

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. 2024. Video generation models as world simulators. (2024). https://openai.com/research/video-generation-models-as-world-simulators

    2024

  6. [6]

    Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. 2023. Pix2video: Video edit- ing using image diffusion. InProceedings of the IEEE/CVF International Conference on Computer Vision. 23206–23217

    work page 2023

  7. [7]

    Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. 2023. Control-a-video: Controllable text-to-video generation with diffusion models.arXiv preprint arXiv:2305.13840(2023)

    arXiv 2023

  8. [8]

    Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ron- neberger. 2016. 3D U-Net: learning dense volumetric segmentation from sparse annotation. InMedical Image Computing and Computer-Assisted Intervention– MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19. Springer, 424–432

    work page 2016

Show all 49 references
  1. [9]

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. 2023. Structure and content-guided video synthesis with diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision. 7346–7356

    2023

  2. [10]

    Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia- Bin Huang, and Devi Parikh. 2022. Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer. arXiv:2204.03638 [cs.CV] https://arxiv. org/abs/2204.03638

    2022 arXiv

  3. [11]

    Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. 2023. Tokenflow: Consis- tent diffusion features for consistent video editing.arXiv preprint arXiv:2307.10373 (2023)

    2023 arXiv

  4. [12]

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. 2023. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725(2023)

    2023 arXiv

  5. [13]

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. 2022. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303(2022)

    2022 arXiv

  6. [14]

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. 2022. Video diffusion models.Advances in Neu- ral Information Processing Systems35 (2022), 8633–8646

    2022

  7. [15]

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. 2022. CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transform- ers. arXiv:2205.15868 [cs.CV] https://arxiv.org/abs/2205.15868

    2022 arXiv

  8. [16]

    Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang

  9. [17]

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. 2024. VBench: Comprehensive Benchmark Suite for Video Generative Mod...

    2024

  10. [18]

    Hyeonho Jeong, Geon Yeong Park, and Jong Chul Ye. 2024. Vmc: Video motion customization using temporal attention adaption for text-to-video diffusion mod- els. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9212–9221

    2024

  11. [19]

    Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Hen- schel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. 2023. Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Gen- erators.arXiv preprint arXiv:2303.13439(2023)

    2023 arXiv

  12. [20]

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. 2023. Pick-a-Pic: An Open Dataset of User Preferences for Text-to- Image Generation

    2023

  13. [21]

    Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. 2021. CCVS: Context- aware Controllable Video Synthesis. arXiv:2107.08037 [cs.CV] https://arxiv.org/ abs/2107.08037

    2021 arXiv

  14. [22]

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. 2023. Fatezero: Fusing attentions for zero-shot text-based video editing. InProceedings of the IEEE/CVF International Conference on Computer Vision. 15932–15942

    2023

  15. [23]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learnin...

    2021

  16. [24]

    Yixuan Ren, Yang Zhou, Jimei Yang, Jing Shi, Difan Liu, Feng Liu, Mingi Kwon, and Abhinav Shrivastava. 2024. Customize-a-video: One-shot motion customization of text-to-video diffusion models.arXiv preprint arXiv:2402.14780(2024)

    2024 arXiv

  17. [25]

    Mikel D Rodriguez, Javed Ahmed, and Mubarak Shah. 2008. Action mach a spatio- temporal maximum average correlation height filter for action recognition. In 2008 IEEE conference on computer vision and pattern recognition. IEEE, 1–8

    2008

  18. [26]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10684–10695

    2022

  19. [27]

    Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MoStGAN-V: Video Generation with Temporal Motion Styles. arXiv:2304.02777 [cs.CV] https://arxiv. org/abs/2304.02777

    2023 arXiv

  20. [28]

    Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. 2024. FreeU: Free Lunch in Diffusion U-Net. InCVPR

    2024

  21. [29]

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. 2022. Make-a-video: Text-to-video generation without text-video data.arXiv preprint arXiv:2209.14792 (2022)

    2022 arXiv

  22. [30]

    Khurram Soomro and Amir R Zamir. 2015. Action recognition in realistic sports videos. InComputer vision in sports. Springer, 181–208

    2015

  23. [31]

    Spencer Sterling. 2023. ZeroScope. https://huggingface.co/cerspense/zeroscope_ v2_576w

    2023

  24. [32]

    Metaxas, and Sergey Tulyakov

    Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov. 2021. A Good Image Generator Is What You Need for High-Resolution Video Synthesis. arXiv:2104.15069 [cs.CV] https://arxiv.org/ abs/2104.15069

    2021 arXiv

  25. [33]

    Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. 2016. Generating videos with scene dynamics.Advances in neural information processing systems29 (2016)

    2016

  26. [34]

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. 2023. Modelscope text-to-video technical report.arXiv preprint arXiv:2308.06571(2023)

    2023 arXiv

  27. [35]

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. 2024. Videocomposer: Composi- tional video synthesis with motion controllability.Advances in Neural Information Processing Systems36 (2024)

    2024

  28. [36]

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. 2023. Internvid: A large-scale video-text dataset for multimodal understanding and generation.arXiv preprint arXiv:2307.06942(2023)

    2023 arXiv

  29. [37]

    Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. 2024. Dreamvideo: Composing your dream videos with customized subject and motion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  30. [38]

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2023. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. InProceedings of the IEEE/CVF International Conf...

    2023

  31. [39]

    Jay Zhangjie Wu, Xiuyu Li, Difei Gao, Zhen Dong, Jinbin Bai, Aishani Singh, Xiaoyu Xiang, Youzeng Li, Zuwei Huang, Yuanxi Sun, Rui He, Feng Hu, Junhua Hu, Hai Huang, Hanyu Zhu, Xu Cheng, Jie Tang, Mike Zheng Shou, Kurt Keutzer, and Forrest Iandola. 2023. CVPR 2023 Text Guided ...

    2023 arXiv

  32. [40]

    Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, and Xiangyu Zhang. 2024. LAMP: Learn A Motion Pattern for Few-Shot Video Generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  33. [41]

    Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. 2023. Bidirectional Cross-Modal Knowledge Exploration for Video Recognition With Pre-Trained Vision-Language Models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2023

  34. [42]

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. 2021. VideoGPT: Video Generation using VQ-VAE and Transformers. arXiv:2104.10157 [cs.CV]

    2021 arXiv

  35. [43]

    Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. 2023. Rerender a video: Zero-shot text-guided video-to-video translation. InSIGGRAPH Asia 2023 Conference Papers. 1–11

    2023

  36. [44]

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. 2023. Show-1: Marrying pixel and la- tent diffusion models for text-to-video generation.arXiv preprint arXiv:2309.15818 (2023)

    2023 arXiv

  37. [45]

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 2023. I2vgen-xl: High-quality image- to-video synthesis via cascaded diffusion models.arXiv preprint arXiv:2311.04145 (2023)

    2023 arXiv

  38. [46]

    Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. 2023. Controlvideo: Training-free controllable text-to-video genera- tion.arXiv preprint arXiv:2305.13077(2023)

    2023 arXiv

  39. [47]

    Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. 2023. Motiondirector: Motion cus- tomization of text-to-video diffusion models.arXiv preprint arXiv:2310.08465 (2023)

    2023 arXiv

  40. [48]

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. 2022. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018(2022). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

    2022 arXiv

  41. [2024]

    Free-bloom: Zero-shot text-to-video generator with llm director and ldm animator.Advances in Neural Information Processing Systems36 (2024)

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.