The reviewed record of science sign in
Pith

arxiv: 2303.12346 · v1 · pith:4U5CC2PN · submitted 2023-03-22 · cs.CV · cs.AI

NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation

Shengming Yin , Chenfei Wu , Huan Yang , Jianfeng Wang , Xiaodong Wang , Minheng Ni , Zhengyuan Yang , Linjie Li
show 8 more authors
Shuguang Liu Fan Yang Jianlong Fu Gong Ming Lijuan Wang Zicheng Liu Houqiang Li Nan Duan
This is my paper

Reviewed by Pithpith:4U5CC2PNopen to challenge →

classification cs.CV cs.AI
keywords longdiffusionvideosgenerationvideoframesmodelextremely
0
0 comments X
read the original abstract

In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation. Most current work generates long videos segment by segment sequentially, which normally leads to the gap between training on short videos and inferring long videos, and the sequential generation is inefficient. Instead, our approach adopts a ``coarse-to-fine'' process, in which the video can be generated in parallel at the same granularity. A global diffusion model is applied to generate the keyframes across the entire time range, and then local diffusion models recursively fill in the content between nearby frames. This simple yet effective strategy allows us to directly train on long videos (3376 frames) to reduce the training-inference gap, and makes it possible to generate all segments in parallel. To evaluate our model, we build FlintstonesHD dataset, a new benchmark for long video generation. Experiments show that our model not only generates high-quality long videos with both global and local coherence, but also decreases the average inference time from 7.55min to 26s (by 94.26\%) at the same hardware setting when generating 1024 frames. The homepage link is \url{https://msra-nuwa.azurewebsites.net/}

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation

    cs.CV 2025-12 conditional novelty 6.0

    Reward Forcing combines EMA-Sink tokens and Rewarded Distribution Matching Distillation to deliver state-of-the-art streaming video generation at 23.1 FPS without copying initial frames.

  2. DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory

    cs.CV 2023-08 unverdicted novelty 6.0

    DragNUWA integrates text, image, and trajectory controls into a diffusion video model using a Trajectory Sampler, Multiscale Fusion, and Adaptive Training to enable fine-grained open-domain video generation.

  3. Matrix-game 2.0: An open-source real-time and streaming interactive world model

    cs.CV 2025-08 unverdicted novelty 5.0

    Matrix-Game 2.0 introduces a scalable data pipeline, action-injection module, and few-step distillation to enable real-time streaming video generation at 25 FPS from game-engine interactions, with open-sourced weights...

  4. Character-Centered Dialogue Generation from Scene-Level Prompts

    cs.CV 2025-05 unverdicted novelty 4.0

    A training-free framework generates expressive, character-grounded dialogue and speech from scene prompts using vision-language encoders, LLMs, and a recursive narrative memory bank for cross-scene consistency.

  5. Scene-Action Prompt Fusion for Coherent Text-to-Video Storytelling

    cs.CV 2025-03 unverdicted novelty 3.0

    A prompt fusion approach combines bidirectional time-weighted latent blending, dynamics-informed prompt weighting via CLIP, and semantic action representations to produce temporally consistent long videos from text wi...