TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation

Chen Hou; Cuiling Lan; Hanxin Zhu; Xingrui Wang; Xin Li; Yaosi Hu; Zhibo Chen

REVIEW 4 major objections 6 minor 67 references

TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TIV-Diffusion reports that decomposing the starting frame into object slots and aligning each slot with the caption lets a diffusion model move the right object with less deformation and disappearance, outscoring prior text-image-to-video…

desk verdict Solid TI2V engineering: the slot module reliably helps on standard benchmarks, but the 'precise control' claim needs a direct metric before the paper earns its headline. read the letter →

arxiv 2412.10275 v2 pith:J4IAACQ5 submitted 2024-12-13 cs.CV

Xingrui Wang , Xin Li , Yaosi Hu , Hanxin Zhu , Chen Hou , Cuiling Lan , Zhibo Chen This is my paper · ORCID

classification cs.CV

keywords text-drivenimage-to-videogenerationobject-centricrepresentationSlotAttentiondiffusionmodelautoregressivevideocross-modalalignmentGumbel-Softmax

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TIV-Diffusion sets out to show that text-driven image-to-video generation improves when the model first decomposes the given first frame into separate object slots and aligns the text caption with each slot individually, instead of treating the image as one global conditioning signal. The framework couples an autoregressive diffusion video generator with a Slot Attention encoder: slots extracted from the initial frame are enriched by cross-attending to the caption, and the text-enhanced slots are fed into the U-Net during denoising alongside a fused text-image embedding. On the paper's MNIST and CATER experiments, this object-centric alignment yields better perceptual-quality scores and better text-to-motion consistency than prior TI2V methods, and it specifically reduces the failure modes where the wrong object moves, the motion does not match the text, or an object deforms or vanishes when frames overlap. A sympathetic reader would care because the proposed mechanism is modular and resource-friendly: it achieves these gains with an autoregressive frame-by-frame generator rather than a large full-video model.

What carries the argument

The load-bearing mechanism is the object-centric textual-visual alignment realized by Slot Attention, an attention process that iteratively partitions image features into a fixed set of slots. A Slot Attention encoder, pre-trained by reconstructing the first frame, produces K object slots plus a background slot, each meant to hold one object's attributes. Each slot is then refined by cross-attending to the caption, so words describing a target object are bound to the slot that carries that object's identity; the result is denoted M. The text-enhanced slots are linearly projected and injected into every upsampling residual block of the U-Net through a Gumbel-Softmax selection (Eqs. 7–9), letting the model pick the slot relevant to the current motion instruction while keeping a residual pathway for object identity. In parallel, the caption and image-patch embeddings are fused by cross-attention and applied as SPADE scale-offset modulation (Eq. 2), which is what carries the global appearance and motion trajectory. The two pathways together give the generator both a global sense of which object goes where and an object-level memory that resists deformation and disappearance.

What would settle it

Hold the slot count at five on CATER-GEN-v2 and test captions in scenes with eight objects, then compare whether the object named in the caption is the one that actually moves; if compliance collapses once the object count passes the slot budget, the claim that slots carry object-text alignment fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that object-centric textual-visual alignment—learning a fixed set of object slots from the reference image and aligning each slot to the caption—is what allows a diffusion-based TI2V model to move the correct objects along the described trajectories while keeping those objects intact. Concretely, TIV-Diffusion reports lower Fréchet Inception Distance (FID) and Fréchet Video Distance (FVD) than the strongest prior methods on CATER-GEN-v1 and CATER-GEN-v2 (for example, FID drops from 23.56 to 6.67 and FVD from 34.93 to 16.48 on CATER-GEN-v2), and similar gains on Single and Double Moving MNIST. The ablation without object-centric representations shows the slot pathway is doing the work: on CATER-GEN-v2 FID rises from 6.67 to 10.66 and FVD from 16.48 to 32.72 when the slots are removed. The paper also shows qualitative cases where prior methods move the wrong visually similar object or lose an object during overlap, while TIV-Diffusion keeps the object's attributes available through the slots.

Load-bearing premise

The load-bearing premise is that the grouping network pretrained to reconstruct the first frame with a fixed number of slots really does place each object in its own slot, and that the slot-text correspondence survives even when the scene holds more objects than slots, contains occlusions, or contains objects that look alike.

Editorial extensions

If this is right

If the slot-based alignment is doing what the paper claims, text-image-to-video systems can be built autoregressively without the usual controllability penalty, because object identity is carried forward in slots rather than reconstructed from scratch each frame.
The reported gains mean object overlap during motion need not cause permanent object loss: the slot injection path keeps supplying the object's attributes even when occlusion makes pixels ambiguous in a particular frame.
The approach is dataset-portable in the paper's telling: training a fresh Slot Attention encoder per dataset and keeping the rest of the diffusion pipeline from scratch was enough to show gains on MNIST, CATER, and the more realistic Weizmann, KTH, and Bridge sets.
The slot count is a real design lever: the ablation shows K=5 is optimal on CATER-GEN-v2 and both too-few and too-many slots hurt the scores, so fixing the budget trades object coverage against noise.
Replacing either the SPADE fusion or the slot pathway with a simpler cross-attention option degrades the metrics, which supports the paper's conclusion that the two conditioning routes are complementary rather than redundant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported metrics mainly measure perceptual fidelity and distribution distance, not whether the text-referenced object is the one that moved; a direct trajectory-compliance metric computed against CATER's known object positions would be a stronger test of the paper's central controllability claim.
Because slots are extracted from the first frame only, the approach implicitly assumes the objects to be moved are present and identifiable at t=0; objects that enter later, or slots that should update as appearance changes, lie outside the current design.
The fixed slot budget is a capacity bottleneck for open-world use: CATER-GEN-v2 contains 3 to 8 objects while the tuned K is 5, so scenes with more objects require sharing or dropping slots; a dynamic slot-count mechanism would be the obvious follow-up.
The same per-slot text alignment could be lifted into other conditional U-Net pipelines, such as image editing or multi-object animation, wherever instance identity must survive cross-attention with language.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's a quick take on TIV-Diffusion. It's a competent, well-put-together TI2V paper, and the object-centric slot module clearly helps on the benchmarks they actually use. But the paper's headline claim about 'precise control' over object movement is more asserted than measured.

The new thing is the specific integration: Slot Attention pretrained by reconstruction, per-slot cross-attention with the caption, Gumbel-Softmax slot selection, and SPADE scale-offset modulation inside an autoregressive diffusion generator. Each piece is familiar, but I don't know a prior TI2V paper that puts them together this way, and the ablations consistently show the slot module improves FID/FVD across CATER-v1/v2 and the MNIST variants. The qualitative comparisons, especially the occlusion cases in Figs. 4 and 6, show real differences from MAGE. So the engineering contribution is genuine.

The soft spots are mostly about evaluation. First, the 'precise control' claim is not directly quantified. FID/FVD/SSIM/LPIPS are aggregate similarity scores; they don't tell you whether the cone moved to the right coordinate or whether the right object moved. The supplementary has a controllability evaluation (Figs. 14-15) but it's qualitative. A scalar metric for text-to-object alignment would make the central claim testable. Second, the MNIST 'SOTA' numbers are on datasets the authors modified to enforce uniform motion. They fairly re-train/reproduce MAGE on the modified data, so the comparison is consistent, but the modified benchmarks are not the original ones, and the reader should weigh the CATER results more heavily. Third, no error bars and no code release. Finally, the slot-count ablation shows K=5 over K=11 is a small margin (FID 6.67 vs 6.81), so the 'each slot corresponds to an object' story is plausible but not strongly pinned down by that table.

None of this is load-bearing. The internal ablation (with vs without slots) is consistently positive, and that's the core empirical claim. The paper is a solid advance for the TI2V subfield, and it deserves a serious referee. My advice to the editor: send it to review, but ask the authors for a direct controllability metric, error bars, and code/checkpoints. If they can provide that, the paper is a nice contribution.

Referee Report

4 major / 6 minor

Summary. The paper proposes TIV-Diffusion, an autoregressive diffusion framework for text-driven image-to-video generation (TI2V). The model fuses text and image embeddings via cross-attention and injects them through SPADE modulation, and adds an object-centric module that extracts slots from the first frame with a Slot Attention encoder, augments each slot with text via cross-attention, and adaptively incorporates slot features into the U-Net through Gumbel-Softmax selection. Experiments are conducted on modified MNIST variants, CATER-GEN-v1/v2, and the Bridge dataset, reporting state-of-the-art FID/FVD and improved qualitative results for object preservation and motion alignment.

Significance. If the claims hold, the object-centric conditioning mechanism could be a useful contribution to the TI2V literature, targeting a real failure mode (object disappearance and deformation during text-specified motion). The paper's ablations are a strength: Tables 2, 5, 6, and 7 consistently show that adding the slot module and its associated components improves quantitative scores across MNIST, CATER, and Bridge, which supports the mechanism's practical value. However, the paper's central claim of 'precise control' and improved 'text-to-object motion alignment' is not directly quantified; the reported metrics are global video-quality scores, and the qualitative controllability evaluation in Supplementary A.3 is illustrative rather than measured. The paper also does not provide a public code release or detailed reproducibility statement, which limits verification of the training and evaluation pipeline.

major comments (4)

[Sec. 4.3, Tables 1 and 2] The central claim that TIV-Diffusion provides 'precise control' and improved text-to-object motion alignment is not supported by the quantitative metrics. FID, FVD, SSIM, PSNR, and LPIPS are global distributional or similarity measures; they do not test whether the object named in the caption is the one that moves, whether the motion direction and magnitude match the caption, or whether the object's identity is preserved under occlusion. The qualitative figures (Figs. 4-7) and the controllability examples in Supplementary A.3 (Figs. 14-15) are selected successes and do not provide an error rate. For CATER datasets, ground-truth object trajectories and action labels are available; please add a quantitative controllability metric, e.g., top-1 accuracy of 'which object moved' and a trajectory-matching score against the caption, or report the success rate of generating the commanded action over the full test set. Without such a metric, the paper's main thesis remains untested.
[Sec. 3.4, Eqs. (4)-(10), Table 4] The proposed 'object-centric textual-visual alignment' does not contain any mechanism that ties a slot to a specific noun phrase or to a semantically meaningful object. The Slot Attention encoder is pre-trained only by reconstructing the initial frame, Eq. (4), with no object supervision. Then Eq. (6) is a standard cross-attention in which each slot attends to the entire text caption, and the final training loss, Eq. (10), is a pure frame-reconstruction loss. Therefore, the correspondence between slots and objects is assumed, not learned or enforced. This assumption is especially problematic for CATER-GEN-v2, which contains 3 to 8 objects, while the slot budget is K+1=6 (and K=5 is chosen as optimal in Table 4); with K=3, the same FID of 6.67 is obtained, further weakening the claim that the slot count reflects object count. Please provide explicit evidence of slot-to-object correspondence (e.g., slot-to-mask IoU on CATER), a slot-to-text alignment metric, or add a training objective that supervises the claimed alignment.
[Sec. 4.1, Table 1] The 'state-of-the-art on existing datasets' claim is compromised by the modification of the three MNIST datasets. The paper states that 'we slightly modify these three datasets' to enforce uniform motion, and MAGE is reproduced on the modified versions. However, the original MNIST benchmarks are the standard evaluation, and the modification changes the task distribution (e.g., removing speed irregularities at boundaries). The comparison with TVP on the Single dataset uses results from the original paper, so it is not apples-to-apples with the modified setting. Please either report results on the original unmodified MNIST datasets, or explicitly state in the abstract and conclusion that the reported state-of-the-art applies only to the modified versions. At minimum, the quantitative comparison on MNIST should include results on the original datasets for both MAGE and TIV-Diffusion.
[Sec. 4.3 and Supplementary A.5, Tables 3 and 4] The hyperparameters K (number of slots) and T (denoising timesteps) appear to be selected on the evaluation sets: Table 4 selects K on CATER-GEN-v2, and Table 3 selects T on CATER-GEN-v1. This is test-set tuning and can inflate the reported numbers. The FID differences in Table 4 are also very small (6.67 vs 6.67 vs 6.72 vs 6.81), so the 'optimal' K=5 is not robustly justified. Please either fix K and T before seeing the test set, or use a held-out validation split to select them, and report the validation-based choice. This would also address the concern that the slot budget does not match the variable object counts in CATER-GEN-v2.

minor comments (6)

[Abstract] There is a typo: 'TIV-Diffuion' should be 'TIV-Diffusion'.
[Sec. 4.3 heading] The heading 'Quantitive Results' should read 'Quantitative Results'.
[Fig. 2 and Sec. 3.4] The module name is misspelled as 'Object Distanglement Fusion Module'; it should be 'Object Disentanglement Fusion Module'.
[Supplementary A.5, Tables 6 and 7] The captions and text for Tables 6 and 7 are inconsistent. Table 6's caption says 'Slot attention encoder' but the text describes replacing the Image Encoder with a convolutional layer and labels the row 'Ours Conv'; Table 7's caption says 'Ablation results of ConvGRU' but the text says the ConvGRU is replaced with the Slot Attention encoder. Please align the captions, text, and row labels.
[Eq. (7)-(9)] The notation for the Gumbel-Softmax selection is under-specified: the normalization axis is stated, but the ranges of indices i, j, and the role of the straight-through estimator in Eq. (8) during inference are not clarified. Please state explicitly whether Gumbel noise is used at inference time or only during training.
[General reproducibility] The paper does not provide a code repository, data splits for the real-world datasets (Weizmann, KTH, Bridge), or seeds. Adding these would substantially improve reproducibility, especially given the dataset modifications in Sec. 4.1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the object-centric module is trained with reconstruction and video losses and evaluated against external baselines; self-citations are background only.

full rationale

The paper's derivation chain is empirical rather than analytic: the Slot Attention encoder is pre-trained with image reconstruction (Eq. 4), slots are augmented with caption features through cross-attention (Eq. 6), slot selection is trained with Gumbel-Softmax and the video diffusion loss (Eqs. 7-10), and the overall method is compared against MAGE, MAGE+, Seer, and TiV-ODE on external datasets plus a with/without ablation (Table 2). No equation in the paper defines a predicted quantity as the same fitted input by construction. The slot-count choice K=5 in Table 4 is hyperparameter selection on evaluation data, which is a soundness concern about tuning, not circularity: the reported FID improvement does not mechanically follow from the chosen K. The paper cites several works by its own authors (e.g., Li et al. 2023; Wang et al. 2024; Ren et al. 2025), but these appear in the preliminary and related-work context and are not load-bearing for the central claim. The gap between the claimed text-to-object motion alignment and the aggregate FID/FVD metrics used for evaluation is an evidence-strength weakness, not a self-referential reduction. Under the required standard of exhibiting a specific equation-to-equation or fit-to-prediction identity, no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard machine-learning assumptions of the diffusion framework, the Slot Attention reconstruction objective, cross-modal alignment via attention, and the validity of FID/FVD as quality proxies. The main hand-tuned quantities are the slot budget K, the denoising timesteps, and dataset-specific slot dimensions; no new physical or ontological entities are introduced.

free parameters (3)

Number of slots K = 5 for CATER-v2 (plus 1 background slot), 1 for Single MNIST, 2 for Double/Modified MNIST
Chosen by comparing FID/PSNR on CATER-GEN-v2 (Table 4); the paper reports K=5 optimal even though CATER-GEN-v2 has 3 to 8 objects, so the slot budget is a hand-tuned quantity that affects alignment.
Denoising timesteps T = 1600 in reported main results; Table 3 also shows T=250, 500, 1000
Selected empirically; Table 3 shows T=1600 gives the best PSNR and FID on CATER-v1, and the main tables report results in this regime.
Slot dimension Dslots = 512 for MNIST, 128 for CATER
Dataset-specific architecture choices stated in Sec. A.4; they change the capacity of the object representations that carry the central mechanism.

assumptions (4)

domain assumption Slot Attention trained with image reconstruction produces disentangled object slots (Eq. 4).
Sec. 3.4 relies on reconstruction loss to force compositional structure; this is a standard but unproven property of Slot Attention on these datasets.
domain assumption Cross-attention between slots and text can bind textual motion words to the correct object slots (Eq. 6).
The model assumes the learned embedding spaces of text and slots are aligned enough for per-slot attention to select the intended target object.
domain assumption Scale-offset (SPADE) modulation can carry fused text-image information into the diffusion U-Net (Eq. 2).
Sec. 3.3 adopts SPADE without a proof that this conditioning channel is sufficient to control object motion trajectories.
domain assumption PSNR, SSIM, LPIPS, FID, and FVD measure the subjective quality and text-image alignment the paper claims.
Sec. 4.1 uses these metrics without a user study; the abstract's 'subjective quality' claim is not directly measured.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation." pith.science (2026). https://pith.science/paper/J4IAACQ5

@misc{pith2026241210275,
  author       = {Pith},
  title        = {Pith review of: TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J4IAACQ5}},
  note         = {Machine review of arXiv:2412.10275}
}

read the original abstract

Text-driven Image to Video Generation (TI2V) aims to generate controllable video given the first frame and corresponding textual description. The primary challenges of this task lie in two parts: (i) how to identify the target objects and ensure the consistency between the movement trajectory and the textual description. (ii) how to improve the subjective quality of generated videos. To tackle the above challenges, we propose a new diffusion-based TI2V framework, termed TIV-Diffusion, via object-centric textual-visual alignment, intending to achieve precise control and high-quality video generation based on textual-described motion for different objects. Concretely, we enable our TIV-Diffuion model to perceive the textual-described objects and their motion trajectory by incorporating the fused textual and visual knowledge through scale-offset modulation. Moreover, to mitigate the problems of object disappearance and misaligned objects and motion, we introduce an object-centric textual-visual alignment module, which reduces the risk of misaligned objects/motion by decoupling the objects in the reference image and aligning textual features with each object individually. Based on the above innovations, our TIV-Diffusion achieves state-of-the-art high-quality video generation compared with existing TI2V methods.

Figures

Figures reproduced from arXiv: 2412.10275 by the authors.

**Figure 2.** Illustration of the TIV-Diffusion framework. Given an image and its corresponding text caption, TIV-Diffusion can [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Object Distanglement Fusion Module (ODFM). [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (28 more)

**Figure 4.** Figure 4: Comparison Results on Double Moving MNIST. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 2.** Figure 2: F m+1 Unet = F m Unet + PK+1 j=1 Aˆm i,jWvM˜ m j PK+1 j=1 Aˆm i,j . (9) Finally, Eq. 3 is modified to: L(θ) = Et X N n=1 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png]

**Figure 5.** Figure 5: Comparison Results on CATER-GEN-v2. We propose object disentanglement for enhanced alignment between text [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Comparison Results on CATER-GEN-v1. We can observe the object’s disappearance or deformation as the video [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Samples generated from CATER-GEN-v2. The initial image on the far left is provided by the user, and above the [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 9.** Figure 9: Comparison results of “Ours” and “Ours w/o”. [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

**Figure 10.** Figure 10: For CATER-GEN-v1, the coordinate system de [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

**Figure 11.** Figure 11: Object disentanglement results on MNIST datasets [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

**Figure 12.** Figure 12: Object disentanglement results on CATER-GEN [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]

**Figure 13.** Figure 13: Object disentanglement results on CATER-GEN [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]

**Figure 26.** Figure 26: Qualitative Results of Seer on CATER-GEN-v2 To illustrate the improvements in video quality of our method compared to generating all frames at once, we present the qualitative results of Seer on CATER-GEN-v2 in [PITH_FULL_IMAGE:figures/full_fig_p012_26.png]

**Figure 14.** Figure 14: Samples generated from CATER-GEN-v1 for controllability evaluation. The initial image and text caption are the [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]

**Figure 15.** Figure 15: Samples generated from CATER-GEN-v1 for controllability evaluation. The initial image and speed are the same, [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]

**Figure 16.** Figure 16: We similarly explore the offset and observe that it [PITH_FULL_IMAGE:figures/full_fig_p014_16.png]

**Figure 16.** Figure 16: How scale in SPADE affects object movement. [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]

**Figure 17.** Figure 17: Qualitative Results of TIV-Diffusion on Weizmann Dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_17.png]

**Figure 18.** Figure 18: Qualitative Results of TIV-Diffusion on KTH Dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_18.png]

**Figure 19.** Figure 19: Comparison Results on Bridge [PITH_FULL_IMAGE:figures/full_fig_p016_19.png]

**Figure 20.** Figure 20: Samples generated from the Bridge dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_20.png]

**Figure 21.** Figure 21: Additional samples generated from CATER-GEN-v2. [PITH_FULL_IMAGE:figures/full_fig_p018_21.png]

**Figure 22.** Figure 22: Additional samples generated from CATER-GEN-v1. [PITH_FULL_IMAGE:figures/full_fig_p019_22.png]

**Figure 23.** Figure 23: Additional samples generated from MNIST datasets. [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]

**Figure 24.** Figure 24: Additional comparison results on CATER-GEN-v2. [PITH_FULL_IMAGE:figures/full_fig_p021_24.png]

**Figure 25.** Figure 25: Additional comparison results on CATER-GEN-v1. [PITH_FULL_IMAGE:figures/full_fig_p022_25.png]

**Figure 26.** Figure 26: Additional comparison results on Double Moving MNIST. [PITH_FULL_IMAGE:figures/full_fig_p022_26.png]

**Figure 27.** Figure 27: Qualitative results of Seer on CATER-GEN-v2. [PITH_FULL_IMAGE:figures/full_fig_p023_27.png]

**Figure 28.** Figure 28: Additional generated samples when initial occlusion exists. [PITH_FULL_IMAGE:figures/full_fig_p023_28.png]

**Figure 29.** Figure 29: Seer’s generation results with initial occlusion. [PITH_FULL_IMAGE:figures/full_fig_p024_29.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 31 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Aldausari, N.; Sowmya, A.; Marcus, N.; and Mohammadi, G. 2022. Video generative adversarial networks: a review. ACM CSUR, 55(2): 1--25

work page 2022
[4]

An, J.; Zhang, S.; Yang, H.; Gupta, S.; Huang, J.-B.; Luo, J.; and Yin, X. 2023. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.08477

arXiv 2023
[5]

Ballas, N.; Yao, L.; Pal, C.; and Courville, A. 2015. Delving deeper into convolutional networks for learning video representations. arXiv preprint arXiv:1511.06432

arXiv 2015
[6]

W.; Fidler, S.; and Kreis, K

Blattmann, A.; Rombach, R.; Ling, H.; Dockhorn, T.; Kim, S. W.; Fidler, S.; and Kreis, K. 2023. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 22563--22575

work page 2023
[7]

Chang, Z.; Zhang, X.; Wang, S.; Ma, S.; Ye, Y.; Xinguang, X.; and Gao, W. 2021. Mau: A motion-aware unit for video prediction and beyond. NeurIPS, 34: 26950--26962

work page 2021
[8]

Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

arXiv 2020

Show all 67 references

[9]

Dosovitskiy, A.; and Brox, T. 2016. Generating images with perceptual similarity metrics based on deep networks. NeurIPS, 29

2016
[10]

Ebert, F.; Yang, Y.; Schmeckpeper, K.; Bucher, B.; Georgakis, G.; Daniilidis, K.; Finn, C.; and Levine, S. 2021. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396

2021 arXiv
[11]

Y.; and Bell, S

Fu, T.-J.; Yu, L.; Zhang, N.; Fu, C.-Y.; Su, J.-C.; Wang, W. Y.; and Bell, S. 2023. Tell me what happened: Unifying text-guided video completion via multimodal masked video generation. In CVPR, 10681--10692

2023
[12]

Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. NeurIPS, 27

2014
[13]

Gorelick, L.; Blank, M.; Shechtman, E.; Irani, M.; and Basri, R. 2007. Actions as space-time shapes. TPAMI, 29(12): 2247--2253

2007
[14]

Gu, X.; Wen, C.; Song, J.; and Gao, Y. 2023. Seer: Language Instructed Video Prediction with Latent Diffusion Models. arXiv preprint arXiv:2303.14897

2023 arXiv
[15]

Gupta, A.; Tian, S.; Zhang, Y.; Wu, J.; Mart \' n-Mart \' n, R.; and Fei-Fei, L. 2022. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206.11894

2022 arXiv
[16]

He, Y.; Yang, T.; Zhang, Y.; Shan, Y.; and Chen, Q. 2022. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221

2022 arXiv
[17]

Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 30

2017
[18]

P.; Poole, B.; Norouzi, M.; Fleet, D

Ho, J.; Chan, W.; Saharia, C.; Whang, J.; Gao, R.; Gritsenko, A.; Kingma, D. P.; Poole, B.; Norouzi, M.; Fleet, D. J.; et al. 2022. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303

2022 arXiv
[19]

Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. NeurIPS, 33: 6840--6851

2020
[20]

Hong, W.; Ding, M.; Zheng, W.; Liu, X.; and Tang, J. 2022. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868

2022 arXiv
[21]

H \"o ppe, T.; Mehrjou, A.; Bauer, S.; Nielsen, D.; and Dittadi, A. 2022. Diffusion models for video prediction and infilling. arXiv preprint arXiv:2206.07696

2022 arXiv
[22]

Hu, Y.; Luo, C.; and Chen, Z. 2022. Make it move: controllable image-to-video generation with text descriptions. In CVPR, 18219--18228

2022
[23]

Hu, Y.; Luo, C.; and Chen, Z. 2023. A Benchmark for Controllable Text-Image-to-Video Generation. TMM

2023
[24]

Huynh-Thu, Q.; and Ghanbari, M. 2008. Scope of validity of PSNR in image/video quality assessment. Electronics letters, 44(13): 800--801

2008
[25]

Jang, E.; Gu, S.; and Poole, B. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144

2016 arXiv
[26]

Jiang, J.; Deng, F.; Singh, G.; and Ahn, S. 2023. Object-centric slot diffusion. arXiv preprint arXiv:2303.10834

2023 arXiv
[27]

S.; Henderson, D.; Howard, R

LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; and Jackel, L. D. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4): 541--551

1989
[28]

G.; Choi, D

Lee, S.; Kim, H. G.; Choi, D. H.; Kim, H.-I.; and Ro, Y. M. 2021. Video prediction recalling long-term motion context via memory alignment learning. In CVPR, 3054--3063

2021
[29]

Li, X.; Ren, Y.; Jin, X.; Lan, C.; Wang, X.; Zeng, W.; Wang, X.; and Chen, Z. 2023. Diffusion Models for Image Restoration and Enhancement--A Comprehensive Survey. arXiv preprint arXiv:2308.09388

2023
[30]

Locatello, F.; Weissenborn, D.; Unterthiner, T.; Mahendran, A.; Heigold, G.; Uszkoreit, J.; Dosovitskiy, A.; and Kipf, T. 2020. Object-centric learning with slot attention. NeurIPS, 33: 11525--11538

2020
[31]

Mei, K.; and Patel, V. 2023. Vidm: Video implicit diffusion models. In AAAI, volume 37, 9117--9125

2023
[32]

Mittal, G.; Marwah, T.; and Balasubramanian, V. N. 2017. Sync-draw: Automatic video generation using deep recurrent attentive architectures. In ACM MM, 1096--1104

2017
[33]

X.; and Min, M

Ni, H.; Shi, C.; Li, K.; Huang, S. X.; and Min, M. R. 2023. Conditional Image-to-Video Generation with Latent Flow Diffusion Models. In CVPR, 18444--18455

2023
[34]

Park, T.; Liu, M.-Y.; Wang, T.-C.; and Zhu, J.-Y. 2019. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2337--2346

2019
[35]

W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In ICML, 8748--8763. PMLR

2021
[36]

Ren, Y.; Li, X.; Li, B.; Wang, X.; Guo, M.; Zhao, S.; Zhang, L.; and Chen, Z. 2025. Moe-diffir: Task-customized diffusion priors for universal compressed image restoration. In ECCV, 116--134. Springer

2025
[37]

Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In CVPR, 10684--10695

2022
[38]

Ronneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 234--241. Springer

2015
[39]

Saito, M.; Matsumoto, E.; and Saito, S. 2017. Temporal generative adversarial nets with singular value clipping. In ICCV, 2830--2839

2017
[40]

Schuldt, C.; Laptev, I.; and Caputo, B. 2004. Recognizing human actions: a local SVM approach. In ICPR, volume 3, 32--36. IEEE

2004
[41]

Shi, X.; Chen, Z.; Wang, H.; Yeung, D.-Y.; Wong, W.-K.; and Woo, W.-c. 2015. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. NeurIPS, 28

2015
[42]

Simonyan, K.; and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

2014 arXiv
[43]

Singer, U.; Polyak, A.; Hayes, T.; Yin, X.; An, J.; Zhang, S.; Hu, Q.; Yang, H.; Ashual, O.; Gafni, O.; et al. 2022. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792

2022 arXiv
[44]

Song, J.; Meng, C.; and Ermon, S. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502

2020 arXiv
[45]

Song, X.; Chen, J.; Zhu, B.; and Jiang, Y.-G. 2022. Text-driven Video Prediction. arXiv preprint arXiv:2210.02872

2022 arXiv
[46]

Sun, M.; Wang, W.; Zhu, X.; and Liu, J. 2023. MOSO: Decomposing MOtion, Scene and Object for Video Prediction. In CVPR, 18727--18737

2023
[47]

Tan, C.; Gao, Z.; Wu, L.; Xu, Y.; Xia, J.; Li, S.; and Li, S. Z. 2023 a . Temporal attention unit: Towards efficient spatiotemporal predictive learning. In CVPR, 18770--18782

2023
[48]

Tan, C.; Li, S.; Gao, Z.; Guan, W.; Wang, Z.; Liu, Z.; Wu, L.; and Li, S. Z. 2023 b . OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning. arXiv preprint arXiv:2306.11249

2023 arXiv
[49]

Tulyakov, S.; Liu, M.-Y.; Yang, X.; and Kautz, J. 2018. Mocogan: Decomposing motion and content for video generation. In CVPR, 1526--1535

2018
[50]

Unterthiner, T.; Van Steenkiste, S.; Kurach, K.; Marinier, R.; Michalski, M.; and Gelly, S. 2018. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717

2018 arXiv
[51]

Van Den Oord, A.; Vinyals, O.; et al. 2017. Neural discrete representation learning. NeurIPS, 30

2017
[52]

N.; Kaiser, .; and Polosukhin, I

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. NeurIPS, 30

2017
[53]

Voleti, V.; Jolicoeur-Martineau, A.; and Pal, C. 2022. MCVD-masked conditional video diffusion for prediction, generation, and interpolation. NeurIPS, 35: 23371--23385

2022
[54]

Vondrick, C.; Pirsiavash, H.; and Torralba, A. 2016. Generating videos with scene dynamics. NeurIPS, 29

2016
[55]

Wang, J.; Yuan, H.; Chen, D.; Zhang, Y.; Wang, X.; and Zhang, S. 2023. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571

2023 arXiv
[56]

Wang, X.; Li, X.; and Chen, Z. 2024. CoNo: Consistency Noise Injection for Tuning-free Long Video Diffusion. arXiv preprint arXiv:2406.05082

2024 arXiv
[57]

C.; Sheikh, H

Wang, Z.; Bovik, A. C.; Sheikh, H. R.; and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. TIP, 13(4): 600--612

2004
[58]

a ckstr \

Weissenborn, D.; T \"a ckstr \"o m, O.; and Uszkoreit, J. 2019. Scaling autoregressive video models. arXiv preprint arXiv:1906.02634

2019 arXiv
[59]

Wu, C.; Huang, L.; Zhang, Q.; Li, B.; Ji, L.; Yang, F.; Sapiro, G.; and Duan, N. 2021. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806

2021 arXiv
[60]

Wu, C.; Liang, J.; Ji, L.; Yang, F.; Fang, Y.; Jiang, D.; and Duan, N. 2022 a . N \"u wa: Visual synthesis pre-training for neural visual world creation. In ECCV, 720--736. Springer

2022
[61]

Z.; Ge, Y.; Wang, X.; Lei, W.; Gu, Y.; Hsu, W.; Shan, Y.; Qie, X.; and Shou, M

Wu, J. Z.; Ge, Y.; Wang, X.; Lei, W.; Gu, Y.; Hsu, W.; Shan, Y.; Qie, X.; and Shou, M. Z. 2022 b . Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565

2022 arXiv
[62]

Xing, J.; Xia, M.; Zhang, Y.; Chen, H.; Wang, X.; Wong, T.-T.; and Shan, Y. 2023 a . Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190

2023 arXiv
[63]

Xing, Z.; Dai, Q.; Hu, H.; Wu, Z.; and Jiang, Y.-G. 2023 b . SimDA: Simple Diffusion Adapter for Efficient Video Generation. arXiv preprint arXiv:2308.09710

2023 arXiv
[64]

Xu, J.; De Mello, S.; Liu, S.; Byeon, W.; Breuel, T.; Kautz, J.; and Wang, X. 2022. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 18134--18144

2022
[65]

Xu, Y.; Li, N.; Goel, A.; Guo, Z.; Yao, Z.; Kasaei, H.; Kasaei, M.; and Li, Z. 2023. Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE. arXiv preprint arXiv:2303.05323

2023 arXiv
[66]

Yang, R.; Srivastava, P.; and Mandt, S. 2022. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203.09481

2022 arXiv
[67]

Zhou, D.; Wang, W.; Yan, H.; Lv, W.; Zhu, Y.; and Feng, J. 2022. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018

2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Aldausari, N.; Sowmya, A.; Marcus, N.; and Mohammadi, G. 2022. Video generative adversarial networks: a review. ACM CSUR, 55(2): 1--25

work page 2022

[4] [4]

An, J.; Zhang, S.; Yang, H.; Gupta, S.; Huang, J.-B.; Luo, J.; and Yin, X. 2023. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.08477

arXiv 2023

[5] [5]

Ballas, N.; Yao, L.; Pal, C.; and Courville, A. 2015. Delving deeper into convolutional networks for learning video representations. arXiv preprint arXiv:1511.06432

arXiv 2015

[6] [6]

W.; Fidler, S.; and Kreis, K

Blattmann, A.; Rombach, R.; Ling, H.; Dockhorn, T.; Kim, S. W.; Fidler, S.; and Kreis, K. 2023. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 22563--22575

work page 2023

[7] [7]

Chang, Z.; Zhang, X.; Wang, S.; Ma, S.; Ye, Y.; Xinguang, X.; and Gao, W. 2021. Mau: A motion-aware unit for video prediction and beyond. NeurIPS, 34: 26950--26962

work page 2021

[8] [8]

Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

arXiv 2020