Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Mimir: Improving Video Diffusion Models for Precise Text Understanding

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mimir shows a video diffusion model can gain precise text understanding by fusing a decoder-only LLM with its T5 branch through a token fuser, improving counts, colors, spatial order, and action sequence following while preserving video…

desk verdict A plausible token-fusion recipe for video diffusion, but the reported gains are confounded with a 500k-clip fine-tune; the architecture deserves peer review, not the evaluation as-is. read the letter →

arxiv 2412.03085 v1 pith:TCXUFKAU submitted 2024-12-04 cs.CV

Shuai Tan , Biao Gong , Yutong Feng , Kecheng Zheng , Dandan Zheng , Shuwei Shi , Yujun Shen , Jingdong Chen
show 1 more author
Ming Yang
This is my paper · ORCID
classification cs.CV
keywords text-to-videogenerationvideodiffusionmodelslargelanguagedecoder-onlytransformertokenfusionsemanticstabilizerzeroconvolutioninstructionfollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video diffusion models inherit their text understanding from encoder-style models such as T5, which struggle with counts, colors, spatial relations, and action order. Mimir claims that a decoder-only large language model, Phi-3.5, can be added on top of a pretrained video diffusion transformer without retraining it from scratch, provided the two text branches are fused carefully. The paper's token fuser normalizes and zero-initializes the LLM branch, then sums it with the encoder branch, and adds four instruction-conditioned stabilizer tokens that absorb the volatility of the LLM's generative tokens. On VBench metrics, a user study, and qualitative examples, Mimir reports gains especially in multiple-object and spatial-relationship following while keeping overall video quality at or above the CogVideoX-5B baseline. The central claim is that this fusion lets the model use the video prior and the LLM's reasoning at the same time.

What carries the argument

The load-bearing object is the Token Fuser, which combines two text branches into one conditioning sequence for the video diffusion transformer. It applies a normalization layer plus learnable scale to the decoder-only LLM tokens, then a zero-convolution layer $Z_\beta$, so the LLM branch starts at zero and grows gradually; it applies a residual zero-convolution $Z_\theta$ to the encoder branch, so the T5 features start unchanged. The two are summed as $e=e_\theta+\alpha e_\beta$, and four learnable stabilizer tokens $e_l$ are added to instruction tokens $e_i$ to form $e_s=e_i+e_l$, which is concatenated with $e$. The zero-convolution mechanism is what prevents the direct summation from collapsing training, and the semantic stabilizer is what absorbs the token-to-token volatility of decoder-only generation.

What would settle it

A concrete check is to sweep the mixing weight $\alpha$ in $e=e_\theta+\alpha e_\beta$ over values such as 0.1, 0.5, and 1.0 with the token fuser fixed; if training diverges or the VBench multiple-object and spatial-relationship scores drop sharply at any moderate $\alpha$, the fuser's stability claim is falsified. The paper reports collapse for direct summation but does not report the $\alpha$ values used, so this sweep is the missing control.

Watch

Extended reading notes

Core claim

The paper's central claim is that the feature gap between an encoder text model and a decoder-only LLM can be bridged by a token fuser, so all query and answer tokens of Phi-3.5 can be concatenated with T5 tokens and fed into the diffusion transformer. The fusion is non-destructive: the decoder branch passes through normalization, a learnable scale, and a zero-convolution layer, so at initialization it contributes nothing and the pretrained video model remains in its original semantic space; the encoder branch passes through a residual zero-conv, so it starts as the original T5 features. A semantic stabilizer appends four learnable tokens to four instruction-prompt tokens, anchoring the volatile next-token-prediction features (different answers for the same prompt, e.g. 'old car' vs. 'dilapidated machine') while retaining the LLM's reasoning. Mimir reports that this design yields high-quality videos with precise text comprehension, with the largest gains in multiple-object and spatial-relationship metrics, and that short, coarse prompts can produce results comparable to long, detailed prompts.

Load-bearing premise

The load-bearing premise is that a normalized, zero-convolution-gated sum of T5 and Phi-3.5 tokens keeps the pretrained video transformer trainable, and that four learnable stabilizer tokens can hold back the LLM's token volatility without destroying its reasoning; if either fails, the direct-summation collapse shown in the ablation is the expected outcome.

Editorial extensions

If this is right

  • Adding the decoder-only branch improves the model's ability to follow prompts about multiple objects and spatial relationships, the two VBench dimensions where the reported gains are largest.
  • Short, coarse prompts can produce videos as detailed as long, fine-grained prompts, because the LLM expands the semantics before fusion.
  • The model can handle temporal instructions such as the order of actions, direction of gaze or motion, object transformation, and illumination changes over time.
  • The pretrained video prior is preserved, as Mimir's aesthetic, imaging, and background-consistency scores stay at or above the CogVideoX-5B baseline.
  • The design applies to any diffusion transformer that already uses T5-style encoder tokens, not only the specific baseline tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fuser's recipe is largely size-agnostic: normalization and zero-init target distribution scale, not Phi-3.5-specific features, so a larger or differently instruction-tuned decoder-only model could replace Phi-3.5 with minimal changes.
  • The same stabilizer idea could transfer to text-to-image diffusion models, where counts, colors, and spatial relations are also failure modes; the four instruction tokens could be extended to user-selected attributes such as style or viewpoint.
  • The unstated mixing weight $\alpha$ is a likely sensitivity point; sweeping it and scheduling it during training, for example by warming up the LLM branch, may be the natural way to control how much reasoning the video model absorbs.
  • If the stabilizer tokens are interpreted as learned task prompts, the approach could support test-time control: changing the instruction set or the number of stabilizer tokens may change which semantic dimensions the model prioritizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Mimir, a fine-tuning framework that augments a pretrained video diffusion transformer (CogVideoX-5B) with a decoder-only LLM (Phi-3.5). The method sums the LLM's token embeddings with the existing T5 text embeddings after normalization, a learnable scale, and zero-conv fusion, and appends four learnable semantic stabilizer tokens derived from instruction prompts. It reports large gains on VBench metrics, particularly Multiple Objects and Spatial Relationship, and a user study against several baselines. The central claim is that the proposed token fuser lets the T2V model leverage learned video priors while capitalizing on the LLM's text abilities without retraining from scratch.

Significance. The idea of combining an encoder-based text branch with a decoder-only LLM branch without retraining the vision transformer from scratch is interesting and, if properly validated, could be a lightweight path to stronger text comprehension in video diffusion models. The paper includes progressive ablations, pseudocode, and visualizations, which are useful for understanding the components. However, the empirical isolation of the fusion mechanism is incomplete, and the missing specification of alpha plus the absence of a matched T5-only control prevent the current evidence from supporting the central claim. The circularity burden is minimal because the self-citations are contextual and no benchmark parameter is fitted.

major comments (3)
  1. [Section 3.1 and Table 3] The 'Baseline' row in Table 3 is the original CogVideoX-5B, while every other row is trained on the newly curated 500k-clip recaptioned dataset with additional compute. No T5-only control fine-tuned on the same 500k clips and the same number of steps is reported, so the improvements in Multiple Objects (65.70 to 92.03) and Spatial Relationship (64.86 to 69.17) could be attributable to the new data, recaptioning, and additional training rather than to the token fuser. Please add a T5-only fine-tuned baseline and a row with the LLM branch but no fusion components.
  2. [Equation (3) and Algorithm 1] Equation (3) defines e = e_theta + alpha * e_beta, but alpha is never specified, varied, or reported in any experiment. Algorithm 1 in the supplementary material omits alpha entirely. Since alpha controls the contribution of the decoder-only branch, the reader cannot assess whether the fusion is balanced or whether the result depends on an unstated hyperparameter.
  3. [Table 3, rows 6 and 7] Mimir adds the 'Norm & Scale' component on top of B+Decoder-only+ZeroConv+SS, but the ablation has no row that isolates Norm & Scale (e.g., ZeroConv+Norm without SS, or ZeroConv+Norm+SS). The final Spatial Relationship jump from 70.16 to 78.67 is therefore attributed to an unstated component, and the progressive ablation does not support the claim that all components are necessary.
minor comments (4)
  1. [Section 2.3] The four instruction prompts used for the Semantic Stabilizer are not listed; please provide the exact prompts so the reader can reproduce the method.
  2. [Section 3.1 and Table 2] The user study reports results from 10 participants and no confidence intervals or significance tests; please report error bars and the number of pairwise comparisons so the reader can judge the reliability of the preference percentages.
  3. [Figure 2] The symbolic notation (✓, %) in Figure 2 is not defined in the caption; please clarify what each symbol indicates.
  4. [Section 4] The sentence 'One the other hand' contains a typo ('One' should be 'On').

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Mimir's fuser is validated against external VBench benchmarks; the two self-citations are contextual and never load-bearing.

full rationale

The paper is an empirical engineering contribution, not a derivation. The load-bearing claim that the token fuser lets the T2V model 'fully leverage learned video priors while capitalizing on the text-related capability of LLMs' is supported by progressive ablations (Table 3), external VBench metrics (Table 1), and a user study (Table 2), none of which are inputs to the training procedure. The diffusion objective in Eq. (4) is the standard noise-prediction loss, and no VBench score is fitted or used to define the model; therefore no 'prediction' reduces to a fitted parameter by construction. The two self-citations (Ranni, Ref. [14], and Check Locate Rectify, Ref. [17]) appear only as background in the Introduction and are not used in any equation, ablation rationale, or inference step, so they are not load-bearing. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via a self-citation: the normalization and zero-conv techniques are attributed to external works [51, 36]. The main experimental weakness is that Table 3's Baseline row reproduces the original CogVideoX-5B numbers, so the reported gains from adding the decoder-only branch are not fully separated from the 500k-clip curated recaptioning budget and additional fine-tuning compute. That is a control/confounding concern rather than a circularity concern, because the benchmark numbers are measured outcomes, not inputs that the method is defined to reproduce. Under the specified standards, no circular step can be exhibited with a quote and a concrete reduction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method is an empirical engineering contribution, not a derivation. It relies on several unproven modeling choices: additive fusion of two token types after re-scaling, the semantic stabilizer as a fix for LLM volatility, and VBench plus a 10-person user study as evidence of text understanding. Training parameters such as alpha and the four learnable tokens are not disclosed, so the central result cannot be reproduced from the manuscript alone.

free parameters (4)
  • alpha (decoder-only token weight) = not reported; omitted from Algorithm 1 pseudocode
    Eq. (3) defines e = e_theta + alpha * e_beta and calls alpha a weight, but the paper never states its initial value, schedule, or whether it is learned.
  • learnable scale and bias after normalization = not reported
    Section 2.3 adds a normalization layer followed by a small learnable scale factor and bias to the decoder-only branch to align magnitudes; final values are network parameters, not disclosed.
  • four learnable semantic tokens e_l = not reported; shape [4, 4096]
    Semantic Stabilizer initializes four learnable tokens added to four instruction tokens; their initialization and final values are not described.
  • four instruction prompts = not quoted in paper
    Section 2.2 says four attribute-specific instructions are used, but the exact templates are omitted, making replication harder.
assumptions (4)
  • domain assumption Feature distribution gap between T5 encoder tokens and Phi-3.5 decoder-only tokens is bridgeable by normalization plus zero-conv summation.
    Section 2.3 and Fig. 7 claim distributions align after normalization; no proof that additive fusion preserves semantic content.
  • domain assumption Answer-token volatility from the LLM can be limited by four learnable tokens added to instruction tokens without losing the LLM's reasoning benefits.
    Section 3.3 describes fluctuations; the stabilizer is validated only by ablations and sample videos, not by a mechanism.
  • domain assumption The 500k-clip filtered dataset and VBench metrics are adequate proxies for general text-to-video quality.
    Data pipeline in supplementary Appendix A and metrics in Appendix B; no cross-dataset generalization test.
  • domain assumption A pretrained CogVideoX-5B-style diffusion transformer can be fine-tuned end-to-end to accept fused tokens without catastrophic forgetting.
    Ablation baseline numbers in Table 3 match CogVideoX-5B in Table 1, but the paper never states the initialization or fine-tuning recipe explicitly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mimir: Improving Video Diffusion Models for Precise Text Understanding." pith.science (2026). https://pith.science/paper/TCXUFKAU

@misc{pith2026241203085,
  author       = {Pith},
  title        = {Pith review of: Mimir: Improving Video Diffusion Models for Precise Text Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TCXUFKAU}},
  note         = {Machine review of arXiv:2412.03085}
}
read the original abstract

Text serves as the key control signal in video generation due to its narrative nature. To render text descriptions into video clips, current video diffusion models borrow features from text encoders yet struggle with limited text comprehension. The recent success of large language models (LLMs) showcases the power of decoder-only transformers, which offers three clear benefits for text-to-video (T2V) generation, namely, precise text understanding resulting from the superior scalability, imagination beyond the input text enabled by next token prediction, and flexibility to prioritize user interests through instruction tuning. Nevertheless, the feature distribution gap emerging from the two different text modeling paradigms hinders the direct use of LLMs in established T2V models. This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser to harmonize the outputs from text encoders and LLMs. Such a design allows the T2V model to fully leverage learned video priors while capitalizing on the text-related capability of LLMs. Extensive quantitative and qualitative results demonstrate the effectiveness of Mimir in generating high-quality videos with excellent text comprehension, especially when processing short captions and managing shifting motions. Project page: https://lucaria-academy.github.io/Mimir/

Figures

Figures reproduced from arXiv: 2412.03085 by the authors.

Figure 1
Figure 1. Samples generated by Mimir. Our model demonstrates a powerful spatiotemporal imagination for input text prompts, e.g., (row-3) physically accurate petals, (row-4) the desert with illumination harmonization, which closely match human cognition. Abstract Text serves as the key control signal in video generation due to its narrative nature. To render text descriptions into video clips, current video diffusion models bo… view at source ↗
Figure 2
Figure 2. The core idea of Mimir. Text Encoder is well suited for fine-tuning pre-trained T2V models (✓), however it struggles with limited text comprehension (%). In contrast, Decoder-only LLM excels at precise text understanding (✓), but cannot be directly used in established video generation models since the feature distribution gap and the feature volatility (%) . Therefore, we propose the token fuser in Mimir to harmoniz… view at source ↗
Figure 3
Figure 3. The framework of Mimir. Given a text prompt, we employ a text encoder and a decoder-only large language model to obtain eθ and eβ. Additionally, we add an instruction prompt which, after processing by the decoder-only model, yields the corresponding instruction token ei. See token details in Sec. 2.2. To prevent any convergence issue in training caused by the feature distribution gap of eθ and eβ, the proposed token… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Comparison between CogVideoX-5B with Mimir in T2V, where Mimir generates the vivid stunning moment of rocket launch. from textual inputs to the Vision Transformer model ϵθ from both the encoder and decoder-only branches, we need to balance their contributions throughou…
Figure 5
Figure 5. Figure 5: Mimir demonstrates spatial comprehension and imagination, e.g., quantities, spatial relationships, colors, etc. A puppy looks left and then right. A lion looks right and then left. A bird first turns its head and then flaps its wings. A rabbit first stands up its ears …
Figure 6
Figure 6. Figure 6: Mimir demonstrates temporal comprehension and imagination, e.g., direction, order of motion and appearance / disappearance. 3. Experiments In this section, we comprehensively evaluate our method and provide a detailed analysis of the reasons behind the ef￾fectiveness o…
Figure 7
Figure 7. Figure 7: Visualization by t-SNE: (a) Given 50 prompts, we obtain the corresponding tokens using Encoder branch, Decoder-only branch and their sum, i.e., Mimir. (b) We feed one prompt into Decoder-only branch for 50 times to generate 50 query tokens, answer tokens and final toke…
Figure 8
Figure 8. Figure 8: We present more cases generated by Mimir. in [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: The pipeline for preparing data. 1 [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: The comparison between results with short & course prompts and long & fine prompts. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: More examples in terms of color rendering. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: More examples in terms of absolute & relative position. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: More examples in terms of counting. A cat looks up, then down. A cat looks down, then up. A cat looks up, then down, and up again [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: More examples in terms of action sequence over time. [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: More examples in terms of light changes, showcasing the illumination harmonization over time. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: More examples in terms of object transformation over time. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis

    cs.CV 2025-08 conditional novelty 5.0 of 10

    EDTalk++ disentangles talking-head video into four orthogonal motion banks (mouth, pose, eyes, expression) and drives them from either video or audio inputs.

  2. Animate-X++: Universal Character Image Animation with Dynamic Backgrounds

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Animate-X++ turns cartoon images into pose-driven animations with text-controlled moving backgrounds, claiming state-of-the-art results on a new synthetic anthropomorphic benchmark.

Reference graph

Works this paper leans on

68 extracted references · 27 canonical work pages · cited by 2 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 2, 3, 5

    arXiv 2024

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 8, 2

    arXiv 2023

  3. [3]

    ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers.arXiv preprint arXiv:2211.01324, 2022

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Ait- tala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers.arXiv preprint arXiv:2211.01324, 2022. 8

    arXiv 2022

  4. [4]

    Llm2vec: Large language models are secretly powerful text encoders

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mos- bach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961 ,

    arXiv

  5. [5]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 8

    2023

  6. [6]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 8

    arXiv 2023

  7. [7]

    Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- α: Fast training of diffusion transformer for photorealistic text-to- image synthesis. ArXiv, abs/2310.00426, 2023. 8

    arXiv

  8. [8]

    Pixart- \sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- \sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692, 2024

    arXiv 2024

Show all 68 references
  1. [9]

    Pixart-{\delta}: Fast and controllable image generation with latent consistency models

    Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-{\delta}: Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.05252, 2024. 8

    2024 arXiv

  2. [10]

    Od-vae: An omni-dimensional video compressor for im- proving latent video diffusion model

    Liuhan Chen, Zongjian Li, Bin Lin, Bin Zhu, Qian Wang, Shenghai Yuan, Xing Zhou, Xinghua Cheng, and Li Yuan. Od-vae: An omni-dimensional video compressor for im- proving latent video diffusion model. arXiv preprint arXiv:2409.01199, 2024. 3

    2024 arXiv

  3. [11]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 2

    2021

  4. [12]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machin...

  5. [13]

    Perceptual quality assessment of smartphone photography

    Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3677–3686,

  6. [14]

    Ranni: Taming text-to-image diffusion for accurate instruction following

    Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to-image diffusion for accurate instruction following. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4744–4753, 2024. 2

    2024

  7. [15]

    Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion trans- formers

    Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion trans- formers. arXiv preprint arXiv:2405.05945, 2024. 8

    2024 arXiv

  8. [16]

    Preserve your own correlation: A noise prior for video diffusion models

    Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Comp...

    2023

  9. [17]

    Check locate rectify: A training- free layout calibration system for text-to-image generation

    Biao Gong, Siteng Huang, Yutong Feng, Shiwei Zhang, Yuyuan Li, and Yu Liu. Check locate rectify: A training- free layout calibration system for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6624–6634, 2024. 2

    2024

  10. [18]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 3

    2023 arXiv

  11. [19]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  12. [20]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 8

    2022

  13. [21]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 2, 8

    2022 arXiv

  14. [22]

    Cogvlm2: Visual language mod- els for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language mod- els for image and video understanding. arXiv preprint arXiv:2408.16500, 2024. 2

    2024 arXiv

  15. [23]

    Ella: Equip diffusion models with llm for enhanced semantic alignment

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024. 8

    2024 arXiv

  16. [24]

    T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image genera- 9 tion

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image genera- 9 tion. Advances in Neural Information Processing Systems , 36:78723–78747, 2023. 2

    2023

  17. [25]

    VBench: Com- prehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Com- prehensive benchmark suite for video generative models....

    2024

  18. [26]

    Musiq: Multi-scale image quality transformer

    Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5148–5157, 2021. 2

    2021

  19. [27]

    Videopoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023. 8

    2023 arXiv

  20. [28]

    Open-sora-plan, 2024

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. 4, 5

    2024

  21. [29]

    Common diffusion noise schedules and sample steps are flawed

    Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 5404–5411, 2024. 5

    2024

  22. [30]

    Videofusion: Decomposed diffusion models for high-quality video generation

    Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2023

  23. [31]

    Exploring the role of large language models in prompt encoding for diffusion models

    Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models. arXiv preprint arXiv:2406.11831, 2024. 2, 8

    2024 arXiv

  24. [32]

    SDXL: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 8

    2023 arXiv

  25. [33]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In ICML, pages 8748–8763, 2021. 2, 8

    2021

  26. [34]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2, 3

    2020

  27. [35]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 8

    2022 arXiv

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684–10695, 2022. 2, 3, 5, 8

    2022

  29. [37]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35:36479–36494, 2022. 8

    2022

  30. [38]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 5

    2022 arXiv

  31. [39]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 3

    2015

  32. [40]

    Stable Diffusion 2.0 Release, 2022

    stability.ai. Stable Diffusion 2.0 Release, 2022. 2

    2022

  33. [41]

    Galip: Generative adversarial clips for text-to-image synthesis

    Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu. Galip: Generative adversarial clips for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14214– 14223, 2023. 8

    2023

  34. [42]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 8

    2023 arXiv

  35. [43]

    Internlm: A multilingual language model with progressively enhanced capabilities, 2023

    InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023

    2023

  36. [44]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023 arXiv

  37. [45]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy F...

    2023 arXiv

  38. [46]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9 (11), 2008. 8

    2008

  39. [47]

    Cogvlm: Visual expert for pretrained language models, 2023

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan 10 Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023. 2

    2023

  40. [48]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023. 2

    2023 arXiv

  41. [49]

    Grit: A gener- ative region-to-text transformer for object understanding

    Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A gener- ative region-to-text transformer for object understanding. In European Conference on Computer Vision, pages 207–224. Springer, 2025. 2

    2025

  42. [50]

    Paragraph-to-image generation with information-enriched diffusion model

    Weijia Wu, Zhuang Li, Yefei He, Mike Zheng Shou, Chun- hua Shen, Lele Cheng, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang. Paragraph-to-image generation with information-enriched diffusion model. arXiv preprint arXiv:2311.14284, 2023. 8

    2023 arXiv

  43. [51]

    Sana: Efficient high-resolution image synthesis with linear diffusion transformers

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Yujun Lin, Zhekai Zhang, Muyang Li, Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629 ,

    arXiv

  44. [52]

    Baichuan 2: Open large-scale language models

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. 8

    2023 arXiv

  45. [53]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 3, 4, 5, 8

    2024 arXiv

  46. [54]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024. 8

    2024 arXiv

  47. [55]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2023

  48. [56]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023

    arXiv

  49. [57]

    Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms

    Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolf- gang Macherey, Yanping Huang, David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms. Advances in Neural Information Processing Systems , ...

    2024

  50. [58]

    Bridging different language models and generative vision models for text-to-image generation

    Shihao Zhao, Shaozhe Hao, Bojia Zi, Huaizhe Xu, and Kwan-Yee K Wong. Bridging different language models and generative vision models for text-to-image generation. arXiv preprint arXiv:2403.07860, 2024. 8

    2024 arXiv

  51. [59]

    Cv-vae: A compatible video vae for latent generative video models

    Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models. arXiv preprint arXiv:2405.20279, 2024. 3

    2024 arXiv

  52. [60]

    Open-sora: Democratizing efficient video production for all, march 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, march 2024. URL https://github. com/hpcaitech/Open-Sora, 1(3):4, 2024. 3, 4, 5, 8 11 Mimir: Impro...

    2024

  53. [61]

    Metadata Extraction: Most of important video proper- ties such as length, width, frame rate, frame count, and duration are obtained and saving using FFmpeg

  54. [62]

    • Videos with a motion score of 0, determined using optical flow, are excluded

    Filtering Rules: • Videos with fewer than 65 frames, a duration of less than 1s, or an aspect ratio (width / height) outside the range [1, 2] are excluded. • Videos with a motion score of 0, determined using optical flow, are excluded. Quality Filtration. At this stage, we cal...

  55. [63]

    Quality Metrics: We use OpenCV to calculate the black area percentage, brightness, and black frame rate

  56. [64]

    Input text prompt

    Filtering Rules: • Black area > 0.8, excluding. • Brightness < 0.2, excluding. • Black frame rate > 0.4, excluding. Aesthetic Filtration. At this stage, we filter videos based on aesthetic-related operators. Algorithm 1 Token Fuser # Inputs # Text prompt provided by the user t...

  57. [65]

    Aesthetic Metrics: We use the aesthetic predictor 1 to calculate aesthetic score and OCR coverage

  58. [66]

    containing watermarks

    Filtering Rules: • Aesthetic score < 4.0, excluding. • OCR coverage > 0.1, excluding. Watermark Filtration. At this stage, videos containing watermarks are excluded. Each video is analyzed using QWen2-VL-7B [2] to detect the presence of watermarks. Videos flagged as “containin...

  59. [67]

    Text Quality Metrics: • N-gram 2 repetition rates • Semantic alignment between the video and the gener- ated caption using CLIP Score

  60. [68]

    top”, “ below

    Filtering Rules: • 2-gram repetition > 0.056, excluding. • 5-gram repetition > 0.047, excluding. • 10-gram repetition > 0.045, excluding. • Semantic consistency (CLIP score) < 0.25, excluding. This pipeline ensures the collection of high-quality video clips with accurate capti...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.