Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Dynamic-I2V: Exploring Image-to-Video Generation Models via Multimodal LLM

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that adding a multimodal LLM's visual and textual features to an image-to-video diffusion transformer yields dramatically more dynamic videos, and that a new evaluation metric (DIVE) reveals gains that static-biased…

desk verdict A useful MLLM-driven adapter and a valid critique of static-biased benchmarks, undercut by headline numbers that don't match the paper's own table. read the letter →

arxiv 2505.19901 v3 pith:IZRENXWQ submitted 2025-05-26 cs.CV

Peng Liu , Xiaoming Ren , Fengkai Liu , Qingsong Xie , Quanlong Zheng , Yanhao Zhang , Haonan Lu , Yujiu Yang This is my paper · ORCID
classification cs.CV
keywords image-to-videogenerationmultimodallargelanguagemodeldiffusiontransformerdynamicvideoevaluationmotiondynamicsqualitybenchmarktext-imageconditioningconditionaladapter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that image-to-video generation models fail at complex scenes because they cannot jointly understand what is in the image and what the text asks them to do. To fix this, the authors integrate a multimodal large language model (Qwen2VL) into the CogVideoX-I2V diffusion transformer and fuse its visual and textual features with the T5 text encoder through a lightweight conditional adapter. They also claim that existing benchmarks like VBench-I2V are biased toward static videos, since even a purely static video scores as well as state-of-the-art models, and they propose a new benchmark, DIVE, that directly measures dynamic range, controllability, and dynamics-based quality. Under DIVE, their model reports large improvements over prior methods, including a 42.5% gain in dynamic range. The broader point is that motion dynamics should be a first-class axis of evaluation and that MLLM-based understanding is a promising route to achieving it.

What carries the argument

The load-bearing mechanism is the Multimodal Conditional Adapter (MCA), a learnable fusion module that combines three feature streams: the MLLM's vision-token features $f_i$, its answer-token features $f_a$, and the T5 text-encoder features $f_t$. The adapter computes $f_c = Z_m(M_i(f_i) + M_a(f_a)) + (f_t + Z_t(f_t))$, where $M_i$ and $M_a$ are MLPs and $Z_m$ and $Z_t$ are zero-initialized convolutional layers; the zero initialization ensures that early training preserves the base model's behavior and then gradually injects the learned multimodal conditions. This $f_c$ feeds into the diffusion transformer's 3D full-attention module, guiding denoising. The other central object is the DIVE evaluation metric, which uses GPT-4o to assign a dynamic degree (1-5) to each prompt and a dynamic score (0-1) to each generated video, then aggregates them into Dynamic Range, Dynamics Controllability, and Dynamics-Based Quality.

What would settle it

Re-run the DIVE evaluation with a new, larger set of image-text pairs and have fresh human raters rank videos for dynamics without knowing which model produced them; if the rank correlation between DIVE scores and human rankings is not significantly positive (say Spearman's rho below 0.5), the claimed 42.5% dynamic-range advantage would not be a reliable measure of human-perceived motion quality.

Watch

Extended reading notes

Core claim

The central claim is that adding a multimodal LLM to an image-to-video diffusion transformer, through a carefully designed adapter, unlocks substantially more dynamic and controllable video generation, and that existing evaluation protocols hide this improvement because they reward static outputs. Concretely, the paper integrates Qwen2VL into CogVideoX-I2V, feeding the MLLM's vision-token features, answer-token features, and T5 text features into a zero-initialized convolutional adapter (the Multimodal Conditional Adapter, MCA). The authors report state-of-the-art results on the VBench-I2V leaderboard and, more pointedly, on their proposed DIVE benchmark, where they claim improvements of 42.5% in dynamic range, 7.9% in dynamics controllability, and 11.8% in dynamics-based quality over the base CogVideoX-I2V-5B model. The paper also constructs the striking demonstration that a video made by duplicating a single frame scores on par with leading models on VBench-I2V, which motivates DIVE as a dynamic-aware alternative aligned with human rankings in a 20-person study.

Load-bearing premise

The DIVE benchmark's GPT-4o-based scoring is assumed to track what humans mean by good, dynamic video, even though only 20 volunteers watched 50 videos to check this.

Editorial extensions

If this is right

  • If the paper is right, image-to-video models that understand images and text jointly should be preferred for complex prompts, and simple text-encoder-only conditioning is a bottleneck.
  • VBench-I2V and similar frame-quality-focused benchmarks can be gamed by static videos, so future I2V evaluation should include a motion-dynamics axis like DIVE.
  • The reported 42.5% improvement in dynamic range means the model can produce both highly dynamic and near-static videos depending on the prompt, which is useful for controllable generation.
  • The MCA design shows that zero-initialized adapters allow adding MLLM features to a frozen, pretrained DiT without destroying its existing capabilities, suggesting a reusable recipe for other conditioning modalities.
  • The model's support for diverse image conditions (e.g., providing an image of an explosion type) suggests MLLM conditioning can carry abstract semantic controls beyond the literal first frame.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is that the MCA fusion recipe should transfer to other conditional video tasks, such as text-to-video editing, where the same joint image-text understanding is needed.
  • The paper's static-video attack on VBench suggests that any video benchmark without an explicit motion axis is vulnerable to reward hacking; future benchmark builders could add adversarial static inputs as a standard sanity check.
  • Because the model was fine-tuned on a filtered subset of OpenVid (123k videos), part of the dynamic-range gain could come from data curation rather than the MLLM alone; a controlled dataset ablation would disentangle these sources.
  • A testable extension: DIVE's reliance on GPT-4o could be validated by comparing against human dynamics ratings on a larger, independent pool of videos; if the correlation holds, DIVE becomes a cheap standard for dynamic I2V evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Dynamic-I2V, an image-to-video generation model that augments CogVideoX-I2V-5B with Qwen2-VL features via a multimodal conditional adapter (MCA), and proposes DIVE, a GPT-4o-based evaluation benchmark for dynamic quality. The authors report state-of-the-art results on DIVE (improvements of 42.5% in DR, 7.9% in DC, 11.8% in DBQ over existing methods) and on VBench-I2V, and additionally present evidence that VBench-I2V's scores are biased toward static videos. The paper includes ablations of the MCA components and a small human study comparing four models.

Significance. The idea of fusing MLLM tokens into a DiT backbone for I2V is timely and potentially impactful, and the DIVE benchmark addresses a real limitation of existing I2V metrics. The static-video sanity check is a falsifiable demonstration that VBench-I2V rewards non-dynamic outputs. However, the central empirical claims are weakened by internal inconsistencies and the reliance on an unreleased evaluation prompt. If the reported gains survive recomputation and the benchmark is released, this could become a useful contribution. The current manuscript does not yet substantiate the SOTA claim.

major comments (4)
  1. [Section 4.2 and Abstract] The abstract and Section 4.2 claim improvements of 42.5%, 7.9%, and 11.8% over existing methods on DIVE. With CogVideoX-I2V-5B as the stated baseline, Table 3 yields relative gains of 54.6% for DR, 9.2% for DC, and 31.1% for DBQ. No baseline or formula reproduces the reported numbers. Please correct the numbers or explicitly specify the baseline and computation used.
  2. [Section 4.2 and 4.3] DIVE's core evaluation prompt and rubric are not provided (the supplementary is referenced but missing), and the human study used only 20 volunteers on 50 videos with no inter-annotator agreement or confidence intervals. Since DIVE is the primary benchmark for the SOTA claim and is introduced by the authors, the lack of a released prompt and of statistical validation prevents independent verification. Please include the prompt, release the evaluation code, and report statistical significance.
  3. [Table 1 and Section 4.1] The VBench-I2V results do not support the broad claim of state-of-the-art performance: Dynamic-I2V's Dynamic Degree (27.15) is substantially below DynamiCrafter (47.40) and SVD (43.17), and its Total Score advantage over CogVideoX-I2V-5B is only 0.24 points. If the authors' thesis is that dynamics matter, the model's low Dynamic Degree on VBench-I2V is concerning. Please clarify how the model can be SOTA on VBench-I2V while being worse on the dynamics dimension that the paper emphasizes.
  4. [Table 6] The ablation shows a large jump from +MLLM+MLPs (DR 27.90) to +MCA (DR 45.77), but the paper does not explain what causes the improvement. The only architectural difference between these two conditions appears to be the zero-initialized convolutions in Eq. (3). Please provide an analysis of the contribution of each MCA component (e.g., zero-conv initialization, residual connection) to the final performance.
minor comments (6)
  1. [Section 3, first sentence] There is a duplicated phrase: "Dynamic-I2V consists of Dynamic-I2V consists of a denosing module"; please fix the typo.
  2. [Section 3.2, text near Eq. (2)] The sentence "These features, fi, fa and ft are then into CA" is missing a verb (likely "are then fed into"); please correct.
  3. [Section 5.2, Implementation Details] The optimizer setting is written as "betas of [0.9, 0.95]" with an unclosed bracket; please check the notation.
  4. [Table 2 and Section 4.1] The protocol for excluding the camera-motion metric is not fully specified: the paper says the exclusion is due to an additional prompt for camera motion, but does not describe that prompt or how the exclusion affects the reported scores. Please clarify.
  5. [Table 5, header] There is a typo: "Text Conpliance" should be "Text Compliance".
  6. [Figure 3] The figure is crowded and the text is small; please improve readability or provide a larger version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the architecture, training loss, and DIVE evaluation are not defined in terms of each other, though the headline DIVE improvements are numerically inconsistent with Table 3 as a correctness issue.

full rationale

The derivation chain is not circular. Dynamic-I2V's conditional features are defined as fc = Zm(Mi(fi)+Ma(fa)) + (ft+Zt(ft)), with fi and fa from the frozen MLLM and ft from the T5 encoder, and the training loss L = E[||eps - eps_theta(z_t,t,fc)||^2] contains no reference to DIVE. DIVE is defined externally via GPT-4o-assigned dynamic degree scores on prompts and dynamic scores on generated videos from the VBench-I2V test set, not via Dynamic-I2V's weights or outputs; the low static-video row (DR 0.89, DBQ 8.26) shows the metric does not tautologically favor this model. The 20-person, 50-video human study is an independent, if small, external check, and the paper explicitly attributes DIVE's methodology to DEVIL [21], an outside work. No load-bearing self-citation is present: Qwen2VL [38] and CogVideoX-I2V [43] are external, and the authors do not cite their own prior work as the basis for any central premise. The DIVE-guided ablation is normal benchmark-driven model selection rather than a fitted parameter renamed as a prediction. The abstract's claimed 42.5%, 7.9%, and 11.8% DIVE improvements do not match recomputation from Table 3 (relative to CogVideoX-I2V-5B one obtains roughly 54.6%, 9.2%, and 31.1%), and the DIVE prompt and rubric are unreleased; these are reproducibility and correctness concerns, not circularity. The missing supplementary prompt is likewise an omitted-support limitation rather than a circular step.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical and rests on evaluation choices rather than on derived equations. The new benchmark DIVE is the main arbiter of success, and its rubrics and prompt are not disclosed, so the model's gains are not independently checkable.

free parameters (4)
  • GPT-4o evaluation prompt and rubric for DIVE = not disclosed
    The dynamic degree (1-5) and dynamic score (0-1) are produced by GPT-4o using an unpublished prompt; the entire DIVE ranking depends on this hand-designed instruction.
  • MLLM user prompt for feature extraction = not disclosed
    The prompt that asks Qwen2VL to emphasize image semantics and temporal action is described only as 'designed user prompt' with details in a missing supplementary (Section 3.2).
  • OpenVid filtering thresholds = 1M to 350k to 123k
    Criteria for excluding uncertain or mixed camera motion and transition animations are hand-set; the 123k training set is the foundation of all reported gains (Section 5.1).
  • DIVE test prompt selection = 355 pairs from VBench-I2V
    No sampling procedure is described; representativeness of these prompts is assumed (Section 4.2).
assumptions (5)
  • domain assumption GPT-4o's judgments of dynamic degree and dynamic score correlate with human perception.
    Invoked in Section 4.2; DIVE relies on LLM-as-judge with no calibration beyond a small human study.
  • domain assumption The VBench-I2V test set is a representative distribution for I2V evaluation.
    DIVE draws its 355 image-text pairs from VBench-I2V (Section 4.2), importing that benchmark's scope and selection.
  • domain assumption Qwen2VL-2B features can be injected into CogVideoX-I2V-5B through an adapter without disrupting the base model.
    The MCA design in Eq. 3 presumes that zero-initialized convolutions and MLPs are sufficient to preserve base capacities (Section 3.2).
  • domain assumption CoTracker-based filtering removes transition-affected videos without removing desirable dynamics.
    Section 5.1 uses CoTracker to verify camera motion and detect transitions; the accuracy of this filter is not analyzed.
  • domain assumption Human volunteer rankings are a reliable gold standard.
    Section 4.3 uses 20 experts and 50 videos; no inter-rater agreement is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic-I2V: Exploring Image-to-Video Generation Models via Multimodal LLM." pith.science (2026). https://pith.science/paper/IZRENXWQ

@misc{pith2026250519901,
  author       = {Pith},
  title        = {Pith review of: Dynamic-I2V: Exploring Image-to-Video Generation Models via Multimodal LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZRENXWQ}},
  note         = {Machine review of arXiv:2505.19901}
}
read the original abstract

Recent advancements in image-to-video (I2V) generation have shown promising performance in conventional scenarios. However, these methods still encounter significant challenges when dealing with complex scenes that require a deep understanding of nuanced motion and intricate object-action relationships. To address these challenges, we present Dynamic-I2V, an innovative framework that integrates Multimodal Large Language Models (MLLMs) to jointly encode visual and textual conditions for a diffusion transformer (DiT) architecture. By leveraging the advanced multimodal understanding capabilities of MLLMs, our model significantly improves motion controllability and temporal coherence in synthesized videos. The inherent multimodality of Dynamic-I2V further enables flexible support for diverse conditional inputs, extending its applicability to various downstream generation tasks. Through systematic analysis, we identify a critical limitation in current I2V benchmarks: a significant bias towards favoring low-dynamic videos, stemming from an inadequate balance between motion complexity and visual quality metrics. To resolve this evaluation gap, we propose DIVE - a novel assessment benchmark specifically designed for comprehensive dynamic quality measurement in I2V generation. In conclusion, extensive quantitative and qualitative experiments confirm that Dynamic-I2V attains state-of-the-art performance in image-to-video generation, particularly revealing significant improvements of 42.5%, 7.9%, and 11.8% in dynamic range, controllability, and quality, respectively, as assessed by the DIVE metric in comparison to existing methods.

Figures

Figures reproduced from arXiv: 2505.19901 by the authors.

Figure 1
Figure 1. Some results generated by Dynamic-I2V. Our results exhibit high dynamics while ensuring the quality of generation. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. I2V difficulty examples and showcases. (C) generates [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overall of Dynamic-I2V. Dynamic-I2V consists of a denosing module, a feature extraction module and a multimodal conditional adapter(MCA). MLLM extracts image features fi and answer features fa from conditional images and prompts, while the text encoder extracts text features ft. These features, fi, fa and ft are then into CA, which integrates them into a comprehensive feature representation fc. This combined feature… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Generation results when using different conditional im [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Subjective comparison between CogVideoX-5B-I2V[ [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EventOD: Event-Aware OD Flow Generation via LLM-Guided Semantic Modulation

    cs.AI 2026-06 conditional novelty 6.0 of 10

    EventOD adapts frozen OD generators to disruptive events by modulating inputs with LLM-derived semantic direction vectors and learned magnitude factors, improving hurricane and pandemic flow reconstruction.

  2. Waver: Wave Your Way to Lifelike Video Generation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Waver unifies text-to-video, image-to-video, and text-to-image generation in a single 12B-parameter DiT with a hybrid dual/single-stream architecture and a cascade refiner, claiming top-three public leaderboard performance.

Reference graph

Works this paper leans on

44 extracted references · 33 canonical work pages · cited by 2 Pith papers

  1. [1]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayi- heng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfe...

    arXiv 2023

  2. [2]

    Frozen in time: A joint video and image encoder for end-to- end retrieval

    Max Bain, Arsha Nagrani, G¨ul Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to- end retrieval. InIEEE International Conference on Computer Vision, 2021. 7

    work page 2021

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. 6

    2023

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz L...

    work page 2020

  5. [5]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Eka- terina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 7

    work page 2024

  6. [6]

    Seine: Short-to-long video diffusion model for generative transition and prediction, 2023

    Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction, 2023. 6

    work page 2023

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024. 3

    2024

  8. [8]

    Glm: General language model pretraining with autoregressive blank infilling, 2022

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling, 2022. 3

    work page 2022

Show all 44 references
  1. [9]

    Guiding instruction-based image editing via multimodal large language models, 2024

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction-based image editing via multimodal large language models, 2024. 3

    2024

  2. [10]

    Seed-x: Multi- modal models with unified multi-granularity comprehension and generation, 2025

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multi- modal models with unified multi-granularity comprehension and generation, 2025. 3

    2025

  3. [11]

    Denoising diffu- sion probabilistic models, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models, 2020. 2

    2020

  4. [12]

    Smartedit: Exploring complex instruction-based image editing with multimodal large language models, 2023

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, and Ying Shan. Smartedit: Exploring complex instruction-based image editing with multimodal large language models, 2023. 3

    2023

  5. [13]

    Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models. InProceedings of the IEEE/CVF Conferenc...

    2024

  6. [14]

    VBench: Com- prehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Com- prehensive benchmark suite for video generative models....

    2024

  7. [15]

    Vbench++: Comprehensive and versatile benchmark suite for video generative models, 2024

    Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench++: Comprehensive and versatile benchmark suite for v...

    2024

  8. [16]

    T2vbench: Benchmarking temporal dynamics for text-to- video generation

    Pengliang Ji, Chuyang Xiao, Huilin Tai, and Mingxiao Huo. T2vbench: Benchmarking temporal dynamics for text-to- video generation. In2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition Workshops (CVPRW), pages 5325–5335, 2024. 3

    2024

  9. [17]

    Miradata: A large-scale video dataset with long durations and structured captions, 2024

    Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions, 2024. 7

    2024

  10. [18]

    Co- tracker: It is better to track together, 2024

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together, 2024. 7

    2024

  11. [19]

    Hunyuanvideo: A systematic framework for large video generative models, 2025

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang 9 Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai,...

    2025

  12. [20]

    Animateanything: Consistent and control- lable animation for video generation, 2024

    Guojun Lei, Chi Wang, Hong Li, Rong Zhang, Yikai Wang, and Weiwei Xu. Animateanything: Consistent and control- lable animation for video generation, 2024. 2, 6

    2024

  13. [21]

    Evaluation of text-to-video generation models: A dynamics perspective.arXiv preprint arXiv:2407.01094,

    Mingxiang Liao, Hannan Lu, Xinyu Zhang, Fang Wan, Tianyu Wang, Yuzhong Zhao, Wangmeng Zuo, Qixiang Ye, and Jing- dong Wang. Evaluation of text-to-video generation models: A dynamics perspective.arXiv preprint arXiv:2407.01094,

    arXiv

  14. [22]

    Video-llava: Learning united visual representa- tion by alignment before projection, 2024

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representa- tion by alignment before projection, 2024. 3

    2024

  15. [23]

    Stiv: Scalable text and image conditioned video generation, 2024

    Zongyu Lin, Wei Liu, Chen Chen, Jiasen Lu, Wenze Hu, Tsu-Jui Fu, Jesse Allardice, Zhengfeng Lai, Liangchen Song, Bowen Zhang, Cha Chen, Yiran Fei, Yifan Jiang, Lezhi Li, Yizhou Sun, Kai-Wei Chang, and Yinfei Yang. Stiv: Scalable text and image conditioned video generation, 2024. 6

    2024

  16. [24]

    Evalcrafter: Benchmarking and evalu- ating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evalu- ating large video generation models. 2023. 3

    2023

  17. [25]

    Qwen2vl-flux: Unifying image and text guidance for controllable image generation, 2024

    Pengqi Lu. Qwen2vl-flux: Unifying image and text guidance for controllable image generation, 2024. 3

    2024

  18. [26]

    Openvid-1m: A large-scale high-quality dataset for text-to- video generation, 2024

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhen- heng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to- video generation, 2024. 7

    2024

  19. [27]

    Kosmos-G: Generating images in context with multimodal large language models.ArXiv, abs/2310.02992, 2023

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-G: Generating images in context with multimodal large language models.ArXiv, abs/2310.02992, 2023. 3

    2023 arXiv

  20. [28]

    Kosmos-g: Generating images in context with multimodal large language models, 2024

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models, 2024. 3

    2024

  21. [29]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Askell Amanda, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, JongWook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Askell Amanda, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from nat- ural language supervision.Cornell University - arX...

    2021

  22. [30]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and PeterJ. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.arXiv: Learning,arXiv: Learning, 2019. 3

    2019

  23. [31]

    Vibe: A text-to-video benchmark for evaluat- ing hallucination in large multimodal models.arXiv preprint arXiv:2411.10867, 2024

    Vipula Rawte, Sarthak Jain, Aarush Sinha, Garv Kaushik, Aman Bansal, Prathiksha Rumale Vishwanath, Samyak Ra- jesh Jain, Aishwarya Naresh Reganti, Vinija Jain, Aman Chadha, et al. Vibe: A text-to-video benchmark for evaluat- ing hallucination in large multimodal models.arXiv p...

    2024 arXiv

  24. [32]

    Consisti2v: Enhancing visual consistency for image-to-video generation, 2024

    Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation, 2024. 6

    2024

  25. [33]

    T2v-compbench: A comprehen- sive benchmark for compositional text-to-video generation,

    Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2v-compbench: A comprehen- sive benchmark for compositional text-to-video generation,

  26. [34]

    Generative multi- modal models are in-context learners, 2024

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multi- modal models are in-context learners, 2024. 3

    2024

  27. [35]

    Mimir: Improving video diffusion models for precise text understanding, 2024

    Shuai Tan, Biao Gong, Yutong Feng, Kecheng Zheng, Dandan Zheng, Shuwei Shi, Yujun Shen, Jingdong Chen, and Ming Yang. Mimir: Improving video diffusion models for precise text understanding, 2024. 3

    2024

  28. [36]

    Mige: A unified framework for multimodal instruction-based image generation and editing,

    Xueyun Tian, Wei Li, Bingbing Xu, Yige Yuan, Yuanzhuo Wang, and Huawei Shen. Mige: A unified framework for multimodal instruction-based image generation and editing,

  29. [37]

    Llama: Open and efficient foundation language mod- els, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation lan...

    2023

  30. [38]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024

  31. [39]

    Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content, 2024

    Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, Fei Yang, Pengfei Wan, and Di Zhang. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content, 2024. 7

    2024

  32. [40]

    Cogvlm: Visual expert for pretrained language models, 2024

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2024. 3

    2024

  33. [41]

    Dynamicrafter: Animating open-domain im- ages with video diffusion priors, 2023

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain im- ages with video diffusion priors, 2023. 6

    2023

  34. [42]

    Easyanimate: A high-performance long video generation method based on transformer architecture, 2024

    Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture, 2024. 2

    2024

  35. [43]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024. 3, 6, 8 10

    2024 arXiv

  36. [44]

    I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models, 2023

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jin- gren Zhou. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models, 2023. 2, 6 11

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.