Pith. sign in

REVIEW 4 major objections 5 minor 93 references

Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Divot claims that a video tokenizer has learned adequate spatial and temporal representations exactly when a video diffusion model can denoise video from those representations, and that the same diffusion model can then decode them back…

desk verdict A genuinely new continuous video tokenizer trained by diffusion denoising, with real empirical promise, but the temporal-encoding claim is under-supported and the generation eval has a transparency gap. read the letter →

arxiv 2412.04432 v1 pith:HAIKJAYG submitted 2024-12-05 cs.CV

Yuying Ge , Yizhuo Li , Yixiao Ge , Ying Shan This is my paper · ORCID
classification cs.CV
keywords videotokenizerdiffusionmodelsself-supervisedrepresentationlearningunifiedunderstandingandgenerationGaussianmixturemodelcontinuousvisualtokensquestionansweringtext-to-video
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Divot, a video tokenizer trained by a self-supervised diffusion task: a video diffusion model must denoise densely sampled video latents using only the tokenizer's features from five sparsely sampled frames. The authors' claim is that the tokenizer has captured both appearance and motion if the denoiser can succeed with those features, and that the denoiser then becomes a natural de-tokenizer for generation. Divot-LLM attaches this tokenizer to a 7-billion-parameter language model and trains the model to read video via next-word prediction and to write video by predicting Gaussian-mixture distributions over the continuous token space. If right, the work shows that one model can both understand and generate video without discrete visual codes, and that video representation learning can be driven by pure video rather than captions.

What carries the argument

The central mechanism is the paired tokenizer and de-tokenizer. The tokenizer is a pretrained ViT encoder followed by a spatial-temporal transformer and a Perceiver Resampler, compressing five frames at 2 fps into 64 continuous tokens. The de-tokenizer is a pretrained video diffusion U-Net (DynamiCrafter) whose denoising of 8 fps latent video is conditioned on those tokens via cross-attention; the denoising loss is the tokenizer's training signal. For generation, the LLM predicts per-token Gaussian Mixture Model parameters (with 16 mixture components) and samples from that distribution to feed the de-tokenizer.

What would settle it

Train an otherwise identical Divot tokenizer whose features come from a single frame instead of five, and compare video-question accuracy plus the fidelity of reconstructed motion (for example FVD on action clips); if the single-frame version matches the five-frame version, the claim that the tokenizer captures temporal dynamics fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a pretrained video diffusion U-Net, stripped of its image-conditioning branch, can serve as both the training objective and the decoder for a continuous video tokenizer. Sparse frames are converted by a ViT encoder, a spatial-temporal transformer, and a Perceiver Resampler into 64 tokens; the U-Net then denoises the compressed latent representation of densely sampled frames conditioned on those tokens through cross-attention, and the noise-prediction error is backpropagated into the tokenizer. After training, the same U-Net decodes the tokens into video clips. For LLM-driven generation, the paper reports that minimizing mean-squared error between LLM outputs and video features yields averaged, repeating video, while predicting the parameters of a 16-component Gaussian Mixture Model and sampling from it gives better generation quality and semantic alignment. The authors also report that the diffusion-trained tokenizer improves video-question benchmarks over the same architecture trained with captioning loss, and that Divot-LLM reaches competitive comprehension results at 7B scale.

Load-bearing premise

The load-bearing premise is that a denoiser's success at reconstructing full video from the tokenizer's five sparse frames is caused by temporal information in those tokens, not by the denoiser's own motion habits or by static appearance cues.

Editorial extensions

If this is right

  • A unified 7B model can answer video questions and generate video from text without converting video into discrete codes.
  • Video representation learning can proceed from large unlabeled video corpora, since the diffusion loss requires no captions.
  • The diffusion decoder doubles as a de-tokenizer, so any representation the tokenizer produces is immediately decodable into a video clip.
  • Probabilistic generation, rather than deterministic regression, is what makes LLM prediction of high-level continuous video features viable.
  • The same tokenizer transfers to new visual domains such as animation by fine-tuning only the de-tokenizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the denoising-conditioning principle transfers, the same recipe could be used to learn tokenizers for other continuous modalities such as audio or sensor streams, wherever a pretrained diffusion decoder exists.
  • Editorial inference: the paper does not isolate how much temporal signal comes from the tokenizer versus the U-Net's prior; a natural next experiment varies the number of input frames and the frame rate while holding the decoder fixed.
  • Editorial inference: because generation samples from a learned GMM, the spread of that distribution could serve as a per-clip uncertainty signal, which the paper does not analyze.
  • Editorial inference: the reported strength of continuous over discrete tokens suggests re-examining hybrid tokenizers where discrete semantic tokens and continuous motion tokens are combined.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Divot, a video tokenizer trained by conditioning a pre-trained video diffusion U-Net (DynamiCrafter) on sparse-frame features and asking it to denoise dense VAE latents of the same video; after training, the same U-Net is used as a de-tokenizer to decode videos from the tokenizer's features. The authors then build Divot-LLM by combining this tokenizer with Mistral-7B, using video-to-text autoregression for comprehension and GMM-based sampling for text-to-video generation. The model is evaluated on several video QA benchmarks, zero-shot MSR-VTT generation, and a video storytelling demo. The paper's central claim is that successful denoising of dense 8 fps video latents conditioned on Divot features certifies that the tokenizer has captured robust spatial and temporal information.

Significance. If the central attribution is established, the paper offers a useful step toward unified video comprehension and generation: a self-supervised tokenizer trained without caption supervision, a diffusion model that doubles as a de-tokenizer, and a GMM framework for fitting continuous video features with an LLM. The release of models and code would aid reproducibility, and the ablations in Tables 6 and 8 address several design choices that the community is actively exploring. The main risk is that the paper's core conceptual claim, that denoising success proves temporal encoding, is not supported by the current experiments; the ablations do not isolate temporal information from spatial appearance or from the U-Net's own motion prior.

major comments (4)
  1. [Sec. 1, Sec. 3.1.2, Sec. 3.1.3] The central claim that successful denoising of dense 8 fps latents from 5 sparse 2 fps frames certifies temporal encoding is not established. Because the conditioning tokenizer sees appearance-rich sparse frames and the denoising U-Net is a pre-trained DynamiCrafter with temporal convolutional and attention priors, the loss could in principle be minimized with appearance-only conditioning plus the U-Net's own motion prior. The ablation in Table 6 compares diffusion loss against caption loss but does not vary frame rate, token count, or temporal order, so it does not rule out this alternative. I would ask for a permutation/shuffle ablation (or a single-frame condition) and, if temporal encoding cannot be demonstrated, the paper should reframe the contribution as a spatial encoder coupled with a pre-trained motion prior rather than as a spatiotemporal tokenizer.
  2. [Sec. 4.3, Table 6] The diffusion-vs-caption tokenizer ablation is confounded with training data. The text says both models are pre-trained on Valley and instruction-tuned on Video-ChatGPT, but the caption-loss tokenizer is pre-trained with captioning loss on Valley, while the Divot tokenizer is pre-trained on 10M pure videos (WebVid-10M and Panda-70M, per Sec. 3.1.2). The improvement in Table 6 could therefore reflect additional training data or data domain rather than the diffusion objective. Please match data scale/domain between the two tokenizer training regimes, or explicitly account for this mismatch in the conclusions.
  3. [Appendix A.2, Table 4] The 'zero-shot' label on the MSR-VTT generation evaluation is not yet supported. Appendix A.2 describes a de-tokenizer adaptation technique that fine-tunes the de-tokenizer on features sampled from the predicted GMM distribution, but does not state which videos are used. If this adaptation uses the MSR-VTT training split, Table 4 is not a zero-shot comparison and must either be re-labeled or the adaptation must be removed/disclosed; the authors should also specify whether the CLIPSIM/FVD evaluation uses one sampled set of 16-frame clips per caption or multiple samples.
  4. [Tables 3 and 4] The quantitative comparisons are reported as single point estimates with no error bars or repeated runs for stochastic procedures (GPT-3.5 scoring, FVD, GMM sampling). As a result, small differences such as the CLIPSIM difference between Divot-LLM (0.2938) and VILA-U (0.2937) in Table 4 are not meaningful, and the claim of 'comparable' performance should be supported by variance estimates or multiple seeds. In addition, the comparison rows in Tables 3 and 4 differ in training data composition and evaluation protocols (e.g., 4.8M video-text pairs for pretraining versus 10M pure videos for tokenizer training), so the table should explicitly note which number is being used for each row.
minor comments (5)
  1. [Abstract] The abstract uses 'Divot-Vicuna' while the rest of the paper consistently uses 'Divot-LLM'; please unify the model name.
  2. [Sec. 3.1.2] The text says 'adds Gaussian noise θ' to the latent; this should be ϵ_t or z_t, consistent with Eq. (1).
  3. [Sec. 3.2 and Sec. 4.3] The text refers to 'Tab. 5' and 'Tab. 7', but the displayed tables are numbered Table 6 and Table 8; fix the cross-references and table numbering.
  4. [Appendix A.1] The appendix states that training uses 'v prediction' for denoising, but Eq. (1) is written for ϵ-prediction; please define the v-prediction parametrization and its relationship to Eq. (1).
  5. [Appendix A.2] The GMM mixture size is fixed to k=16 with no sensitivity analysis; please report at least one alternative value of k to justify the choice.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the diffusion-conditioning premise is an explicit assumption, and the paper's load-bearing results are benchmarked externally.

full rationale

The central premise (Sec. 1, Sec. 3.1.2) is a stated operational posit rather than a hidden derivation: 'if a video diffusion model can effectively de-noise video clips by taking the features of a video tokenizer as the condition, then the tokenizer has successfully captured robust spatial and temporal information.' The diffusion training objective (Eq. 1) minimizes noise-prediction error conditioned on tokenizer features, and the paper then checks the downstream value of the learned representations against external benchmarks (EgoSchema, Perception-Test, MVBench, MSVD, ActivityNet, MSR-VTT). Those evaluations are not constructed from the same MSE loss, so the comprehension and generation results do not reduce by construction to the training objective. The use of a fixed pre-trained DynamiCrafter U-Net (Sec. 3.1.3), including one from an overlapping author group, is a standard architectural backbone rather than a self-citation invoked to forbid alternatives, and the code-release claim makes it reproducible. The GMM sampling and the de-tokenizer adaptation (App. A.2) are explicit training stages; they adapt a decoder to the LLM's feature distribution, but they do not rename a fitted parameter as a prediction, and the caption-to-feature mapping is still independently evaluated. The absence of a temporal-order or frame-rate ablation means the causal attribution of temporal encoding to the denoising objective is not proven, but that is a correctness/evidence weakness, not circular equivalence. No equation or fitted quantity in the paper is equivalent to the claimed predictions by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions about what denoising success means, the transferability of the tokenizer to comprehension, and the choice of GMM modeling. The main free choices are hyperparameters that affect generation quality and are not derived from theory.

free parameters (4)
  • GMM mixture components k = 16
    Number of Gaussian components used to model each continuous video feature; chosen without an ablation sweep, and generation quality depends on this modeling choice (Section 4.3, Appendix A.2).
  • Video token count = 64 for LLM, 125 for de-tokenizer
    Perceiver Resampler output sizes chosen by hand; affects how much information the LLM must predict and reconstruction quality (Section 3.1.3, Appendix A.1).
  • Frame sampling rates = 2 fps sparse, 8 fps dense
    Sparse sampling chosen for semantic redundancy; this choice determines what temporal dynamics the tokenizer can capture and is not ablated for comprehension (Section 3.1.2).
  • Condition dropout probability = 5%
    Used only during de-tokenizer fine-tuning to enable classifier-free guidance; a hyperparameter that affects generation quality (Appendix A.1).
assumptions (4)
  • domain assumption Successful denoising conditioned on tokenizer features implies the tokenizer captures robust spatial and temporal information.
    Introduced in Section 1 and Section 3.1.2 as the core premise; no formal proof, only empirical support via downstream benchmarks and an ablation.
  • domain assumption The DynamiCrafter U-Net, with image concatenation removed, can serve as a faithful de-tokenizer for high-level Divot features.
    The paper fine-tunes DynamiCrafter to remove low-level image conditioning and assumes the remaining temporal prior is sufficient to decode feature tokens; Appendix A.1 describes this modification but does not isolate its effect.
  • ad hoc to paper Continuous video features are better modeled by a GMM than by MSE regression or diffusion denoising of features.
    This is an empirical finding from the paper's own ablation (Table 7), asserted as a design choice, not derived from theory.
  • domain assumption Pre-trained models (ViT-H, Mistral-7B, DynamiCrafter, VAE) remain valid under the new training regimes.
    The method initializes from these checkpoints and fine-tunes them; performance depends on their quality, which is not independently verified here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation." pith.science (2026). https://pith.science/paper/HAIKJAYG

@misc{pith2026241204432,
  author       = {Pith},
  title        = {Pith review of: Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HAIKJAYG}},
  note         = {Machine review of arXiv:2412.04432}
}
read the original abstract

In recent years, there has been a significant surge of interest in unifying image comprehension and generation within Large Language Models (LLMs). This growing interest has prompted us to explore extending this unification to videos. The core challenge lies in developing a versatile video tokenizer that captures both the spatial characteristics and temporal dynamics of videos to obtain representations for LLMs, and the representations can be further decoded into realistic video clips to enable video generation. In this work, we introduce Divot, a Diffusion-Powered Video Tokenizer, which leverages the diffusion process for self-supervised video representation learning. We posit that if a video diffusion model can effectively de-noise video clips by taking the features of a video tokenizer as the condition, then the tokenizer has successfully captured robust spatial and temporal information. Additionally, the video diffusion model inherently functions as a de-tokenizer, decoding videos from their representations. Building upon the Divot tokenizer, we present Divot-Vicuna through video-to-text autoregression and text-to-video generation by modeling the distributions of continuous-valued Divot features with a Gaussian Mixture Model. Experimental results demonstrate that our diffusion-based video tokenizer, when integrated with a pre-trained LLM, achieves competitive performance across various video comprehension and generation benchmarks. The instruction tuned Divot-Vicuna also excels in video storytelling, generating interleaved narratives and corresponding videos.

Figures

Figures reproduced from arXiv: 2412.04432 by the authors.

Figure 1
Figure 1. We utilize the diffusion procedure to learn a video tok￾enizer in a self-supervised manner for unified comprehension and generation, where the spatiotemporal representations serve as the condition of a diffusion model to de-noise video clips. Additionally, the proxy diffusion model functions as a de-tokenizer to decode realistic video clips from the video representations. relatively under-explored. Achieving unified… view at source ↗
Figure 2
Figure 2. Overview of Divot tokenization and de-tokenization. During training, sparsely sampled video frames are fed into the tokenizer to obtain spatiotemporal representations. These representations serve as the conditions for a U-Net, which is trained to de-noise the noisy VAE latents of densely sampled video frames. During inference, the video representations from the Divot tokenizer can be decoded into realistic video cli… view at source ↗
Figure 3
Figure 3. Overview of Divot-LLM. Video features from the Divot tokenizer are fed into the LLM to perform next-word prediction for video comprehension, while learnable queries are input into the LLM to model the distributions of Divot features using a Gaussian Mixture Model (GMM) for video generation. During inference, video features are sampled from the predicted GMM distribution to decode videos using the de-tokenizer. the t… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Paradigms for modeling video representations from the Divot tokenizer with a LLM for video generation. (a) MSE Regression, where the LLM output is trained to minimize its distance with video features using Mean Squared Error (MSE) loss; (b) Diffusion Modeling, where th…
Figure 5
Figure 5. Figure 5: Reconstructed videos, where the Divot tokenizer obtains spatiotemporal representations of sparsely sampled video frames and the [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison of text-to-video generation with MLLMs that are capable of unified video comprehension and generation. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Qualitative examples of video storytelling by Divot-LLM. Given a story instruction, Divot-LLM can generate rich textual narratives [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: More qualitative examples of reconstructed videos, where the Divot tokenizer obtains spatiotemporal representations of sparsely [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: More qualitative examples of text-to-video generation by Divot-LLM, which effectively generates videos that are both semantically [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Qualitative examples of video comprehension by Divot-LLM. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

93 extracted references · 20 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716–23736,

  2. [2]

    Frozen in time: A joint video and image encoder for end-to- end retrieval

    Max Bain, Arsha Nagrani, G¨ul Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to- end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1728–1738, 2021. 4, 5, 6, 13

    2021

  3. [3]

    Label-efficient seman- tic segmentation with diffusion models

    Dmitry Baranchuk, Ivan Rubachev, Andrey V oynov, Valentin Khrulkov, and Artem Babenko. Label-efficient seman- tic segmentation with diffusion models. arXiv preprint arXiv:2112.03126, 2021. 3

    arXiv 2021

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 7

    arXiv 2023

  5. [5]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 7

    2023

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1

    1901

  7. [7]

    Collecting highly parallel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th an- nual meeting of the association for computational linguistics: human language technologies, pages 190–200, 2011. 6

    2011

  8. [8]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xi- aodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 3

    arXiv

Show all 93 references
  1. [9]

    Panda-70m: Captioning 70m videos with multiple cross- modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Eka- terina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross- modality teachers. In Proceedings of the IEEE/CVF Con-...

    2024

  2. [10]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 6

    2024 arXiv

  3. [11]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. 1

    2022 arXiv

  4. [12]

    Dreamllm: Synergistic multimodal com- prehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal com- prehension and creation. arXiv preprint arXiv:2309.11499,

    arXiv

  5. [13]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  6. [14]

    Preserve your own correlation: A noise prior for video diffusion models

    Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Comp...

    2023

  7. [15]

    Planting a seed of vision in large language model.arXiv preprint arXiv:2307.08041, 2023

    Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model.arXiv preprint arXiv:2307.08041, 2023. 1, 2

    2023 arXiv

  8. [16]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218, 2023

    2023 arXiv

  9. [17]

    Seed-x: Multi- modal models with unified multi-granularity comprehension and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multi- modal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024. 1, 2, 4

    2024 arXiv

  10. [18]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller- Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In...

    2017

  11. [19]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...

    2022

  12. [20]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  13. [21]

    Cogvideo: Large-scale pretraining for text-to-video gen- eration via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video gen- eration via transformers. arXiv preprint arXiv:2205.15868,

    arXiv

  14. [22]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 6

    2021 arXiv

  15. [23]

    Soda: Bottle- 9 neck diffusion models for representation learning

    Drew A Hudson, Daniel Zoran, Mateusz Malinowski, An- drew K Lampinen, Andrew Jaegle, James L McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottle- 9 neck diffusion models for representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  16. [24]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  17. [25]

    Unified language-vision pretraining with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04669,

  18. [26]

    Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization

    Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. arXiv preprint arXiv:2402.03161, 2024. 1, 2, 3, 5, 6, 7

    2024

  19. [27]

    The kinetics hu- man action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics hu- man action video dataset. arXiv preprint arXiv:1705.06950,

    arXiv

  20. [28]

    Auto-encoding vari- ational bayes

    Diederik P Kingma and Max Welling. Auto-encoding vari- ational bayes. arXiv preprint arXiv:1312.6114 , 2013. 2, 4

    2013 arXiv

  21. [29]

    Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125, 2023

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125, 2023. 2, 7

    2023 arXiv

  22. [30]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chun- yuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 6

    2024 arXiv

  23. [31]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. arXiv preprint arXiv:2311.17005, 2023. 6

    2023 arXiv

  24. [32]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  25. [33]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024. 2, 4, 5, 8

    2024 arXiv

  26. [34]

    Tgif: A new dataset and benchmark on animated gif description

    Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641–4650, 2016. 5

    2016

  27. [35]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 6

    2023 arXiv

  28. [36]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  29. [37]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 5, 6

    2023 arXiv

  30. [38]

    Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, 2024. 6

    2024

  31. [39]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 5

    2024

  32. [40]

    World model on million-length video and language with ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268 , 2024. 1, 2, 6

    2024 arXiv

  33. [41]

    Unified-io 2: Scaling autoregressive multi- modal models with vision, language, audio, and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Anirud- dha Kembhavi. Unified-io 2: Scaling autoregressive multi- modal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023. 2

    2023 arXiv

  34. [42]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207,

    arXiv

  35. [43]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 5, 6, 7

    2023 arXiv

  36. [44]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 6

    2023

  37. [45]

    Snap video: Scaled spatiotemporal transformers for text-to-video synthe- sis

    Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthe- sis. In Proceedings of the IEEE/CVF Conferen...

    2024

  38. [46]

    Gpt-4v(ision) system card, 2023

    OpenAI. Gpt-4v(ision) system card, 2023. 6

    2023

  39. [47]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. 6

    2024

  40. [48]

    Per- ception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing ...

    2024

  41. [49]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  42. [50]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 3

    2022

  43. [51]

    Laion-5b: An open large-scale dataset for training next gen- eration image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next gen- eration image-text models. Advances in Neural Infor...

    2022

  44. [52]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  45. [53]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792,

    arXiv

  46. [54]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286, 2023. 2, 4

    2023 arXiv

  47. [56]

    Generative pretraining in multi- modality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multi- modality. arXiv preprint arXiv:2307.05222, 2023

    2023 arXiv

  48. [57]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 1, 2

    2024 arXiv

  49. [58]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 6

    2023 arXiv

  50. [59]

    Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 6

    2024 arXiv

  51. [60]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  52. [61]

    Givt: Generative infinite-vocabulary transformers

    Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite-vocabulary transformers. In Eu- ropean Conference on Computer Vision , pages 292–309. Springer, 2025. 5, 7

    2025

  53. [62]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 5, 6

    2018 arXiv

  54. [63]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 2

    2017

  55. [64]

    Larp: Tokenizing videos with a learned autoregressive generative prior

    Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with a learned autoregressive generative prior. arXiv preprint arXiv:2410.21264, 2024. 7

    2024 arXiv

  56. [65]

    Diffusion feedback helps clip see better

    Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better. arXiv preprint arXiv:2407.20171, 2024. 3

    2024 arXiv

  57. [66]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024. 7

    2024

  58. [67]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 2, 3

    2024 arXiv

  59. [68]

    Internvid: A large-scale video-text dataset for multimodal un- derstanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal un- derstanding and generation. arXiv preprint arXiv:2307.06942,

    arXiv

  60. [69]

    Loong: Gen- erating minute-level long videos with autoregressive language models

    Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Gen- erating minute-level long videos with autoregressive language models. arXiv preprint arXiv:2410.02757, 2024. 2, 6, 7

    2024 arXiv

  61. [70]

    Diffusion models as masked autoencoders

    Chen Wei, Karttikeya Mangalam, Po-Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16284–16294, 2023. 3

    2023

  62. [71]

    Godiva: Generating open-domain videos from natural descriptions

    Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021. 6

    2021 arXiv

  63. [72]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848, 2024. 2

    arXiv

  64. [73]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023

    2023 arXiv

  65. [74]

    Vila-u: a unified foundation model inte- grating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model inte- grating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. 1, 2, 6, 7

    2024 arXiv

  66. [75]

    Denoising diffusion autoencoders are unified self-supervised 11 learners

    Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised 11 learners. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 15802–15812, 2023. 3

    2023

  67. [76]

    Next-qa: Next phase of question-answering to explaining tem- poral actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining tem- poral actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786,

  68. [77]

    Show-o: One single transformer to unify multimodal understanding and genera- tion

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and genera- tion. arXiv preprint arXiv:2408.12528, 2024. 1, 2, 3

    2024 arXiv

  69. [78]

    Dynamicrafter: Animating open- domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open- domain images with video diffusion priors. In European Conference on Computer Vision, pages 399–417. Springer,

  70. [79]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 5, 6

    2016

  71. [80]

    Open-vocabulary panoptic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955–2966, 2023. 3

    2023

  72. [81]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 6

    2024 arXiv

  73. [82]

    Videogpt: Video generation using vq-vae and transform- ers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srini- vas. Videogpt: Video generation using vq-vae and transform- ers. arXiv preprint arXiv:2104.10157, 2021. 5

    2021 arXiv

  74. [83]

    Mmar: Towards lossless multi-modal auto-regressive prababilistic modeling

    Jian Yang, Dacheng Yin, Yizhou Zhou, Fengyun Rao, Wei Zhai, Yang Cao, and Zheng-Jun Zha. Mmar: Towards lossless multi-modal auto-regressive prababilistic modeling. arXiv preprint arXiv:2410.10798, 2024. 2

    2024 arXiv

  75. [84]

    Seed-story: Multimodal long story generation with large language model

    Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen. Seed-story: Multimodal long story generation with large language model. arXiv preprint arXiv:2407.08683, 2024. 2

    2024 arXiv

  76. [85]

    Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442, 2019

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442, 2019. 5

    1910 arXiv

  77. [86]

    Scaling autoregressive multi- modal models: Pretraining and instruction tuning

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi- modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2023. 1, 2

    2023 arXiv

  78. [87]

    Capsfu- sion: Rethinking image-text data at scale

    Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfu- sion: Rethinking image-text data at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14022–14032, 2024. 5

    2024

  79. [88]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9127–9134, 2019. 5, 6

    2019

  80. [89]

    Monoformer: One transformer for both diffusion and autore- gression

    Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang. Monoformer: One transformer for both diffusion and autore- gression. arXiv preprint arXiv:2409.16280, 2024. 2

    2024 arXiv

  81. [90]

    Unleashing text-to-image diffusion mod- els for visual perception

    Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion mod- els for visual perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5729– 5739, 2023. 3

    2023

  82. [91]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michi- hiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 1, 2

    2024 arXiv

  83. [92]

    Towards auto- matic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards auto- matic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelli- gence, 2018. 5

    2018

  84. [93]

    Vl-gpt: A generative pre-trained transformer for vision and language un- derstanding and generation

    Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. Vl-gpt: A generative pre-trained transformer for vision and language un- derstanding and generation. arXiv preprint arXiv:2312.09251,

    arXiv

  85. [2023]

    Curious George

    1, 2 12 A. Implementation Details A.1. Divot Tokenization. Model Architecture. The Divot tokenizer is composed of a pre-trained ViT-H/14, a Spatial-Temporal Transformer and a Perceiver Resampler. Specifically, given a video clip with a duration of two seconds, we sample 5 fram...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.