Pith. sign in

REVIEW 4 major objections 5 minor 70 references

HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims a single transformer can both understand and generate images and videos by initializing its three parts from pretrained vision, language, and diffusion models, then aligning them with a multimodal warmup.

desk verdict A genuinely novel training recipe for unified multimodal transformers, but the core warmup losses are undefined and the reported numbers are internally inconsistent, so the central claim is not yet verifiable. read the letter →

arxiv 2506.02975 v1 pith:P3IBQU7W submitted 2025-06-03 cs.CV cs.AI

Yicheng Xiao , Lin Song , Rui Yang , Cheng Cheng , Zunnan Xu , Zhaoyang Zhang , Yixiao Ge , Xiu Li
show 1 more author
Ying Shan
This is my paper
classification cs.CVcs.AI
keywords unifiedmultimodalmodelsingletransformerwarmupfeaturepre-scalingAdaLNvideounderstandinggenerationdiffusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a single-transformer architecture can handle both multimodal understanding and multimodal generation without paying the usual training cost of unified models. The idea is to build the transformer in three depth-wise parts, initialize the parts from specialized pretrained models — a vision transformer for visual encoding, an LLM for language, and a diffusion transformer for visual decoding — and then train the parts to work together through a multimodal warmup, connector alignment, and unified tuning. Two additional mechanisms, feature pre-scaling and multimodal AdaLN, are introduced to resolve cross-modal feature-scale differences that otherwise slow or destabilize training. If the claims hold, the result is a practical recipe for upgrading a strong text LLM into a unified image-and-video model at reduced GPU cost, with scores on benchmarks such as SEED-Bench, POPE, MVBench, and VBench that are competitive with or better than prior unified models.

What carries the argument

The load-bearing machinery is the HaploOmni block, a transformer block whose normalization is a multimodal AdaLN: an adaptive layer-normalization that stores scale, shift, and gate parameters for both condition and noise tokens and selects them by a softmax switch score for each input token. Around this block the model uses a hybrid attention mask — causal attention for text and timestep tokens, bidirectional attention for visual and noise tokens — plus pre-connector and post-connector modules that map features into a common space, and feature pre-scaling that compensates for the roughly tenfold amplitude gap between noise tokens and ViT-distilled visual features. The training procedure does the real work: a multimodal warmup stage distills each specialized teacher into its decoder segment while teaching identity mapping, followed by connector alignment and end-to-end unified tuning with next-token-prediction loss and diffusion loss together.

What would settle it

Train the same 9B Qwen2.5-based model on the same understanding and generation data with and without the multimodal warmup, and compare SEED-Bench, POPE, MVBench, and VBench scores; if the with-warmup version does not clearly beat the no-warmup version, the warmup is not the cause. A second check is to measure the cosine distance between CLIP-ViT-L outputs and the pre-decoder's visual outputs after warmup: if the distance grows sharply, the distillation loss failed to preserve visual knowledge.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that pretrained specialized models can be reused as the three segments of a single transformer decoder: a pre-decoder initialized from a ViT, a base-decoder from an LLM, and a post-decoder from a diffusion transformer. During multimodal warmup, the pre-decoder and post-decoder are fine-tuned with identity and distillation losses so they conform to the LLM's autoregressive inference paradigm while keeping their specialized knowledge. After connector alignment and unified tuning, the same parameters perform both understanding and generation; the paper reports state-of-the-art results among unified models on most tested benchmarks, including notable video understanding scores (47.1 EgoSchema, 52.9 MVBench) and strong video generation results on VBench.

Load-bearing premise

The load-bearing premise is that the multimodal warmup can graft a pretrained vision encoder and a pretrained video-diffusion decoder onto an LLM using identity and distillation losses without eroding what either teacher knows, so the benchmark gains come from the alignment rather than from the strong Qwen2.5 backbone alone.

Editorial extensions

If this is right

  • Unified image-video models can inherit most of their capability from existing pretrained checkpoints, so building one no longer requires training a large transformer from scratch.
  • A single 9B parameter model can serve understanding and generation with one set of weights, which would simplify deployment and reduce memory overhead relative to encoder-decoder-plus-diffusion pipelines.
  • The reported GPU-hours gap (5,792 for HaploOmni versus 856,481 for Chameleon and 21,504 for Janus) suggests that prior-knowledge initialization, if it holds up, is the main route to affordable unified models.
  • Video understanding and generation can share one transformer rather than needing separate video-specific models, as shown by the MVBench and VBench gains over VILA-U.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Read as a recipe rather than an architecture search, the method implies that any strong LLM can be upgraded into a unified image-video model by attaching a warmed-up ViT pre-decoder and a warmed-up DiT post-decoder; the specifics of the base LLM may matter less than the alignment procedure.
  • A controlled next experiment would hold the Qwen2.5 base and the training data fixed and toggle multimodal warmup, feature pre-scaling, and multimodal AdaLN one at a time, measuring full benchmark deltas rather than loss curves alone; that would show which mechanism carries the reported gains.
  • The same warmup-plus-connector scheme could plausibly extend to audio or 3D modalities whenever a pretrained specialist encoder and a pretrained diffusion decoder exist, since the alignment machinery is modality-agnostic in form.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HaploOmni, a single-transformer architecture for unified image/video understanding and generation. The model is split into pre-decoder, base-decoder, and post-decoder components that are initialized from CLIP-ViT-L, Qwen2.5-7B, and CogVideoX-2B, respectively, and then trained in three stages: a multimodal warmup with identity and distillation losses, a connector alignment stage, and a final unified tuning stage. The authors introduce feature pre-scaling and a multimodal AdaLN module to handle cross-modal feature mismatches. They report competitive or state-of-the-art results among unified models on several image and video understanding and generation benchmarks while using substantially fewer GPU-hours than prior unified single-transformer models such as Chameleon and Janus.

Significance. If the claimed results are reproducible, the paper offers a practically valuable recipe: reuse strong pretrained vision, language, and diffusion models to assemble a single unified transformer at a fraction of the training cost of from-scratch unified models. The paper makes the efficiency story concrete with a GPU-hours comparison (5792 versus 856481 for Chameleon and 21504 for Janus) and commits to releasing code. The proposed multimodal warmup and multimodal AdaLN are architecturally interesting. However, the central mechanism of the warmup is not specified in verifiable form, the reported benchmark numbers are internally inconsistent, and the generation evaluation is incomplete. These gaps currently prevent the reader from assessing whether the reported gains come from the proposed method or from the strong Qwen2.5/CogVideoX priors alone.

major comments (4)
  1. [§3.3, Stage 1 (Multimodal Warmup)] The identity loss and distillation loss are named but never defined: there are no loss equations, no target definitions, no loss-weighting scheme, and no explicit list of parameters updated in each warmup step. This is load-bearing because the central claim is that warmup preserves the knowledge of the pretrained ViT and DiT teachers while adapting them to the LLM's autoregressive paradigm. The only ablation is Fig. 5(a), a single qualitative image. Please provide the exact loss definitions and a quantitative ablation, e.g., benchmark scores with and without the warmup stage and with and without teacher initialization.
  2. [Tables 1, 2, and §4.3] The paper reports three different sets of numbers for the same model: SEED is 74.8 in the text of §4.3, 74.6 in Table 1, and 74.0 in Table 2; POPE is 87.9 in the text, 88.3 in Table 1, and 89.6 in Table 2. These inconsistencies make it impossible to verify the central benchmark claims. Please reconcile the tables and the text, and state the exact evaluation protocol used for each reported number.
  3. [Table 3 and §4.3 (VBench)] The VBench evaluation reports only 5 of the 16 benchmark dimensions and gives no error bars or number of evaluation runs. The text claims HaploOmni 'exhibits strong performance across most evaluated aspects' and is 'competitive with pure generative models,' but the selected subset does not support that claim. Please report the full VBench suite, with multiple seeds or confidence intervals where applicable, and clarify the selection criterion for the reported dimensions.
  4. [§3.2, Eq. (3) and Algorithm 1] The definition of the multimodal AdaLN is not fully specified: Eq. (3) uses a learnable matrix W_Ada, while Algorithm 1 uses W_MAL, and the dimensions of the state matrix and the switch-score computation are not fully tied together. In addition, the motivating claim in §3.2 that noise-token amplitudes are 'about 10 times larger' than visual features is stated without supporting measurements. Please provide the exact dimensions, unify the notation, and quantify the amplitude mismatch that motivates feature pre-scaling.
minor comments (5)
  1. [§3.3] There is a typo: 'This adjHaploOmniment enables compatibility' should read 'This adjustment enables compatibility.'
  2. [§3.2, Eq. (4)–(6)] The dimensions of W′, W_SN, and the indicator functions I_0 and I_1 are not specified; please clarify the shapes and how the switch scores interact with the learnable transformations.
  3. [§A.3] The sentence 'we relax the training for the post-decoder' is unclear; please clarify whether parameters are released or relaxed, and state exactly which parameters are trainable in each of the three connector-alignment steps.
  4. [Table 4] The HaploOmni-Block ablation reports only three benchmarks and does not state the training data, training steps, or evaluation protocol; please provide these details so the comparison with the 'standard block' is meaningful.
  5. [References] References [28] and [29] are identical entries for SEED-Bench; please deduplicate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HaploOmni's claims are benchmark-driven and independently testable; undefined warmup losses are a reproducibility gap, not a circular derivation.

full rationale

The paper's central claims are empirical and verified against external benchmarks (SEED, POPE, MVBench, VBench, MMMU, AI2D, etc.), so there is no derivation chain in which an output is equivalent to an input by construction. The proposed components (multimodal warmup, feature pre-scaling, multimodal AdaLN) are described algorithmically and are not defined in terms of the benchmark numbers they are claimed to explain. The Stage 1 identity and distillation losses are never specified with equations, targets, or weights; this is a serious reproducibility and verification gap, but it is not circularity because the paper does not derive its benchmark results from those losses. Self-citations to SEED-Bench and HaploVLM exist and share authors with this paper, but they are used as evaluation tools or background baselines rather than as load-bearing justification for the method's validity; the main results are compared against many independently authored models and benchmarks. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in solely via self-citation. The paper is therefore self-contained against external evidence, and the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The model has many learned parameters, which is standard for ML models, but the paper introduces few hand-tuned free parameters beyond the AdaLN layer count. The core assumptions are domain-level: that pretrained components can be fused without degradation, that the benchmarks measure what matters, and that the proposed scaling fix addresses the fundamental difficulty. The only genuinely new entity is the Multimodal AdaLN module, which lacks independent evidence.

free parameters (1)
  • number of multimodal AdaLN layers = 2
    Set by default in Appendix A.3, with no ablation varying this count shown.
assumptions (3)
  • domain assumption Pretrained ViT, LLM, and DiT can be reassembled into a single transformer via warmup without loss of capability.
    The entire method rests on the assumption that identity/distillation fine-tuning can align modalities while preserving each teacher's knowledge. Invoked in Section 3.3, Stage 1.
  • domain assumption The selected benchmarks are accurate and reliable measures of multimodal understanding and generation quality.
    Claims of state-of-the-art are based on SEED, POPE, MVBench, and VBench; the paper provides no external validation or error analysis. Used throughout Section 4.
  • ad hoc to paper Amplitude mismatch between visual and noise features is the main cause of training difficulty.
    Stated in Section 3.2 under Feature Pre-scaling; motivation is empirical and specific to this training configuration.
invented entities (1)
  • Multimodal AdaLN
    purpose: Input-dependent scale, shift, and gate normalization to harmonize text, vision, and noise features within one transformer.
    New normalization module introduced in Section 3.2; evaluated only through internal ablations and loss curves, with no external predictions or formal analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation." pith.science (2026). https://pith.science/paper/P3IBQU7W

@misc{pith2026250602975,
  author       = {Pith},
  title        = {Pith review of: HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P3IBQU7W}},
  note         = {Machine review of arXiv:2506.02975}
}
read the original abstract

With the advancement of language models, unified multimodal understanding and generation have made significant strides, with model architectures evolving from separated components to unified single-model frameworks. This paper explores an efficient training paradigm to build a single transformer for unified multimodal understanding and generation. Specifically, we propose a multimodal warmup strategy utilizing prior knowledge to extend capabilities. To address cross-modal compatibility challenges, we introduce feature pre-scaling and multimodal AdaLN techniques. Integrating the proposed technologies, we present the HaploOmni, a new single multimodal transformer. With limited training costs, HaploOmni achieves competitive performance across multiple image and video understanding and generation benchmarks over advanced unified models. All codes will be made public at https://github.com/Tencent/HaploVLM.

Figures

Figures reproduced from arXiv: 2506.02975 by the authors.

Figure 1
Figure 1. Illustration of our HaploOmni-attention mechanism and HaploOmni Block. We implement [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The progressive training stages of our HaploOmni, including multimodal warmup, connector [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison on image and video understanding capability. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance comparison on video generation capabilities. The resolution of the generated [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visualization results for the ablation of different components. (d) indicates the final version of our model. 0.3 0.7Diffusion Loss W/O Multimodal LN W/ Multimodal LN 0.0 8.0NTP Loss W/O Pre-scaling W/ Pre-scaling (a) Ablation on Pre-Connector (b) Ablation on Post-Conn…
Figure 7
Figure 7. Figure 7: Qualitative results of HaploOmni. The resolution of all the generated videos is 480 ×720. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: More qualitative results about video generation. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 26 canonical work pages

  1. [1]

    arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Al- tenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  3. [3]

    In: IEEE International Conference on Computer Vision (2021)

    Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end retrieval. In: IEEE International Conference on Computer Vision (2021)

    2021

  4. [4]

    Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., Ta¸ sırlar, S.: Introducing our multimodal models (2023),https://www.adept.ai/blog/fuyu-8b

    work page 2023

  5. [5]

    arXiv preprint arXiv:2310.19512 (2023)

    Chen, H., Xia, M., He, Y ., Zhang, Y ., Cun, X., Yang, S., Xing, J., Liu, Y ., Chen, Q., Wang, X., et al.: Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512 (2023)

    arXiv 2023

  6. [6]

    arXiv:2311.12793 (2023)

    Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., Lin, D.: Sharegpt4v: Improving large multi-modal models with better captions. arXiv:2311.12793 (2023)

    arXiv 2023

  7. [7]

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y ., Chen, Z., Duan, H., Wang, J., Qiao, Y ., Lin, D., et al.: Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330 (2024)

    arXiv 2024

  8. [8]

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling (2025)

    work page 2025

Show all 70 references
  1. [9]

    In: CVPR (2024)

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: CVPR (2024)

    2024

  2. [10]

    arXiv preprint arXiv:2402.03766 (2024)

    Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y ., Sun, X., Hu, Y ., Lin, X., Zhang, B., et al.: Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766 (2024)

    2024 arXiv

  3. [11]

    In: NeurIPS (2023)

    Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.C.H.: Instructblip: Towards general-purpose vision-language models with instruction tuning. In: NeurIPS (2023)

    2023

  4. [12]

    arXiv:2406.11832 (2024)

    Diao, H., Cui, Y ., Li, X., Wang, Y ., Lu, H., Wang, X.: Unveiling encoder-free vision-language models. arXiv:2406.11832 (2024)

    2024 arXiv

  5. [13]

    arXiv preprint arXiv:2309.11499 (2023)

    Dong, R., Han, C., Peng, Y ., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., et al.: Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499 (2023)

    2023 arXiv

  6. [14]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    2020 arXiv

  7. [15]

    arXiv:2407.21783 (2024)

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv:2407.21783 (2024)

    2024 arXiv

  8. [16]

    arXiv preprint arXiv:2404.14396 (2024)

    Ge, Y ., Zhao, S., Zhu, J., Ge, Y ., Yi, K., Song, L., Li, C., Ding, X., Shan, Y .: Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396 (2024)

    2024 arXiv

  9. [17]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Goyal, Y ., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6904–6913 (2017)

    2017

  10. [18]

    arXiv preprint arXiv:2307.04725 (2023) 12

    Guo, Y ., Yang, C., Rao, A., Liang, Z., Wang, Y ., Qiao, Y ., Agrawala, M., Lin, D., Dai, B.: Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 (2023) 12

    2023 arXiv

  11. [19]

    arXiv preprint arXiv:2211.13221 (2022)

    He, Y ., Yang, T., Zhang, Y ., Shan, Y ., Chen, Q.: Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 (2022)

    2022 arXiv

  12. [20]

    Advances in Neural Information Processing Systems35, 8633–8646 (2022)

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. Advances in Neural Information Processing Systems35, 8633–8646 (2022)

    2022

  13. [21]

    arXiv preprint arXiv:2205.15868 (2022)

    Hong, W., Ding, M., Zheng, W., Liu, X., Tang, J.: Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 (2022)

    2022 arXiv

  14. [22]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huang, Z., He, Y ., Yu, J., Zhang, F., Si, C., Jiang, Y ., Zhang, Y ., Wu, T., Jin, Q., Chanpaisit, N., et al.: Vbench: Comprehensive benchmark suite for video generative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21807–2...

    2024

  15. [23]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6700–6709 (2019)

    2019

  16. [24]

    arXiv preprint arXiv:2402.03161 (2024)

    Jin, Y ., Sun, Z., Xu, K., Chen, L., Jiang, H., Huang, Q., Song, C., Liu, Y ., Zhang, D., Song, Y ., et al.: Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. arXiv preprint arXiv:2402.03161 (2024)

    2024

  17. [25]

    In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A diagram is worth a dozen images. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14. pp. 235–251. Springer (2016)

    2016

  18. [26]

    https://klingai.com/ (2024)

    Kling, a.: Kuaishou. https://klingai.com/ (2024)

    2024

  19. [27]

    arXiv:2408.03326 (2024)

    Li, B., Zhang, Y ., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y ., Liu, Z., Li, C.: Llava-onevision: Easy visual task transfer. arXiv:2408.03326 (2024)

    2024 arXiv

  20. [29]

    arXiv preprint arXiv:2307.16125 (2023)

    Li, B., Wang, R., Wang, G., Ge, Y ., Ge, Y ., Shan, Y .: Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023)

    2023 arXiv

  21. [30]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, K., Wang, Y ., He, Y ., Li, Y ., Wang, Y ., Liu, Y ., Wang, Z., Xu, J., Chen, G., Luo, P., et al.: Mvbench: A comprehensive multi-modal video understanding benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22195–22206 (2024)

    2024

  22. [31]

    In: European Conference on Computer Vision

    Li, Y ., Wang, C., Jia, J.: Llama-vid: An image is worth 2 tokens in large language models. In: European Conference on Computer Vision. pp. 323–340. Springer (2025)

    2025

  23. [32]

    arXiv preprint arXiv:2305.10355 (2023)

    Li, Y ., Du, Y ., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023)

    2023 arXiv

  24. [33]

    arXiv preprint arXiv:2412.00131 (2024)

    Lin, B., Ge, Y ., Cheng, X., Li, Z., Zhu, B., Wang, S., He, X., Ye, Y ., Yuan, S., Chen, L., et al.: Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00131 (2024)

    2024 arXiv

  25. [34]

    arXiv preprint arXiv:2311.10122 (2023)

    Lin, B., Ye, Y ., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    2023 arXiv

  26. [35]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., Han, S.: Vila: On pre-training for visual language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26689–26699 (2024)

    2024

  27. [36]

    arXiv preprint (2024)

    Liu, H., Yan, W., Zaharia, M., Abbeel, P.: World model on million-length video and language with ringattention. arXiv preprint (2024)

    2024

  28. [37]

    In: CVPR (2024) 13

    Liu, H., Li, C., Li, Y ., Lee, Y .J.: Improved baselines with visual instruction tuning. In: CVPR (2024) 13

    2024

  29. [38]

    Liu, H., Li, C., Li, Y ., Li, B., Zhang, Y ., Shen, S., Lee, Y .J.: Llava-next: Improved reasoning, ocr, and world knowledge (2024)

    2024

  30. [39]

    In: NeurIPS (2024)

    Liu, H., Li, C., Wu, Q., Lee, Y .J.: Visual instruction tuning. In: NeurIPS (2024)

    2024

  31. [40]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  32. [41]

    Advances in Neural Information Processing Systems 36, 46212–46244 (2023)

    Mangalam, K., Akshulakov, R., Malik, J.: Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems 36, 46212–46244 (2023)

    2023

  33. [42]

    openai: Sora (2024),https://openai.com/sora

    2024

  34. [43]

    Advances in neural information processing systems35, 27730–27744 (2022)

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in neural information processing systems35, 27730–27744 (2022)

    2022

  35. [44]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4195–4205 (2023)

    2023

  36. [45]

    https://pika.art/home/ (2023)

    Pika, L.: Pika. https://pika.art/home/ (2023)

    2023

  37. [46]

    arXiv preprint arXiv:2307.01952 (2023)

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)

    2023 arXiv

  38. [47]

    arXiv preprint arXiv:2412.03069 (2024)

    Qu, L., Zhang, H., Liu, Y ., Wang, X., Jiang, Y ., Gao, Y ., Ye, H., Du, D.K., Yuan, Z., Wu, X.: Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069 (2024)

    2024 arXiv

  39. [48]

    https://runwayml.com/research/introducing-gen-3-alpha/ (2024)

    Runway: Gen-3 alpha: A new frontier for video generation. https://runwayml.com/research/introducing-gen-3-alpha/ (2024)

    2024

  40. [49]

    Advances in Neural Information Processing Systems36 (2024)

    Shen, Y ., Song, K., Tan, X., Li, D., Lu, W., Zhuang, Y .: Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems36 (2024)

    2024

  41. [50]

    arXiv preprint arXiv:2010.02502 (2020)

    Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)

    2020 arXiv

  42. [51]

    Advances in Neural Information Processing Systems36(2024)

    Sun, K., Pan, J., Ge, Y ., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y ., et al.: Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems36(2024)

    2024

  43. [52]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sun, Q., Cui, Y ., Zhang, X., Zhang, F., Yu, Q., Wang, Y ., Rao, Y ., Liu, J., Huang, T., Wang, X.: Generative multimodal models are in-context learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14398–14409 (2024)

    2024

  44. [53]

    arXiv preprint arXiv:2405.09818 (2024)

    Team, C.: Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 (2024)

    2024 arXiv

  45. [54]

    arXiv preprint arXiv:2409.18869 (2024)

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y ., Wang, J., Zhang, F., Wang, Y ., Li, Z., Yu, Q., et al.: Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869 (2024)

    2024 arXiv

  46. [55]

    arXiv preprint arXiv:2410.13848 (2024)

    Wu, C., Chen, X., Wu, Z., Ma, Y ., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al.: Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848 (2024)

    2024 arXiv

  47. [56]

    arXiv preprint arXiv:2309.05519 (2023)

    Wu, S., Fei, H., Qu, L., Ji, W., Chua, T.S.: Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519 (2023)

    2023 arXiv

  48. [57]

    arXiv preprint arXiv:2409.04429 (2024) 14

    Wu, Y ., Zhang, Z., Chen, J., Tang, H., Li, D., Fang, Y ., Zhu, L., Xie, E., Yin, H., Yi, L., et al.: Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429 (2024) 14

    2024 arXiv

  49. [58]

    arXiv preprint arXiv:2409.11340 (2024)

    Xiao, S., Wang, Y ., Zhou, J., Yuan, H., Xing, X., Yan, R., Wang, S., Huang, T., Liu, Z.: Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340 (2024)

    2024 arXiv

  50. [59]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xiao, Y ., Luo, Z., Liu, Y ., Ma, Y ., Bian, H., Ji, Y ., Yang, Y ., Li, X.: Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18709...

    2024

  51. [60]

    arXiv preprint arXiv:2505.13031 (2025)

    Xiao, Y ., Song, L., Chen, Y ., Luo, Y ., Chen, Y ., Gan, Y ., Huang, W., Li, X., Qi, X., Shan, Y .: Mindomni: Unleashing reasoning generation in vision language models with rgpo. arXiv preprint arXiv:2505.13031 (2025)

    2025 arXiv

  52. [61]

    Advances in Neural Information Processing Systems37, 75329–75354 (2024)

    Xiao, Y ., Song, L., Wang, J., Song, S., Ge, Y ., Li, X., Shan, Y ., et al.: Mambatree: Tree topology is all you need in state space model. Advances in Neural Information Processing Systems37, 75329–75354 (2024)

    2024

  53. [62]

    arXiv preprint arXiv:2408.12528 (2024)

    Xie, J., Mao, W., Bai, Z., Zhang, D.J., Wang, W., Lin, K.Q., Gu, Y ., Chen, Z., Yang, Z., Shou, M.Z.: Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528 (2024)

    2024 arXiv

  54. [63]

    arXiv:2407.10671 (2024)

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al.: Qwen2 technical report. arXiv:2407.10671 (2024)

    2024 arXiv

  55. [64]

    5 technical report

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al.: Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  56. [65]

    arXiv preprint arXiv:2503.14694 (2025)

    Yang, R., Song, L., Xiao, Y ., Huang, R., Ge, Y ., Shan, Y ., Zhao, H.: Haplovl: A single- transformer baseline for multi-modal understanding. arXiv preprint arXiv:2503.14694 (2025)

    2025 arXiv

  57. [66]

    arXiv preprint arXiv:2408.06072 (2024)

    Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y ., Hong, W., Zhang, X., Feng, G., et al.: Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 (2024)

    2024 arXiv

  58. [67]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., Huang, F.: mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13040–13051 (2024)

    2024

  59. [68]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yue, X., Ni, Y ., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y ., et al.: Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  60. [69]

    arXiv preprint arXiv:2404.01258 (2024)

    Zhang, R., Gui, L., Sun, Z., Feng, Y ., Xu, K., Zhang, Y ., Fu, D., Li, C., Hauptmann, A., Bisk, Y ., et al.: Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258 (2024)

    2024 arXiv

  61. [70]

    arXiv preprint arXiv:2408.11039 (2024)

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., Levy, O.: Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039 (2024)

    2024 arXiv

  62. [71]

    arXiv:2304.10592 (2023) 15

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304.10592 (2023) 15

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.