pith. machine review for the scientific record. sign in

arxiv: 2409.04429 · v3 · submitted 2024-09-06 · 💻 cs.CV · cs.LG

Recognition: 3 theorem links

· Lean Theorem

VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

Yecheng Wu , Zhuoyang Zhang , Junyu Chen , Haotian Tang , Dacheng Li , Yunhao Fang , Ligeng Zhu , Enze Xie
show 4 more authors
Hongxu Yin Li Yi Song Han Yao Lu
Authors on Pith no claims yet

Pith reviewed 2026-05-16 00:21 UTC · model grok-4.3

classification 💻 cs.CV cs.LG
keywords unified foundation modelautoregressive predictionvisual understandingimage generationvision-language modeltoken alignmentdiffusion alternative
0
0 comments X

The pith

VILA-U integrates visual understanding and generation using a single autoregressive next-token prediction framework.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

VILA-U is a unified foundation model for video, image, and language understanding and generation. It replaces the usual separate modules for understanding and generating images with one autoregressive system that predicts the next token for both tasks. This removes the need for diffusion models and other extra components. The key enablers are a vision tower that aligns visual tokens with text during pretraining and training on high-quality data so that autoregressive generation matches diffusion quality. Readers should care because it points to simpler, more integrated architectures for multimodal AI that still deliver strong results.

Core claim

VILA-U employs a single autoregressive next-token prediction framework for both visual understanding and generation tasks. This eliminates the need for additional components like diffusion models while achieving near state-of-the-art performance. The success stems from a unified vision tower that aligns discrete visual tokens with textual inputs during pretraining and from the ability of autoregressive image generation to reach similar quality as diffusion models when using high-quality datasets.

What carries the argument

unified vision tower aligning discrete visual tokens with textual inputs during pretraining, supporting a fully token-based autoregressive framework for both understanding and generation

Load-bearing premise

That the unified vision tower sufficiently aligns discrete visual tokens with text and that autoregressive generation trained on high-quality data can achieve quality comparable to diffusion models.

What would settle it

Observing that VILA-U's generated images score substantially lower on FID or other quality metrics than diffusion models trained on equivalent data, or that understanding tasks show misalignment errors due to poor token alignment.

read the original abstract

VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation. Traditional visual language models (VLMs) use separate modules for understanding and generating visual content, which can lead to misalignment and increased complexity. In contrast, VILA-U employs a single autoregressive next-token prediction framework for both tasks, eliminating the need for additional components like diffusion models. This approach not only simplifies the model but also achieves near state-of-the-art performance in visual language understanding and generation. The success of VILA-U is attributed to two main factors: the unified vision tower that aligns discrete visual tokens with textual inputs during pretraining, which enhances visual perception, and autoregressive image generation can achieve similar quality as diffusion models with high-quality dataset. This allows VILA-U to perform comparably to more complex models using a fully token-based autoregressive framework.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript presents VILA-U, a unified foundation model integrating video, image, and language understanding and generation. It proposes a single autoregressive next-token prediction framework for both tasks, removing separate modules such as diffusion models. Success is attributed to a unified vision tower that aligns discrete visual tokens with textual inputs during pretraining and to autoregressive generation on high-quality datasets achieving quality comparable to diffusion models, with claims of near state-of-the-art performance in both understanding and generation.

Significance. If the empirical claims are substantiated with quantitative evidence, the work would be significant for demonstrating that a purely token-based autoregressive architecture can unify visual understanding and generation without hybrid components, potentially simplifying multimodal model design and reducing the need for separate generative modules while preserving performance.

major comments (3)
  1. [Abstract] Abstract: The central claim of near state-of-the-art performance and successful elimination of diffusion models is unsupported by any quantitative metrics (e.g., FID, CLIP-score, accuracy deltas), baselines, or ablation studies, leaving the assertion that the unified vision tower and high-quality dataset suffice unverified.
  2. [Method] Method section (vision tower description): The claim that the unified vision tower 'aligns discrete visual tokens with textual inputs during pretraining' is presented qualitatively without the alignment objective, loss function, or pretraining details (e.g., no equation or procedure for token-text alignment), which is load-bearing for the argument that this component compensates for discrete-token AR limitations such as error accumulation.
  3. [Experiments] Experiments or Results section: No comparative results, dataset specifications, or ablation isolating the contribution of the autoregressive framework versus dataset quality are provided to support the equivalence to diffusion models, undermining the claim that high-quality data alone enables comparable generation fidelity.
minor comments (2)
  1. [Abstract] The abstract would be clearer if it included at least one key quantitative result (e.g., a specific metric value) to ground the 'near state-of-the-art' claim.
  2. Notation for the unified vision tower and tokenization process should be defined explicitly on first use to aid readability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight areas where additional detail and evidence will strengthen the manuscript. We address each major comment below and will incorporate the requested changes in the revised version.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim of near state-of-the-art performance and successful elimination of diffusion models is unsupported by any quantitative metrics (e.g., FID, CLIP-score, accuracy deltas), baselines, or ablation studies, leaving the assertion that the unified vision tower and high-quality dataset suffice unverified.

    Authors: We agree that the abstract would benefit from explicit quantitative support. In the revision we will insert specific metrics (FID, CLIP-score, benchmark accuracies) together with direct comparisons to relevant baselines and a brief reference to the key ablation results that isolate the contribution of the unified vision tower and the high-quality training data. revision: yes

  2. Referee: [Method] Method section (vision tower description): The claim that the unified vision tower 'aligns discrete visual tokens with textual inputs during pretraining' is presented qualitatively without the alignment objective, loss function, or pretraining details (e.g., no equation or procedure for token-text alignment), which is load-bearing for the argument that this component compensates for discrete-token AR limitations such as error accumulation.

    Authors: We acknowledge that the current description lacks the necessary technical detail. We will expand the Method section to include the alignment objective, the concrete loss function (contrastive plus reconstruction terms), the pretraining schedule, and the corresponding equations that show how discrete visual tokens are aligned with text embeddings. This will clarify the mechanism by which the vision tower mitigates error accumulation in the autoregressive decoder. revision: yes

  3. Referee: [Experiments] Experiments or Results section: No comparative results, dataset specifications, or ablation isolating the contribution of the autoregressive framework versus dataset quality are provided to support the equivalence to diffusion models, undermining the claim that high-quality data alone enables comparable generation fidelity.

    Authors: We will add a dedicated comparative-results subsection that reports FID, CLIP-score, and other standard metrics against both diffusion-based and autoregressive baselines. Dataset details (size, sources, filtering criteria) will be provided, and we will include ablation tables that separately vary the autoregressive training objective and the dataset quality to quantify their individual contributions to generation fidelity. revision: yes

Circularity Check

0 steps flagged

No significant circularity; claims framed as empirical training outcomes

full rationale

The paper presents VILA-U as a trained model using a single autoregressive next-token framework plus unified vision tower. Central claims of simplification and near-SOTA performance are attributed to design choices (unified tower alignment during pretraining, high-quality dataset) and resulting empirical results, with no equations, fitted parameters, or derivations shown. No self-definitional structures, fitted inputs renamed as predictions, or load-bearing self-citations appear in the abstract or described chain. The derivation does not reduce to its inputs by construction; it remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the domain assumption that a single next-token autoregressive objective can jointly optimize visual perception and high-quality generation once visual tokens are aligned with text; no explicit free parameters or invented entities are named in the abstract.

axioms (2)
  • domain assumption A unified vision tower can align discrete visual tokens with textual inputs during pretraining to enhance perception.
    This alignment is stated as the first key factor enabling the unified framework.
  • domain assumption Autoregressive image generation on high-quality data can match diffusion-model quality without extra components.
    This is presented as the second key factor allowing the single-framework design.

pith-pipeline@v0.9.0 · 5479 in / 1387 out tokens · 82517 ms · 2026-05-16T00:21:04.442936+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation

    cs.CV 2025-03 unverdicted novelty 7.0

    Text-to-image models show significant limitations in integrating world knowledge, as measured by the new WISE benchmark and WiScore metric across 20 models.

  2. Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

    cs.CV 2024-10 unverdicted novelty 7.0

    Janus decouples visual encoding into task-specific pathways inside a single autoregressive transformer to unify multimodal understanding and generation while outperforming earlier unified models.

  3. What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion

    cs.CV 2026-05 unverdicted novelty 6.0

    Prior-Aligned AutoEncoders shape latent manifolds with spatial coherence, local continuity, and global semantics to improve latent diffusion, achieving SOTA gFID 1.03 on ImageNet 256x256 with up to 13x faster convergence.

  4. VibeToken: Scaling 1D Image Tokenizers and Autoregressive Models for Dynamic Resolution Generations

    cs.CV 2026-04 unverdicted novelty 6.0

    VibeToken enables autoregressive image generation at arbitrary resolutions using 64 tokens for 1024x1024 images with 3.94 gFID, constant 179G FLOPs, and better efficiency than diffusion or fixed AR baselines.

  5. Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs

    cs.CV 2026-04 unverdicted novelty 6.0

    IMU-to-4D uses wearable IMU data and repurposed LLMs to predict coherent 4D human motion plus coarse scene structure, outperforming cascaded state-of-the-art pipelines in temporal stability.

  6. CLEAR: Unlocking Generative Potential for Degraded Image Understanding in Unified Multimodal Models

    cs.CV 2026-04 unverdicted novelty 6.0

    CLEAR uses degradation-aware fine-tuning, a latent representation bridge, and interleaved reinforcement learning to connect generative and reasoning capabilities in multimodal models for better degraded image understanding.

  7. MMaDA: Multimodal Large Diffusion Language Models

    cs.CV 2025-05 unverdicted novelty 6.0

    MMaDA is a unified multimodal diffusion model using mixed chain-of-thought fine-tuning and a new UniGRPO reinforcement learning algorithm that outperforms specialized models in reasoning, understanding, and text-to-im...

  8. HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model

    cs.CV 2025-03 unverdicted novelty 6.0

    HybridVLA unifies diffusion and autoregression in a single VLA model via collaborative training and ensemble to raise robot manipulation success rates by 14% in simulation and 19% in real-world tasks.

  9. SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation

    cs.CV 2024-04 unverdicted novelty 6.0

    SEED-X is a unified multimodal foundation model that handles multi-granularity visual semantics for both comprehension and generation across arbitrary image sizes and ratios.

  10. SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture

    cs.CV 2026-05 unverdicted novelty 5.0

    SenseNova-U1 presents native unified multimodal models that match top understanding VLMs while delivering strong performance in image generation, infographics, and interleaved tasks via the NEO-unify architecture.

  11. Steering Visual Generation in Unified Multimodal Models with Understanding Supervision

    cs.CV 2026-05 unverdicted novelty 5.0

    Using understanding tasks as direct supervision during post-training improves image generation and editing in unified multimodal models.

  12. ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning

    cs.RO 2026-04 unverdicted novelty 5.0

    ReFineVLA adds teacher-generated reasoning steps to VLA training and reports state-of-the-art success rates on SimplerEnv WidowX and Google Robot benchmarks.

  13. Emerging Properties in Unified Multimodal Pretraining

    cs.CV 2025-05 unverdicted novelty 5.0

    BAGEL is a unified decoder-only model that develops emerging complex multimodal reasoning abilities after pretraining on large-scale interleaved data and outperforms prior open-source unified models.

  14. BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

    cs.CV 2025-05 conditional novelty 5.0

    BLIP3-o uses a diffusion transformer to generate CLIP image features and a sequential pretraining strategy to build open models that perform strongly on both image understanding and generation benchmarks.

  15. Show-o2: Improved Native Unified Multimodal Models

    cs.CV 2025-06 unverdicted novelty 4.0

    Show-o2 unifies text, image, and video understanding and generation in a single autoregressive-plus-flow-matching model built on 3D causal VAE representations.

  16. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

    cs.AI 2025-01 conditional novelty 3.0

    Scaling data, model size, and training optimization on the Janus architecture yields better multimodal understanding and more stable, instruction-following text-to-image generation.

  17. Cosmos World Foundation Model Platform for Physical AI

    cs.CV 2025-01 unverdicted novelty 3.0

    The Cosmos platform supplies open-source pre-trained world models and supporting tools for building fine-tunable digital world simulations to train Physical AI.

  18. Evolution of Video Generative Foundations

    cs.CV 2026-04 unverdicted novelty 2.0

    This survey traces video generation technology from GANs to diffusion models and then to autoregressive and multimodal approaches while analyzing principles, strengths, and future trends.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · cited by 18 Pith papers · 10 internal anchors

  1. [1]

    ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6,

    work page 2023

  3. [3]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pp. 248–255. Ieee, 2009a. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE c...

    work page 2009

  4. [4]

    Planting a seed of vision in large language model

    Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041, 2023a. Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. In The Twelfth International Conference on Learning Representations, 2023b...

    work page arXiv

  5. [5]

    Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering

    11 Published as a conference paper at ICLR 2025 Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR) ,

    work page 2025

  6. [6]

    Classifier-Free Diffusion Guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Mixtral of Experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv:2401.04088,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization

    Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. arXiv preprint arXiv:2402.03161,

    work page arXiv

  10. [10]

    SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 , 2023a. Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image gen...

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730–19742. PMLR, 2023b. Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gem...

    work page arXiv

  12. [12]

    Evaluating text-to-visual generation with image-to-text generation

    12 Published as a conference paper at ICLR 2025 Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291,

    work page arXiv 2025

  13. [13]

    World model on million-length video and language with ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268, 2024a. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b. Jiasen Lu, Christopher Clark, Sangho Lee, Ziche...

    work page arXiv

  14. [14]

    Deem: Diffusion models serve as the eyes of large language models for image perception

    Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405.15232,

    work page arXiv

  15. [15]

    OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

    URL https: //arxiv.org/abs/2406.06525. Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286, 2023a. Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hong...

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer

    13 Published as a conference paper at ICLR 2025 Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint arXiv:2401.10208, 2024a. Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng,...

    work page arXiv 2025

  19. [19]

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen

    URL https://proceedings.neurips.cc/paper_files/paper/2017/ file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694,

    work page arXiv 2017

  20. [20]

    Show-o: One Single Transformer to Unify Multimodal Understanding and Generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627,

    work page arXiv

  23. [23]

    Scaling autoregressive multi-modal models: Pretraining and instruction tuning

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2(3), 2023a. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wa...

    work page arXiv

  24. [24]

    Anygpt: Unified multimodal llm with discrete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with discrete sequence modeling. arXiv preprint arXiv:2402.12226,

    work page arXiv

  25. [25]

    Open-sora: Democratizing efficient video production for all, march

    14 Published as a conference paper at ICLR 2025 Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, march

    work page 2025

  26. [26]

    However, as these tokenizers are trained solely with a reconstruction objective, the resulting tokens lack rich semantic information

    15 Published as a conference paper at ICLR 2025 APPENDIX A D IFFERENCE WITH RELATED WORKS Prior to VILA-U, unified visual language models were dominated by two mainstream approaches: (1) Represented by LWM, CM3Leon and Show-o which utilizes a VQGAN-based tokenizer to convert visual inputs into discrete tokens. However, as these tokenizers are trained sole...

    work page 2025

  27. [27]

    We present qualitative reconstruction results in Figure 8 for our 256 / 384 resolution vision tower

    B Q UALITATIVE RESULTS B.1 R ECONSTRUCTION Original Image Reconstruction Image (256 resolution) Reconstruction Image (384 resolution) Figure 8: Visualization of the reconstruction results from text-aligned discrete visual tokens. We present qualitative reconstruction results in Figure 8 for our 256 / 384 resolution vision tower. These vision towers effect...

    work page 2025

  28. [28]

    VILA-U successfully answers the questions accurately. 17 Published as a conference paper at ICLR 2025 B.3 I N-CONTEXT LEARNING EXAMPLES Input few images + target image Output Underground Congress Soulomes 2+1=3 5+6=11 3x6=18 Romanticism Surrealism Impressionism The company is famous for its search engine. The company is famous for iPhone and Mac. The comp...

    work page 2025

  29. [29]

    VILA-U exhibits good in-context learning capabilties. B.4 V ISUAL GENERATION A snowy mountain.: An oil painting of a garden where every flower is in full bloom, showcasing a rainbow of colors.: A cube made of denim: An extreme close-up of an gray- haired man with a beard in his 60s: An elephant walking under The sea: Knolling of a drawing tools for painter...

    work page 2025