Pith. sign in

REVIEW 4 major objections 4 minor 94 references

UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read UniCode² claims that a 500K-entry codebook built by clustering SigLIP patch embeddings, paired with a frozen-plus-trainable cascaded design, makes large-scale visual tokenization stable and semantically rich enough to lift unified…

desk verdict Solid empirical scaling story for codebook-based MLLMs; the missing continuous-feature control is a real gap but not a reason to desk reject. read the letter →

arxiv 2506.20214 v2 pith:QDM2HLO2 submitted 2025-06-25 cs.CV cs.MM

Yanzhe Chen (Yen-chieh Chan) , Huasong Zhong , Yan Li , Zhenheng Yang This is my paper
classification cs.CVcs.MM
keywords 500K-entryvisualcodebookcascadedcodebooksSigLIPsemanticembeddingsdiscretetokensmultimodalunderstandingandgenerationutilizationdiffusiondecoderintegration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to show that the visual vocabulary used to turn images into discrete tokens—a codebook—can be made very large without becoming unstable or unused. Instead of learning the vocabulary from pixel reconstruction, it clusters image patches from a pretrained vision-language model, producing 500,000 entries that inherit language-aligned meaning. A two-stage design keeps that large vocabulary stable: a frozen stage picks the discrete token ID, and a trainable stage adapts the ID into an embedding for the language model. The authors report about 99% of the 500,000 entries are actually used, with understanding scores that beat a 14B discrete-token baseline while using a 7B backbone, and text-to-image generation that works by feeding predicted tokens into a frozen diffusion decoder. This would matter because the small, fragile vocabularies of earlier codebook models were a major reason discrete-token systems lagged behind continuous-visual systems.

What carries the argument

The load-bearing object is the cascaded codebook pair. The first stage is a frozen 500K-entry codebook $\mathcal{C}^{(1)}$ obtained by K-means clustering of patch embeddings from a SigLIP vision encoder; it assigns each image patch to its nearest centroid by $\arg\min_k \|\mathbf{z}_i - \mathbf{c}^{(1)}_k\|^2$, producing stable discrete token IDs. The second stage is a trainable codebook $\mathcal{C}^{(2)}$ initialized from the same centroids, which re-embeds each ID into the vector space the language model consumes. This split means indexing never updates—so dead-code and partial-update instability do not arise—while adaptation still happens. The same token IDs are generated autoregressively for synthesis, and a small learned map $M_\phi$ sends their embeddings into a frozen diffusion decoder's latent space, supervised by a contrastive loss against the prompt text embedding.

What would settle it

Train the same pipeline with the frozen 500K codebook replaced by a frozen randomly initialized codebook of identical size and identical training budget, then compare understanding benchmark scores and token-to-image reconstruction quality; if the random codebook matches the semantic one after the trainable stage adapts, the SigLIP clustering is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The paper's discovery is that a 500K-entry codebook does not have to collapse if its entries are first built by clustering SigLIP patch embeddings and then held frozen, while a second trainable codebook with the same vocabulary provides task-specific refinement. In the reported experiments, this cascaded setup keeps utilization near 99% at codebook sizes of 10K, 100K, and 500K, whereas a reconstruction-trained VQGAN codebook falls to 2.3% utilization at 500K and its benchmark scores drop as the vocabulary grows. Equipped with the large semantic codebook, the 7B model reports the best understanding results among the discrete-input models it compares against on SEED-Bench, GQA, TextVQA, AI2D, RealWorldQA, MMMU, and MME, beating the 14B TokenFlow baseline on most of them and approaching continuous-input models. For generation, predicted token IDs are mapped into the latent space of frozen FLUX or SDXL decoders, and after training only a lightweight mapping on 200K image-caption pairs the system reaches GenEval 0.65 and DPG-Bench 83.5 overall, comparable to strong diffusion and hybrid systems.

Load-bearing premise

The construction depends on the 500,000 clusters of a pretrained vision-language model's image-patch embeddings preserving enough visual detail—exact colors, small text, spatial layout—that nothing needed for understanding or generation is lost when every patch is replaced by its nearest cluster ID.

Editorial extensions

If this is right

  • Codebook size can be scaled roughly thirtyfold over typical 16K-entry visual vocabularies while keeping utilization near 99%, removing the main practical barrier to large discrete token spaces.
  • Discrete-token unified models can match continuous-input models closely enough on understanding benchmarks that the autoregressive next-token paradigm remains a viable single framework for both modalities.
  • A semantically aligned token space acts as a reusable interface: new pretrained diffusion decoders can be attached by training only a small projection, without fine-tuning the decoder or the language model.
  • Separating indexing from adaptation stabilizes training, so the codebook itself can stay fixed while downstream tasks refine only the token embeddings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same offline clustering recipe could be rerun with a different vision-language encoder or patch resolution to test whether the gains come from SigLIP semantics specifically or from large cluster count; the paper reports only the SigLIP version.
  • Because the first-stage codebook is frozen, a domain-specific codebook could in principle be swapped in for medical or satellite imagery, though token IDs would change and the language model would need a remapped or extended vocabulary.
  • The paper's information-theoretic argument predicts that measured mutual information between token IDs and object or attribute labels should be higher for this codebook than for reconstruction-trained codebooks; that prediction is untested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes UniCode^2, a two-stage cascaded visual codebook for unified multimodal understanding and generation. It builds a 500K-entry codebook by k-means clustering of SigLIP patch embeddings from ~558K images, freezes this first-stage codebook for nearest-neighbor quantization, and uses a trainable second-stage embedding lookup for task-specific refinement. A Qwen2.5-7B LLM consumes the resulting discrete visual tokens for understanding; for generation, the LLM autoregressively predicts visual token IDs, which are mapped by a learned projection and used to condition frozen SDXL or FLUX diffusion decoders. The paper reports strong benchmark results, ~99% codebook utilization at 500K, ablations over codebook size and codebook configuration, and plug-and-play generation results with minimal training data.

Significance. If the central claim holds, this is a useful result for discrete-token unified MLLMs: it shows that a 500K semantic codebook can avoid the utilization collapse and instability that plague reconstruction-trained VQGAN-style tokenizers, and that semantically aligned tokens can be attached to frozen diffusion decoders cheaply. The utilization and ablation tables (Tables 3 and 4) are informative, and the plug-and-play generation experiments with FLUX and SDXL are a strength. However, the paper does not yet establish that the quantization step preserves the information needed for understanding and generation, and the generation pathway's conditioning mechanism is underspecified; both points need work before the broad claims are supported.

major comments (4)
  1. [§3.3, Eq. (5); Table 4] The ablation labeled 'Train Encoder (No Codebook)' does not provide the clean control needed to support the claim that 500K-entry quantization preserves semantic information. That setting changes two variables at once: the encoder is trainable and the trainable parameter count is 3.3 GB versus 2.0 GB for the cascaded configuration, so the 75.6 versus 77.4 average could reflect optimization or capacity effects rather than the absence of quantization loss. A proper control should keep the frozen SigLIP encoder and the Qwen2.5 backbone fixed and feed the continuous patch embeddings through the same MLP projector, removing only the codebook. Without such a control, the paper's core claim that scaling visual token spaces 'without sacrificing semantics' is empirically incomplete, because both understanding (Eq. 7) and generation (Eqs. 8-10) consume the same quantized tokens.
  2. [§3.1, Eq. (1); §3.2; Table 3] The motivating objective in Eq. (1) includes a utilization-regularization term, but the actual codebook construction is plain (hierarchical) k-means on SigLIP embeddings, which minimizes only the semantic-alignment term and does not directly enforce balanced cluster sizes or assignment entropy. The paper should clarify whether any entropy or balance regularization is actually applied during clustering, and it should report cluster-size statistics rather than only the single utilization number. In addition, utilization in Table 3 is computed on the validation set, and the paper does not specify the overlap between that validation set and the 558K images used for clustering, so the 99% figure is not yet fully interpretable as a property of the codebook rather than of evaluation-set overlap.
  3. [§3.5, Eqs. (9)-(10); Table 2] The generation pathway is underspecified at a load-bearing point: the mapping M_phi is trained only with the contrastive loss in Eq. (10), which aligns a sequence-level embedding to the prompt embedding, and no reconstruction, diffusion, or per-token spatial loss is described. The paper does not state how the mapped per-token vectors e_z_t are arranged or aggregated to condition the frozen FLUX/SDXL decoder, nor whether the same frozen decoder is used for all generation results in Tables 2 and 5. Because the strong GenEval and DPG-Bench numbers rest on this mechanism, please provide the exact architecture, the objective used to train M_phi, and a comparison against directly interpolating frozen SigLIP features or a linear map from the LLM's hidden states.
  4. [§3.2; §1 (contributions)] The claim that the codebook 'preserves vision-language alignment' is inherited from the pretrained SigLIP encoder, and the paper does not independently quantify what the nearest-centroid quantization preserves. A semantic codebook evaluation (e.g., linear-probe classification or retrieval on the discrete token IDs, nearest-neighbor consistency in the codebook space, or reconstruction error on the SigLIP features) would make the central claim falsifiable and would also address the concern that within-cluster variation is irreversibly discarded. The current qualitative token-to-patch visualization (Figure 5) is suggestive but not sufficient.
minor comments (4)
  1. [§3.1, Eq. (2)] The lower bound in Eq. (2) writes the expected quantization error as at least (1/K) times the sum of intra-cluster variances; this is not generally true because the expectation weights each cluster by its assignment probability q(k), and the codewords may not coincide with conditional means. Appendix A.3 gives the correct weighted expression in Eq. (12), so Eq. (2) should be corrected or removed.
  2. [§1, contributions] The contribution list contains a typo, 'Emperimental Validation', which should read 'Experimental Validation'.
  3. [§4.2, Table 2 and surrounding text] The benchmark is referred to as 'DPO-Bench' in the text and 'DPG-Bench' in the table; this should be made consistent (the benchmark is DPG-Bench).
  4. [§4.3, Table 3; A.1] The definition of utilization as the fraction of codewords activated at least once over the validation set should be supplemented with the codebook's training-set utilization and the distribution of per-code frequencies; otherwise the 99% figure is hard to compare with prior utilization metrics.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: external SigLIP features, explicit contrastive adaptation, and external benchmarks support the claims; minor naming and control issues do not make the derivation circular.

full rationale

UniCode²'s load-bearing computations are not equivalent to their inputs. The 500K codebook is built by K-means on externally pretrained SigLIP patch embeddings (§3.2), and the semantic alignment of the resulting tokens is explicitly attributed to SigLIP rather than derived from the codebook objective. Understanding results (Table 1) are evaluated on external benchmarks (SEEDB, GQA, MMMU, etc.) using a frozen SigLIP encoder plus Qwen2.5 backbone; generation results (Table 2) are evaluated on GenEval and DPG-Bench, which are external to the 200K JourneyDB contrastive training pairs. The contrastive loss L_CL (Eq. 10) is an explicit adaptation that fits a mapping to the frozen diffusion text encoder; evaluating on held-out prompts is a legitimate generalization test, not a fitted quantity renamed as a prediction. Utilization numbers are direct counts of activated codewords, and the codebook-size ablation compares against VQGAN codebooks trained with a different objective, so the comparison is informative rather than circular. The main caveats are: (i) Eq. 1 labels the K-means distortion term 'semantic alignment,' which is a definitional naming choice; the vision-language semantics are inherited from SigLIP and not produced by the clustering objective. (ii) The 'Train Encoder (No Codebook)' ablation changes both the trainable encoder and the parameter count relative to the cascaded run, and no quantization-error/reconstruction analysis is provided, so the information-preservation claim is under-tested. These are experimental-control and framing weaknesses, not circular reductions. Self-citations (e.g., Show-O [73], MAI [12]) appear only as background related work and are not load-bearing for the central claims. The stated limitation (fixed codebook size) is likewise orthogonal to circularity. Accordingly, no circular step meets the evidentiary bar; score 2 reflects the minor definitional/naming issue and the self-citation presence, not a circular derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the cascaded codebook and mapping are architectural modules, not invented entities in the sense of new particles or forces.

free parameters (4)
  • Codebook size K = 500,000
    Target vocabulary size chosen by hand; the paper's central claim depends on this scale, and the ablation only tests 10K/100K/500K without a principled selection.
  • Number of coarse clusters (K1) = 1,000
    Used in the two-stage FAISS k-means; a design choice that affects clustering quality and codebook coverage.
  • Contrastive temperature (tau) = not reported
    Hyperparameter in Eq. (10) for the contrastive loss in the generation pathway; its value is not specified.
  • Diffusion guidance scale = 3.5 (FLUX); default (SDXL)
    Inference hyperparameter for the diffusion decoder; affects synthesis quality and is fixed during evaluation.
assumptions (5)
  • domain assumption SigLIP patch embeddings are semantically aligned with language and provide a suitable feature space for semantic tokenization.
    The entire codebook construction (§3.2) and frozen quantization codebook (§3.3) rely on this. The paper does not assess how well SigLIP captures fine-grained visual details needed for generation.
  • domain assumption K-means clustering on ~150M patch embeddings yields 500K centroids that cover the visual manifold without collapse.
    The paper assumes the two-stage FAISS clustering is sufficient and only removes NaN/Inf centroids, but does not analyze cluster quality or coverage.
  • domain assumption Pretrained diffusion decoders (FLUX, SDXL) have latent spaces that can be effectively conditioned by the learned mapping M_phi.
    The plug-and-play generation claim depends on the existence of a mapping from UniCode2 token embeddings to the diffusion conditioning space, learned via contrastive loss on 200K pairs.
  • standard math Classical rate-distortion and entropy inequalities (Eqs. 2, 15, 17) hold.
    Used in the motivation (§3.1 and A.3) but not load-bearing for the empirical results; they are standard results.
  • domain assumption The LLaVA-OneVision and JourneyDB training data cover the benchmark evaluation domains.
    The model is trained on these open-source corpora and evaluated on separate benchmarks; distribution shift could affect results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation." pith.science (2026). https://pith.science/paper/QDM2HLO2

@misc{pith2026250620214,
  author       = {Pith},
  title        = {Pith review of: UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDM2HLO2}},
  note         = {Machine review of arXiv:2506.20214}
}
abstract

Unified multimodal large language models (MLLMs) have shown promise in jointly advancing multimodal understanding and generation, with visual codebooks discretizing images into tokens for autoregressive modeling. Existing codebook-based methods either rely on small vocabularies (~16K entries) that lack fine-grained semantics or naively scale up, resulting in low token utilization and unstable training. We propose UniCode$^2$, a cascaded codebook framework enabling large-scale, semantically aligned, and stable visual tokenization. By clustering millions of SigLIP sequence embeddings, we build a 500K-entry codebook that preserves vision-language alignment while expanding capacity. Stability is ensured via a cascaded design: a frozen codebook anchors the embedding space, and a trainable codebook refines task-specific semantics. This decoupling promotes high utilization and robust learning. Moreover, the alignment of our visual tokens with textual semantics enables seamless integration with pretrained diffusion decoders, supporting high-quality visual synthesis with minimal adaptation. UniCode^2 delivers strong performance across diverse benchmarks, demonstrating the viability of scaling visual token spaces without sacrificing stability, semantics, or modularity.

Figures

Figures reproduced from arXiv: 2506.20214 by the authors.

Figure 1
Figure 1. (a) Comparison among four models across various understanding benchmarks. (b) Motivation: existing methods suffer sharp drops in performance and utilization as codebook size grows; our proposed UniCode² maintains ~99% utilization and strong performance. The limitations above call for rethinking the objective of codebook construction—from mere com￾pression to a mechanism for semantic abstraction and cross-modal align… view at source ↗
Figure 2
Figure 2. UniCode² framework overview. (a) High-capacity codebook initialized by clustering large￾scale patch embeddings. (b) During training, visual inputs are tokenized via cascaded codebooks and optimized via unified autoregressive and contrastive objectives. (c) Inference supports both understanding and generation, with visual tokens seamlessly decoded via diffusion models. where c¯ is the codebook centroid, and λ > 0 con… view at source ↗
Figure 3
Figure 3. Images generated in a resolution of 512 × 512 with UniCode². Plug-in Generation Quality. Beyond the results in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison between UniCode² and FLUX on text-to-image generation. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Token-to-patch mapping comparison between UniCode² and Chameleon. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

94 extracted references · 13 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Foundation models defining a new era in vision: a survey and outlook.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundation models defining a new era in vision: a survey and outlook.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  3. [3]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  5. [5]

    Factorized visual tokenization and generation.arXiv preprint arXiv:2411.16681, 2024

    Zechen Bai, Jianxiong Gao, Ziteng Gao, Pichao Wang, Zheng Zhang, Tong He, and Mike Zheng Shou. Factorized visual tokenization and generation.arXiv preprint arXiv:2411.16681, 2024

    arXiv 2024

  6. [6]

    Beit: Bert pre-training of image trans- formers.arXiv preprint arXiv:2106.08254, 2021

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers.arXiv preprint arXiv:2106.08254, 2021

    arXiv 2021

  7. [7]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  8. [8]

    Efficient-vqgan: Towards high-resolution image generation with efficient vision transformers

    Shiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu, Xin Zhao, Deli Zhao, and Kaigi Huang. Efficient-vqgan: Towards high-resolution image generation with efficient vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7368–7377, 2023

    2023

Show all 94 references
  1. [9]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    arXiv

  2. [10]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision, pages 370–387. Springer, 2024

    2024

  3. [11]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    2024 arXiv

  4. [12]

    Mai: A multi-turn aggregation- iteration model for composed image retrieval

    Yanzhe Chen, Zhiwen Yang, Jinglin Xu, and Yuxin Peng. Mai: A multi-turn aggregation- iteration model for composed image retrieval. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  5. [13]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  6. [14]

    Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation.arXiv preprint arXiv:2503.06764, 2025

    Zisheng Chen, Chunwei Wang, Xiuwei Chen, Hang Xu, Jianhua Han, and Xiaodan Liang. Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation.arXiv preprint arXiv:2503.06764, 2025

    2025

  7. [15]

    Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation.arXiv preprint arXiv:2407.06135, 2024

    Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation.arXiv preprint arXiv:2407.06135, 2024

    2024 arXiv

  8. [16]

    Dreamllm: Synergistic multimodal comprehension and creation.arXiv preprint arXiv:2309.11499, 2023

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation.arXiv preprint arXiv:2309.11499, 2023. 10

    2023 arXiv

  9. [17]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  10. [18]

    Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171, 2024

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171, 2024

    2024

  11. [19]

    Eva: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, page...

    2023

  12. [20]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  13. [21]

    Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    2023

  14. [22]

    A survey on self-supervised learning: Algorithms, applications, and future trends.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  15. [23]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    2024 arXiv

  16. [24]

    Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement.arXiv preprint arXiv:2504.01934, 2025

    Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, et al. Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement.arXiv preprint arXiv:2504.01934, 2025

    2025 arXiv

  17. [25]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  18. [26]

    Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding.arXiv preprint arXiv:2504.04423, 2025

    Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding.arXiv preprint arXiv:2504.04423, 2025

    2025 arXiv

  19. [27]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. S...

    2016

  20. [28]

    Black Forest Labs.https://github.com/black-forest-labs/flux, 2 2024

    2024

  21. [29]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523–11532, 2022

    2022

  22. [30]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  23. [31]

    Seed- bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  24. [32]

    A survey of multimodel large language models

    Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. InProceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pages 405–409, 2024. 11

    2024

  25. [33]

    Toklip: Marry visual tokens to clip for multimodal comprehension and generation, 2025

    Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. Toklip: Marry visual tokens to clip for multimodal comprehension and generation, 2025

    2025

  26. [34]

    World model on million-length video and language with blockwise ringattention.arXiv preprint arXiv:2402.08268, 2024

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention.arXiv preprint arXiv:2402.08268, 2024

    2024 arXiv

  27. [35]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  28. [36]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  29. [37]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  30. [38]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  31. [39]

    Ovis: Structural embedding alignment for multimodal large language model.arXiv preprint arXiv:2405.20797, 2024

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model.arXiv preprint arXiv:2405.20797, 2024

    2024 arXiv

  32. [40]

    Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321, 2025

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321, 2025

    2025

  33. [41]

    Janusflow: Harmonizing autoregres- sion and rectified flow for unified multimodal understanding and generation.arXiv preprint arXiv:2411.07975, 2024

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregres- sion and rectified flow for unified multimodal understanding and generation.arXiv preprint arXiv:2411.07975, 2024

    2024 arXiv

  34. [42]

    Beit v2: Masked image modeling with vector-quantized visual tokenizers.arXiv preprint arXiv:2208.06366, 2022

    Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers.arXiv preprint arXiv:2208.06366, 2022

    2022 arXiv

  35. [43]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  36. [44]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation.arXiv preprint arXiv:2412.03069, 2024

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation.arXiv preprint arXiv:2412.03069, 2024

    2024 arXiv

  37. [45]

    Be- yond next-token: Next-x prediction for autoregressive visual generation.arXiv preprint arXiv:2502.20388, 2025

    Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Be- yond next-token: Next-x prediction for autoregressive visual generation.arXiv preprint arXiv:2502.20388, 2025

    2025 arXiv

  38. [46]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  39. [47]

    Taming scalable visual tokenizer for autoregressive image generation.arXiv preprint arXiv:2412.02692, 2024

    Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang, Ying Shan, and Limin Wang. Taming scalable visual tokenizer for autoregressive image generation.arXiv preprint arXiv:2412.02692, 2024

    2024 arXiv

  40. [48]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  41. [49]

    Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies.arXiv preprint arXiv:2503.14324, 2025

    Wei Song, Yuran Wang, Zijia Song, Yadong Li, Haoze Sun, Weipeng Chen, Zenan Zhou, Jianhua Xu, Jiaqi Wang, and Kaicheng Yu. Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies.arXiv preprint arXiv:2503.14324, 2025. 12

    2025 arXiv

  42. [50]

    Journeydb: A benchmark for generative image understanding.Advances in neural information processing systems, 36:49659–49678, 2023

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding.Advances in neural information processing systems, 36:49659–49678, 2023

    2023

  43. [51]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  44. [52]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2024

  45. [53]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023

    2023 arXiv

  46. [54]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  47. [55]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  48. [56]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction.Advances in neural information processing systems, 37:84839–84865, 2024

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction.Advances in neural information processing systems, 37:84839–84865, 2024

    2024

  49. [57]

    Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal under- standing and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    2024 arXiv

  50. [58]

    Neural discrete representation learning.Advances in neural information processing systems, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advances in neural information processing systems, 30, 2017

    2017

  51. [59]

    Discrete visual tokens of autoregression, by diffusion, and for reasoning.arXiv preprint arXiv:2505.07538, 2025

    Bohan Wang, Zhongqi Yue, Fengda Zhang, Shuo Chen, Li’an Bi, Junzhe Zhang, Xue Song, Kennard Yanting Chan, Jiachun Pan, Weijia Wu, et al. Discrete visual tokens of autoregression, by diffusion, and for reasoning.arXiv preprint arXiv:2505.07538, 2025

    2025 arXiv

  52. [60]

    Illume: Illuminating your llms to see, draw, and self-enhance.arXiv preprint arXiv:2412.06673, 2024

    Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your llms to see, draw, and self-enhance.arXiv preprint arXiv:2412.06673, 2024

    2024 arXiv

  53. [61]

    Omnitok- enizer: A joint image-video tokenizer for visual generation.Advances in Neural Information Processing Systems, 37:28281–28295, 2024

    Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitok- enizer: A joint image-video tokenizer for visual generation.Advances in Neural Information Processing Systems, 37:28281–28295, 2024

    2024

  54. [62]

    Image under- standing makes for a good tokenizer for image generation.Advances in Neural Information Processing Systems, 37:31015–31035, 2024

    Luting Wang, Yang Zhao, Zijian Zhang, Jiashi Feng, Si Liu, and Bingyi Kang. Image under- standing makes for a good tokenizer for image generation.Advances in Neural Information Processing Systems, 37:31015–31035, 2024

    2024

  55. [63]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  56. [64]

    Image as a foreign lan- guage: Beit pretraining for vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign lan- guage: Beit pretraining for vision and vision-language tasks. InProceedings of the IEEE/CVF Conference o...

    2023

  57. [65]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 13

    2024 arXiv

  58. [66]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    arXiv

  59. [67]

    Liquid: Language models are scalable multi-modal generators.arXiv preprint arXiv:2412.04332, 2024

    Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators.arXiv preprint arXiv:2412.04332, 2024

    2024 arXiv

  60. [68]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. InForty-first International Conference on Machine Learning, 2024

    2024

  61. [69]

    Harmonizing visual representations for unified multimodal understanding and generation.arXiv preprint arXiv:2503.21979, 2025

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. Harmonizing visual representations for unified multimodal understanding and generation.arXiv preprint arXiv:2503.21979, 2025

    2025 arXiv

  62. [70]

    Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024

    2024 arXiv

  63. [71]

    Grok-1.5 vision preview, 6 2024

    x.ai. Grok-1.5 vision preview, 6 2024

    2024

  64. [72]

    Omnigen: Unified image generation.arXiv preprint arXiv:2409.11340, 2024

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation.arXiv preprint arXiv:2409.11340, 2024

    2024 arXiv

  65. [73]

    Show-o: One single trans- former to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  66. [74]

    Muse-vl: Modeling unified vlm through semantic discrete encoding.arXiv preprint arXiv:2411.17762, 2024

    Rongchang Xie, Chen Du, Ping Song, and Chang Liu. Muse-vl: Modeling unified vlm through semantic discrete encoding.arXiv preprint arXiv:2411.17762, 2024

    2024 arXiv

  67. [75]

    Efficientsam: Leveraged masked image pretraining for efficient segment anything

    Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. InProceedings of the IEEE/CVF Conference on Computer...

    2024

  68. [76]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  69. [77]

    Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627, 2021

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan.arXiv preprint arXiv:2110.04627, 2021

    2021 arXiv

  70. [78]

    Language model beats diffusion–tokenizer is key to visual generation.arXiv preprint arXiv:2310.05737, 2023

    Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation.arXiv preprint arXiv:2310.05737, 2023

    arXiv

  71. [79]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  72. [80]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  73. [81]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 14

    2023

  74. [82]

    Token dynamics: Towards efficient and dynamic video token representation for video large language models.arXiv preprint arXiv:2503.16980, 2025

    Haichao Zhang and Yun Fu. Token dynamics: Towards efficient and dynamic video token representation for video large language models.arXiv preprint arXiv:2503.16980, 2025

    2025

  75. [83]

    Unified multimodal understanding and generation models: Advances, challenges, and opportunities.arXiv preprint arXiv:2505.02567, 2025

    Xinjie Zhang, Jintao Guo, Shanshan Zhao, Minghao Fu, Lunhao Duan, Guo-Hua Wang, Qing- Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. Unified multimodal understanding and generation models: Advances, challenges, and opportunities.arXiv preprint arXiv:2505.02567, 2025

    2025

  76. [84]

    Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation.arXiv preprint arXiv:2502.05178, 2025

    Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbühl, and De-An Huang. Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation.arXiv preprint arXiv:2502.05178, 2025

    2025 arXiv

  77. [85]

    Online clustered codebook

    Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22798–22807, 2023

    2023

  78. [86]

    Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  79. [87]

    Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%.arXiv preprint arXiv:2406.11837, 2024

    Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%.arXiv preprint arXiv:2406.11837, 2024

    2024 arXiv

  80. [88]

    Addressing representation collapse in vector quantized models with one linear layer.arXiv preprint arXiv:2411.02038, 2024

    Yongxin Zhu, Bocheng Li, Yifei Xin, and Linli Xu. Addressing representation collapse in vector quantized models with one linear layer.arXiv preprint arXiv:2411.02038, 2024

    2024

  81. [89]

    a blue pizza

    Jialv Zou, Bencheng Liao, Qian Zhang, Wenyu Liu, and Xinggang Wang. Omnimamba: Efficient and unified multimodal understanding and generation via state space models.arXiv preprint arXiv:2503.08686, 2025. 15 A Supplementary Material A.1 More Qualitative Analysis a photo of a blu...

    2025 arXiv

  82. [90]

    Let E={e 1, ...,eN } ⊂Rd denote visual embeddings sampled from a distribution p(e), and let C={c 1, ...,cK} ⊂Rd be a codebook with K discrete centroids

    Expected Quantization Distortion. Let E={e 1, ...,eN } ⊂Rd denote visual embeddings sampled from a distribution p(e), and let C={c 1, ...,cK} ⊂Rd be a codebook with K discrete centroids. The quantization operator Q:R d → {1, ..., K}maps each embedding to its nearest codeword: ...

  83. [91]

    Entropy Regularization and Trade-off. To prevent such collapse, we regularize the codebook with the Shannon entropy of the assignment distribution: H(q) =− KX k=1 q(k) logq(k).(13) Maximizing H(q) encourages uniform codeword usage, promoting representational diversity and bett...

  84. [92]

    Let y be a semantic target label (e.g., object class, scene type), and let v=Q(e) be the discrete token assigned to embeddinge

    Information-Theoretic Capacity. Let y be a semantic target label (e.g., object class, scene type), and let v=Q(e) be the discrete token assigned to embeddinge. The mutual information between tokens and labels satisfies: I(y;v) =H(v)−H(v|y)≤H(v) =H(q),(15) where equality holds ...

  85. [93]

    Asymptotic Distortion Bound and Uniformity. In classical high-resolution quantization theory, the optimal rate-distortion function satisfies: Dquant ≥C d ·K −2/d,(17) where Cd depends on the intrinsic dimension of the embedding space. Crucially, this bound is only achievable u...

  86. [94]

    Summary and Design Implication. Together, these results show that an optimal codebook should simultaneously minimize semantic distortion and maintain high usage entropy: C∗ = arg min C ( KX k=1 q(k)·E pk [∥e−c k∥2]−βH(q) ) .(18) Our SigLIP-based codebook construction approxima...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.