Pith. sign in

REVIEW 2 major objections 2 minor 1 cited by

Vision Foundation Models as Generalist Tokenizers for Image Generation

T0 review · 2 major / 2 minor · reviewed 2026-05-20 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A frozen vision foundation model can be used directly as the encoder for a generalist image tokenizer that operates in both discrete and continuous spaces.

desk verdict VFMTok uses a frozen VFM with region-adaptive quantization and semantic reconstruction to cut tokens and hit strong gFID numbers on ImageNet, but the no-fine-tuning assumption needs direct testing. read the letter →

arxiv 2605.18390 v1 pith:TG3ZPEBZ submitted 2026-05-18 cs.CV

Anlin Zheng , Qi Han , Xin Wen , Chuofan Ma , Lanxi Gong , Gang Yu , Xiangyu Zhang , Xiaojuan Qi This is my paper
classification cs.CV
keywords visionfoundationmodelsimagetokenizationautoregressivegenerationsynthesislatentquantizationsemanticreconstructionNetclass-conditional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes that a vision foundation model pre-trained with global contrastive learning and latent masked image modeling provides representations that work well as the basis for image tokenization without any encoder fine-tuning. The authors add a region-adaptive quantization step to cut spatial redundancy in the 2D feature grid and a semantic reconstruction objective that keeps decoded outputs aligned with the original VFM features. These changes produce VFMTok, a tokenizer usable for autoregressive models in discrete token spaces and for denoising models in continuous spaces. A sympathetic reader would care because the approach yields faster model convergence, fewer tokens, and strong generation metrics on ImageNet while removing the need for classifier-free guidance during inference.

What carries the argument

Region-adaptive quantization framework paired with a semantic reconstruction objective applied to features from a frozen vision foundation model encoder, which removes spatial redundancy while preserving semantic fidelity for downstream generation.

What would settle it

Train an otherwise identical tokenizer using a VFM pre-trained with only one of the two objectives (contrastive learning or latent masked image modeling) and check whether gFID rises above 1.36 or convergence slows below the reported three-fold speedup on the same ImageNet class-conditional task.

Watch

Extended reading notes

Core claim

VFMTok is built by taking a frozen VFM as encoder and adding region-adaptive quantization to remove spatial redundancy from 2D grid features together with a semantic reconstruction objective that aligns decoded outputs with VFM representations. This produces a generalist tokenizer that works seamlessly in discrete latent spaces for autoregressive generation and in continuous spaces for denoising-based generation. On ImageNet class-conditional synthesis the discrete version reaches a gFID of 1.36 with three times faster convergence while the continuous version reaches 1.25 gFID; both achieve high-fidelity results without classifier-free guidance.

Load-bearing premise

Representations from a VFM pre-trained with global contrastive learning plus latent masked image modeling stay optimal for tokenization and generation without any encoder fine-tuning or adaptation.

Editorial extensions

If this is right

  • Discrete autoregressive generators converge three times faster.
  • Class-conditional synthesis reaches a gFID of 1.36 on ImageNet.
  • Continuous-space generation with a denoising model reaches a gFID of 1.25.
  • High-fidelity synthesis succeeds without classifier-free guidance in both paradigms.
  • Tokenizer quality depends on the exact combination of self-supervised objectives used in VFM pre-training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same frozen-VFM approach could be tested on video or 3D data to see whether region-adaptive quantization still reduces redundancy effectively.
  • Smaller generative models paired with VFMTok might preserve quality while using even fewer parameters overall.
  • Out-of-distribution images could be used to measure how much the semantic reconstruction objective protects against domain shift.
  • Future tokenizers might be designed by first selecting VFM pre-training objectives that maximize downstream generation metrics rather than designing new quantization schemes from scratch.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes VFMTok, a generalist visual tokenizer built atop a frozen vision foundation model (VFM) encoder. It introduces region-adaptive quantization to reduce spatial redundancy in 2D grid features and a semantic reconstruction objective to align decoded outputs with VFM representations. VFMTok supports both discrete and continuous latent spaces for image generation, reporting SOTA gFID of 1.36 on ImageNet class-conditional discrete AR synthesis (with 3x faster convergence) and 1.25 for continuous denoising models, plus CFG-free generation due to rich semantics. The work also investigates SSL objectives, finding that global contrastive learning combined with latent masked image modeling yields optimal VFM representations for tokenization.

Significance. If the empirical claims hold under full verification, the results indicate that frozen VFMs can serve as effective generalist tokenizers with targeted quantization and reconstruction losses, yielding substantial gains in synthesis quality, token efficiency, and inference speed. The finding on SSL objective combinations provides concrete guidance for selecting pre-trained encoders in future tokenizer designs and could reduce the need for end-to-end training of visual encoders in generative pipelines.

major comments (2)
  1. [Abstract / VFM pre-training objectives paragraph] Abstract and § on VFM pre-training objectives: the central claim that a frozen VFM (pre-trained with global contrastive + latent MIM) remains optimal for tokenization without encoder fine-tuning is load-bearing for the reported gFID 1.36/1.25 and 3x convergence, yet no ablation compares this to joint adaptation of the encoder with the region-adaptive quantization and semantic reconstruction objectives. If joint fine-tuning better preserves spatial semantics, the efficiency and quality gains may not represent the strongest instantiation.
  2. [Experiments / Results tables] Experiments section (results on ImageNet AR and continuous generation): the gFID scores and convergence claims lack reported error bars, number of runs, or full baseline comparisons (including recent tokenizers and fine-tuned VFM variants), making it difficult to assess whether the 1.36 gFID and 3x speedup are robust or sensitive to implementation details.
minor comments (2)
  1. [Method] Notation for region-adaptive quantization could be clarified with an explicit equation or diagram showing how patch selection varies per region.
  2. [Discussion] The manuscript would benefit from a dedicated limitations paragraph discussing potential failure modes when the VFM's pre-training data distribution differs from the target generation dataset.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We are grateful to the referee for the thorough review and constructive suggestions. Below we respond to each major comment, outlining our planned revisions where appropriate.

read point-by-point responses

  1. Referee: [Abstract / VFM pre-training objectives paragraph] Abstract and § on VFM pre-training objectives: the central claim that a frozen VFM (pre-trained with global contrastive + latent MIM) remains optimal for tokenization without encoder fine-tuning is load-bearing for the reported gFID 1.36/1.25 and 3x convergence, yet no ablation compares this to joint adaptation of the encoder with the region-adaptive quantization and semantic reconstruction objectives. If joint fine-tuning better preserves spatial semantics, the efficiency and quality gains may not represent the strongest instantiation.

    Authors: We thank the referee for this observation. The manuscript's focus is on demonstrating that frozen VFMs, without any encoder fine-tuning, can serve as effective generalist tokenizers when combined with our proposed region-adaptive quantization and semantic reconstruction. This design choice emphasizes efficiency and the reusability of pre-trained models. While we acknowledge that joint fine-tuning could potentially yield further improvements, it would deviate from the generalist and frozen paradigm we aim to explore. In the revision, we will add a paragraph discussing this limitation and why the frozen setting is of particular interest, including references to works that do perform fine-tuning. This constitutes a partial revision as we will enhance the discussion but not conduct new joint fine-tuning experiments at this stage. revision: partial

  2. Referee: [Experiments / Results tables] Experiments section (results on ImageNet AR and continuous generation): the gFID scores and convergence claims lack reported error bars, number of runs, or full baseline comparisons (including recent tokenizers and fine-tuned VFM variants), making it difficult to assess whether the 1.36 gFID and 3x speedup are robust or sensitive to implementation details.

    Authors: We agree that providing error bars and details on the number of runs would enhance the credibility of the empirical results. In the revised manuscript, we will report the mean and standard deviation of gFID scores over multiple runs (specifically, we will run the experiments three times and include the statistics). We will also clarify the convergence speed measurements. For baseline comparisons, we have compared against several established tokenizers; we will expand the experimental section to include more recent methods and add a note on fine-tuned VFM variants, explaining that our work prioritizes the frozen case. These changes will be incorporated in the next version. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical tokenizer design and objective investigation are self-contained

full rationale

The paper's central results rest on constructing VFMTok atop a frozen VFM encoder, applying region-adaptive quantization and a semantic reconstruction loss, then reporting downstream gFID, convergence speed, and CFG-free generation metrics on ImageNet. These are external, falsifiable benchmarks rather than quantities derived from the paper's own equations or fitted parameters. The investigation into which VFM pre-training objectives (global contrastive + latent MIM) yield better tokenizers is likewise an empirical comparison across frozen models, not a self-referential reduction or self-citation chain. No load-bearing step equates a claimed prediction to its input by construction.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central approach rests on the domain assumption that frozen VFM representations are directly usable for tokenization once augmented with the two proposed components; no explicit free parameters or new invented entities are described in the abstract.

assumptions (1)
  • domain assumption Representations from a frozen vision foundation model pre-trained with global contrastive learning and latent masked image modeling are suitable and optimal for building a generalist image tokenizer.
    This premise is invoked when the encoder is kept frozen and when the authors conclude that specific SSL objectives dictate tokenizer effectiveness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vision Foundation Models as Generalist Tokenizers for Image Generation." pith.science (2026). https://pith.science/paper/TG3ZPEBZ

@misc{pith2026260518390,
  author       = {Pith},
  title        = {Pith review of: Vision Foundation Models as Generalist Tokenizers for Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TG3ZPEBZ}},
  note         = {Machine review of arXiv:2605.18390}
}
read the original abstract

In this work, we explore the largely unexplored direction of building a generalist image tokenizer directly on top of a frozen vision foundation model (VFM). To build this tokenizer, we utilize a frozen VFM as the encoder and introduce two key innovations: (1) a region-adaptive quantization framework to eliminate spatial redundancy in standard 2D grid features, and (2) a semantic reconstruction objective that aligns the decoded outputs with the VFM's representations to preserve semantic fidelity. Grounded in these designs, we propose VFMTok, a generalist visual tokenizer capable of operating seamlessly in both discrete and continuous latent spaces. VFMTok achieves substantial improvements in synthesis quality while drastically enhancing token efficiency. For discrete autoregressive (AR) generation, it accelerates model convergence by \textbf{3 times} and achieves a state-of-the-art gFID of \textbf{1.36} on ImageNet class-conditional synthesis. Similarly, for continuous-space generation, integrating VFMTok with a denoising model yields an exceptional gFID of \textbf{1.25}. Furthermore, because the latent space inherently captures rich spatial semantics, VFMTok enables high-fidelity class-conditional synthesis without classifier-free guidance (\textbf{w/o CFG}) across both generative paradigms, significantly accelerating inference speed. Beyond these remarkable empirical results, we systematically investigate the underlying mechanisms of our approach. We discover that the specific self-supervised learning objectives utilized during VFM pre-training dictate its effectiveness as a tokenizer. Specifically, a VFM jointly optimized with global contrastive learning and latent masked image modeling provides the optimal representations for image tokenization. These insights establish a strong foundation and offer valuable guidance for the design of future image tokenizers.

Figures

Figures reproduced from arXiv: 2605.18390 by the authors.

Figure 1
Figure 1. VFMTok introduces novel features, including: a). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework of VFMTok/VFMAE. VFMTok/VFMAE utilizes a frozen VFM to extract multi-level image features. A deformable Transformer [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 7
Figure 7. ). For optimal clarity, please zoom in [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Fig.3. Autoregressive class-conditional image generation with classifier-free guidance (CFG). [PITH_FULL_IMAGE:figures/full_fig_p018_3.png]
Figure 4
Figure 4. Figure 4: Fig.4. Autoregressive class-conditional image generation without classifier-free guidance (CFG). [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

Discussion (0). Sign in to comment.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Modeling sparse visual state updates instead of full images cuts generated visual tokens ~55.6% and improves interleaved multimodal reasoning over full-image ULMMs.

Reference graph

Works this paper leans on

104 extracted references · 104 canonical work pages · cited by 1 Pith paper

  1. [1]

    Building Normalizing Flows with Stochastic Interpolants

    Michael S Albergo and Eric Vanden-Eijnden. Building nor- malizing flows with stochastic interpolants. arXiv preprint arXiv:2209.15571, 2022

    work page Pith review arXiv 2022

  2. [2]

    arXiv preprint arXiv:2502.13967 , year=

    Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, O˘ guzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Za- mir, and Afshin Dehghan. Flextok: Resampling images into 1d to- ken sequences of flexible length. arXiv preprint arXiv:2502.13967, 2025

    work page arXiv 2025

  3. [3]

    Autoencoders

    Dor Bank, Noam Koenigstein, and Raja Giryes. Autoencoders. Machine learning for data science handbook: data mining and knowledge discovery handbook, pages 353–374, 2023

    work page 2023

  4. [4]

    Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Esti- mating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013

    work page Pith review arXiv 2013

  5. [5]

    VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models

    Tianci Bi, Xiaoyi Zhang, Yan Lu, and Nanning Zheng. Vision foundation models can be good tokenizers for latent diffusion models. arXiv preprint arXiv:2510.18457, 2025

    work page Pith review arXiv 2025

  6. [6]

    Understanding disentangling in $\beta$-VAE

    Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Under- standing disentangling in β-vae. arXiv preprint arXiv:1804.03599, 2018

    work page Pith review arXiv 2018

  7. [7]

    Emerging proper- ties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging proper- ties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021

    work page 2021

  8. [8]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022

    work page 2022

Show all 104 references
  1. [9]

    Aligning visual foundation encoders to tokenizers for diffusion models

    Bowei Chen, Sai Bi, Hao Tan, He Zhang, Tianyuan Zhang, Zhengqi Li, Yuanjun Xiong, Jianming Zhang, and Kai Zhang. Aligning visual foundation encoders to tokenizers for diffusion models. arXiv preprint arXiv:2509.25162, 2025

    2025

  2. [10]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020

    2020

  3. [11]

    Im- proved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Im- proved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020

    2003 arXiv

  4. [12]

    Detection in crowded scenes: One proposal, multiple predictions

    Xuangeng Chu, Anlin Zheng, Xiangyu Zhang, and Jian Sun. Detection in crowded scenes: One proposal, multiple predictions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12214–12223, 2020

    2020

  5. [13]

    Deformable convolutional networks

    Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 764–773, 2017

    2017

  6. [14]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bo- janowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023

    2023 arXiv

  7. [15]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  8. [16]

    Bert: Pre-training of deep bidirectional transform- ers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transform- ers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    2019

  9. [17]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021

    2021

  10. [18]

    An introduction to variational autoencoders

    P Kingma Diederik and Welling Max. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4):307–392, 2019

    2019

  11. [19]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  12. [21]

    Scaling rectified flow transformers for high- resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high- resolution image synthesis. In Forty-first international conference on machine...

    2024

  13. [22]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873–12883, 2021

    2021

  14. [23]

    One layer is enough: Adapting pretrained visual encoders for image generation

    Yuan Gao, Chen Chen, Tianrong Chen, and Jiatao Gu. One layer is enough: Adapting pretrained visual encoders for image generation. arXiv preprint arXiv:2512.07829, 2024

    2024

  15. [24]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commun Acm, 2020

    2020

  16. [25]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural i...

    2020

  17. [26]

    Learnings from scaling visual tokenizers for reconstruction and generation

    Philippe Hansen-Estruch, David Yan, Ching-Yao Chung, Orr Zohar, Jialiang Wang, Tingbo Hou, Tao Xu, Sriram Vishwanath, Peter Vajda, and Xinlei Chen. Learnings from scaling visual tokenizers for reconstruction and generation. arXiv preprint arXiv:2501.09755, 2025

    2025

  18. [27]

    Masked autoencoders are scalable vision learn- ers

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learn- ers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022

    2022

  19. [28]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Gir- shick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019

    1911

  20. [29]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016

    2016

  21. [30]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017

    2017

  22. [31]

    Burgess, Xavier Glorot, Matthew M

    Irina Higgins, Loïc Matthey, Arka Pal, Christopher P . Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2016

    2016

  23. [32]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020

    2020

  24. [33]

    Image-to-image translation with conditional adversarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017

    2017

  25. [34]

    Image-to-image translation with conditional adversarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1125–1134, 2017

    2017

  26. [35]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916....

    2021

  27. [36]

    Guiding a diffusion model with a bad version of itself

    Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehti- nen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. Advances in Neural Information Processing Systems, 37:52996–53021, 2024

    2024

  28. [37]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019

    2019

  29. [38]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  30. [39]

    Eq-vae: Equivariance regularized latent space for improved generative image modeling

    Theodoros Kouzelis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. Eq-vae: Equivariance regularized latent space for improved generative image modeling. arXiv preprint arXiv:2502.09509, 2025. 14

    2025

  31. [40]

    Boosting generative image modeling via joint image-feature synthesis

    Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. Boosting generative image modeling via joint image-feature synthesis. arXiv preprint arXiv:2504.16064, 2025

    2025

  32. [41]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Mohamad Hassan Mohamad Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object det...

    1956

  33. [42]

    Improved precision and recall metric for assessing generative models

    Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehti- nen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019

    2019

  34. [43]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook- Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523–11532, 2022

    2022

  35. [44]

    Repa-e: Unlocking vae for end- to-end tuning with latent diffusion transformers

    Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end- to-end tuning with latent diffusion transformers. arXiv preprint arXiv:2504.10483, 2025

    2025

  36. [45]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024

    2024

  37. [46]

    Imagefolder: Autoregressive image generation with folded tokens

    Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. arXiv preprint arXiv:2410.01756, 2024

    2024

  38. [47]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017

    2017

  39. [48]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980–2988, 2017

    2017

  40. [49]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  41. [50]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022

    2022 arXiv

  42. [51]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  43. [52]

    Open-magvit2: An open-source project toward democratizing auto-regressive visual generation

    Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.04410, 2024

    2024

  44. [53]

    Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pages 23–40. Springer, 2024

    2024

  45. [54]

    Finite scalar quantization: Vq-vae made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023

    2023 arXiv

  46. [55]

    Conditional generative adversarial nets

    Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014

    2014 arXiv

  47. [56]

    One-d-piece: Image tokenizer meets quality- controllable compression

    Keita Miwa, Kento Sasaki, Hidehisa Arai, Tsubasa Takahashi, and Yu Yamaguchi. One-d-piece: Image tokenizer meets quality- controllable compression. arXiv preprint arXiv:2501.10064, 2025

    2025

  48. [57]

    Improved denois- ing diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denois- ing diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021

    2021

  49. [58]

    Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V . Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas ...

    2023 arXiv

  50. [59]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  51. [60]

    To- kenflow: Unified image tokenizer for multimodal understanding and generation

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. To- kenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069, 2024

    2024

  52. [61]

    Learning transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language super- vision. In International C...

    2021

  53. [62]

    Unsupervised representation learning with deep convolutional generative adver- sarial networks

    Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adver- sarial networks. arXiv preprint arXiv:1511.06434, 2015

    2015 arXiv

  54. [63]

    Improving language understanding by generative pre-training, 2018

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018

    2018

  55. [64]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in Neural Information Processing Systems, 32, 2019

    2019

  56. [65]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021

    2021

  57. [66]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gon- tijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informati...

    2022

  58. [67]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016

    2016

  59. [68]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016

    2016

  60. [69]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  61. [70]

    Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies

    Wei Song, Yuran Wang, Zijia Song, Yadong Li, Haoze Sun, Weipeng Chen, Zenan Zhou, Jianhua Xu, Jiaqi Wang, and Kaicheng Yu. Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies. arXiv preprint arXiv:2503.14324, 2025

    2025 arXiv

  62. [71]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  63. [72]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  64. [73]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2818–2826, 2016

    2016

  65. [74]

    Unilip: Adapting clip for unified multimodal understanding, generation and editing

    Hao Tang, Chenwei Xie, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang. Unilip: Adapting clip for unified multimodal understanding, generation and editing. arXiv preprint arXiv:2507.23278, 2025

    2025

  66. [75]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905, 2024

    2024

  67. [76]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Na- man Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023 arXiv

  68. [77]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, an...

    2025 arXiv

  69. [78]

    Conditional 15 image generation with pixelcnn decoders

    Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional 15 image generation with pixelcnn decoders. Advances in Neural Information Processing Systems, 29, 2016

    2016

  70. [79]

    Neural discrete representation learning

    Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017

    2017

  71. [80]

    Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion

    Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010

    2010

  72. [81]

    Ddt: Decoupled diffusion transformer

    Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer. arXiv preprint arXiv:2504.05741, 2025

    2025

  73. [82]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004

    2004

  74. [83]

    Principal components

    Xin Wen, Bingchen Zhao, Ismail Elezi, Jiankang Deng, and Xiaojuan Qi. “Principal components” enable a new language of images. arXiv preprint arXiv:2503.08685, 2025

    2025

  75. [84]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  76. [85]

    Gigatok: Scaling visual tokenizers to 3 billion param- eters for autoregressive image generation

    Tianwei Xiong, Jun Hao Liew, Zilong Huang, Jiashi Feng, and Xihui Liu. Gigatok: Scaling visual tokenizers to 3 billion param- eters for autoregressive image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18770–18780, 2025

    2025

  77. [86]

    Fasterdit: Towards faster diffusion transformers training with- out architecture modification

    Jingfeng Yao, Cheng Wang, Wenyu Liu, and Xinggang Wang. Fasterdit: Towards faster diffusion transformers training with- out architecture modification. Advances in Neural Information Processing Systems, 37:56166–56189, 2024

    2024

  78. [87]

    Reconstruction vs

    Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15703–15712, 2025

    2025

  79. [88]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021

    2021 arXiv

  80. [89]

    Scaling autoregressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-ric...

    2022

  81. [90]

    Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang

    Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2023

  82. [91]

    Randomized autoregressive visual generation

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Randomized autoregressive visual generation. arxiv, 2024

    2024

  83. [92]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. arXiv preprint arXiv:2406.07550, 2024

    2024

  84. [93]

    Representation alignment for generation: Training diffusion transformers is easier than you think

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940, 2024

    2024 arXiv

  85. [94]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lu- cas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  86. [95]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018

    2018

  87. [96]

    Holistic tokenizer for autoregressive image generation

    Anlin Zheng, Haochen Wang, Yucheng Zhao, Weipeng Deng, Tiancai Wang, Xiangyu Zhang, and Xiaojuan Qi. Holistic tokenizer for autoregressive image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16916–16926, 2025

    2025

  88. [97]

    Vision foundation models as effective visual tokenizers for autoregressive image generation

    Anlin Zheng, Xin Wen, Xuanyang Zhang, Chuofan Ma, Tiancai Wang, Gang Yu, Xiangyu Zhang, and Xiaojuan Qi. Vision foundation models as effective visual tokenizers for autoregressive image generation. arXiv preprint arXiv:2507.08441, 2025

    2025

  89. [98]

    Diffusion transformers with representation autoencoders

    Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders. arXiv preprint arXiv:2510.11690, 2025

    2025 arXiv

  90. [99]

    Movq: Modulating quantized vectors for high-fidelity image generation

    Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35:23412–23425, 2022

    2022

  91. [100]

    Fast training of diffusion models with masked transformers

    Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandku- mar. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306.09305, 2023

    2023

  92. [101]

    ibot: Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022

    2022

  93. [102]

    Unpaired image-to-image translation using cycle-consistent ad- versarial networks

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent ad- versarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017

    2017

  94. [103]

    Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%

    Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406.11837, 2024

    2024

  95. [104]

    Deformable detr: Deformable transformers for end-to- end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to- end object detection. arXiv preprint arXiv:2010.04159, 2020

    2010 arXiv

  96. [105]

    Stabilize the latent space for image autoregressive modeling: A unified perspective

    Yongxin Zhu, Bocheng Li, Hang Zhang, Xin Li, Linli Xu, and Lidong Bing. Stabilize the latent space for image autoregressive modeling: A unified perspective. arXiv preprint arXiv:2410.12490, 2024. 16 APPENDIX This section first illustrates the implementation of discrete VFMTok ...

    2024

Pith tools

Reviewed May 20, 2026 · model on record in the stance chip above.