Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

VQBridge, a training-only projector, enables vector-quantized image tokenizers to achieve 100% codebook usage even at 262k entries, achieving state-of-the-art reconstruction and boosting autoregressive generation beyond larger diffusion mod

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 18:05 UTC pith:YUQEMDTC

load-bearing objection Solid tokenizer-engineering paper with a real result that deserves peer review, provided the codebook usage protocol is disclosed and the 'full mutual interaction' claim is toned down. the 3 major comments →

arxiv 2509.10140 v1 pith:YUQEMDTC submitted 2025-09-12 cs.CV

Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization

Yifan Chang , Jie Qin , Limeng Qiao , Xiaofeng Wang , Zheng Zhu , Lin Ma , Xingang Wang This is my paper
classification cs.CV
keywords vector quantizationcodebook utilizationdiscrete tokenizerstraight-through estimatorautoregressive image generationimage reconstructionlearning annealingVQBridge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that training instability and codebook collapse in vector-quantized image tokenizers stem from three concrete issues with the straight-through estimator: biased gradients, one-step-behind updates, and sparse codebook gradients. It proposes VQBridge, a compress–process–recover projector trained jointly with the codebook, plus learning-rate annealing, to keep every code vector active. This yields 100% codebook usage across all tested sizes up to 262k, reconstruction rFID of 0.88 on ImageNet 256×256, and generation FID of 2.07 for LlamaGen-XL, beating larger VAR and DiT models. The significance is that a better tokenizer alone can unlock autoregressive image generation, and the method adds no inference cost.

Core claim

The paper establishes that the three STE challenges—straight-through estimation bias, one-step-behind update, and sparse codebook gradients—can be overcome by strong joint optimization of the codebook through a learned projector plus learning annealing. VQBridge compresses the codebook into p summary vectors, processes them with ViT blocks, and recovers the full codebook, enabling rapid alignment between encoder outputs and code distribution. FVQ, the combined method, attains 100% codebook usage with a 262k codebook and 256-dim vectors, improves reconstruction over all compared discrete tokenizers, and when used with LlamaGen surpasses visual autoregressive and diffusion baselines.

What carries the argument

VQBridge is a compress–process–recover projector. It patchifies the K code vectors into p groups, compresses each group via a shared linear projection plus LayerNorm into one summary, runs N ViT blocks over the p summaries for global interaction, then expands each summary back via a shared linear projection and reshape to reconstruct K code vectors of the same dimension. This mapped codebook is used for quantization during training; only the mapped codebook is kept for inference, so VQBridge itself is discarded.

Load-bearing premise

The load-bearing assumption is that a shared linear projection compressing each group of code vectors to one summary, followed by a transformer over only those summaries, can represent all the intra-group and inter-group relationships needed to train the full codebook; if some code vectors require individual nonlinear treatment, the VQBridge model is incomplete.

What would settle it

Train FVQ on a 262k codebook but replace the fixed contiguous grouping with a random permutation of code-vector order before patchification. If codebook usage or rFID degrades materially, the group-summary representation is not capturing arbitrary code relationships. Alternatively, measure training dynamics when the ViT is removed entirely; if 100% usage persists, the claimed mechanism of global mutual interaction is not the cause.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Discrete image tokenizers can be scaled to very large codebooks and high vector channels without collapse, yielding predictable reconstruction gains (rFID 1.30 to 0.88 as codebook grows 16k to 262k).
  • Autoregressive image generators benefit directly: LlamaGen-XL FID improves from 3.39 to 2.07, surpassing VAR-d20 (2.57) and DiT-XL/2 (2.27), showing tokenizer quality is the bottleneck for AR generation.
  • The training-only design means existing VQNs can adopt VQBridge without any inference-time overhead or architectural change.
  • The method generalizes to multi-code variants: FVQ(RQ-VAE) reaches rFID 2.98 in 10 epochs vs 3.20 for RQ-VAE in 50 epochs, and FVQ(VAR) improves rFID from 1.00 to 0.80.
  • With 100% usage at a 262k codebook and 512-dim vectors (rFID 1.09), the path to semantic tokenizers such as CLIP-dim 512 becomes feasible.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the ViT sees only p compressed summaries, the claimed 'full mutual interaction' is approximate; scaling p with codebook size may eventually stress the compressor, so probing VQBridge with random or overlapping group assignments could reveal whether intra-group code structure is under-modeled.
  • The compress–process–recover pattern is agnostic to the underlying image model and could be applied to audio, video, or multimodal discrete tokenizers where codebook collapse is also a bottleneck.
  • If the mechanism is really about rapid distribution alignment, the learning-annealing schedule may be doing more of the work than the paper's ablations show; ablating annealing at various codebook sizes would isolate its contribution.
  • The 512-dim channel scaling suggests VQBridge could serve as the bridge between generative and semantic tokenization, uniting reconstruction and understanding tasks in one codebook—something the paper states as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper analyzes three challenges in vector-quantized network training—straight-through estimation bias, one-step-behind codebook updates, and sparse codebook gradients—and proposes VQBridge, a training-only projector that compresses codebook vectors into p group summaries, processes them with ViT blocks, and expands them back to the full codebook. Combined with learning-rate annealing, the resulting method (FVQ) is reported to achieve 100% codebook usage across codebook sizes from 4k to 262k and vector channels from 4 to 512 on ImageNet, reconstruction rFID of 0.88 with a 262k codebook, and a LlamaGen-XL generation FID improvement from 3.39 to 2.07. Ablations cover patch size, latent dimension, ViT depth, annealing, codebook scaling, training duration, and generalization to RQ-VAE and VAR.

Significance. If the empirical claims hold, this is a significant contribution: VQBridge is architecturally simple, train-only, introduces no inference cost, and the scaling evidence in Tables 3, 4, and 8 is broader than in most tokenizer papers. The generation result—a vanilla autoregressive model surpassing VAR-d20 and DiT-XL/2 on ImageNet 256x256—supports the paper's central argument that tokenizer quality is a bottleneck for AR visual generation. The paper also provides extensive ablations and reports results across multiple datasets and VQ variants. However, the headline '100% codebook usage' claim is not backed by a formal metric definition or measurement protocol, and the main quantitative comparisons are reported on the same validation set used to select key hyperparameters. These issues are load-bearing for the paper's central claims and need to be addressed before the results can be fully assessed.

major comments (3)
  1. [§2, Fig. 3, Tables 1 and 3] The central claim of 100% codebook usage is not verifiable because the metric is never formally defined. The text says only that usage is 'the number of code vectors used' but does not state whether it is measured on the training set, the validation set, a single batch, an entire epoch, or the full dataset, nor at what point during training. If usage counts every codebook entry selected at least once over the 1.28M-image training set, 100% coverage is a weak statement; validation-set coverage is the meaningful statistic. Baseline usage numbers in Tables 1 and 8 (e.g., VQGAN 5.9%, LlamaGen 97%, Open-MAGVIT2 100%) almost certainly come from different protocols, so the comparison in these tables is unreliable. Please define usage precisely, describe the measurement protocol, and report usage on the validation set for FVQ and for re-run baselines.
  2. [§4.2, Tables 1 and 3] The main results are reported on the same ImageNet 50k validation set that was used to select VQBridge hyperparameters (patch size, latent dimension, ViT depth, annealing schedule) in Figure 6 and Table 9. This makes the reported SOTA rFID values potentially optimistic and does not permit an unbiased comparison with baselines that did not receive the same validation-set tuning. Please either report primary results on a held-out set—COCO is currently used only as a secondary evaluation—or provide a clear statement of the selection protocol and the sensitivity of the final numbers to the chosen hyperparameters.
  3. [§3.2, Eq. (8)] Equation (8) is not dimensionally consistent with the text. For a group C_g in R^{(K/p)×d} and W_comp in R^{d×d'}, C_g W_comp is a (K/p)×d' matrix, not a single summary vector h_g in R^{d'}. The aggregation operation that actually compresses each group to one vector (e.g., mean pooling) is omitted, making the core module impossible to reproduce from the text. Relatedly, the claim that the ViT provides 'full mutual interaction' is stronger than what is implemented: the ViT operates on p group summaries, so interactions among the K/p code vectors within one group are folded into the linear compression and expansion rather than individually modeled. Please specify the exact compression operation and qualify the interaction claim accordingly.
minor comments (6)
  1. [Appendix D] The derivation treats the selected code vector z_q as the variable updated by the commitment loss. In standard VQ, the updated object is the codebook entry c_x; this should be made explicit. In addition, the 'ideal update' in Eq. (7) uses z_e^{(t+1)}, which is unknown at step t, and the subsequent upper-bound discussion is heuristic. This appendix is not load-bearing for the empirical results, but as written it is imprecise.
  2. [Appendix B] The limitations section states the method 'can be further applied to models like UniTok', but no evidence is provided. Please mark this as a plan rather than a claim.
  3. [Tables 8 and 4] There are notation/typo issues: Open-MAGVIT2 rows in Table 8 list vector channel 0, and the '68010step' row in Table 4 is ambiguous. Please fix these.
  4. [Table 2] The tokenizer checkpoint used for the generation results (40-epoch or 120-epoch FVQ) is not stated. Please specify which checkpoint was used for the FID numbers.
  5. [Abstract vs. Table 8] The abstract says 'state-of-the-art reconstruction performance' without qualification, but Table 8 reports VAE-SD with rFID 0.87 versus FVQ's 0.88. Please qualify the claim as 'among discrete tokenizers'.
  6. [General] Minor typos: 'BACKGROUD' in Section 2, 'desin' in the Figure 6 caption, and inconsistent spacing in 'V AR' throughout the text.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims are empirical and its theoretical components are standard, externally cited, or derived from first principles.

full rationale

The paper's central claims—100% codebook usage, rFID=0.88, and LlamaGen FID improvement—are reported from training and held-out evaluation, not derived from the method's own construction. The one-step-behind update (Eq. 6) is derived by direct differentiation of the commitment loss in Appendix D, and the annealing analysis in Appendix E uses standard Lipschitz and triangle-inequality bounds; neither imports the target conclusion. The linear-reparameterization premise is credited to external prior work (Huh et al., 2023; Zhu et al., 2024a,b), and none of those citations are by the present authors, so there is no self-citation chain. VQBridge is a new architecture that compresses, processes, and recovers code vectors; it does not by construction guarantee that every code vector is selected as a quantization center unless 'usage' is defined as 'vectors receiving gradient updates,' which the paper never explicitly does. On the standard reading, codebook usage is an emergent empirical outcome measured during training, not an identity forced by the equations. The paper does leave the usage-measurement protocol under-specified (training set vs. validation set, batch size, time of measurement), which is a reproducibility and correctness concern, but not a circularity. No fitted parameter is renamed as a prediction, and no result is equivalent to an input by definition.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on a small set of free hyperparameters selected by validation-set ablations, and on domain assumptions about STE, the shared-size codebook simplification, and the viability of the ViT projector. No new physical entities are introduced.

free parameters (4)
  • patch size p = 16 for 16k codebook, 64 for 262k codebook
    Chosen via ablation on ImageNet validation (Figure 6, right), scaled by 4x with codebook size.
  • latent dimension d' = 256 (equal to vector channel)
    Set after ablation showing d' equal to vector channel is best (Figure 6, left).
  • ViT depth N = 2
    Selected via ablation (Figure 6, middle); N=1 underperforms, N=4 degrades.
  • learning annealing schedule = warmup 4 epochs, decay to 1% of base lr
    Tuned on validation; qualitative and ablation results (Table 9) show warmup matters.
axioms (5)
  • standard math The straight-through estimator provides valid gradients through the non-differentiable arg min quantization.
    Invoked in Section 2, Equation (4), following Bengio et al. (2013).
  • domain assumption The codebook C and mapped codebook Ĉ share the same size and vector channel.
    Explicitly stated in Section 3.2 as a simplifying assumption.
  • domain assumption Learning rate annealing reduces the one-step-behind update error because the correction term is quadratic in the learning rate.
    Derived in Appendix D; relies on the Lipschitz and bounded-change analysis in Appendix E.
  • domain assumption Codebook usage is a meaningful complement to reconstruction loss for measuring VQN optimization quality.
    Stated in Observation 1, Section 3.1, referencing Huh et al. (2023).
  • ad hoc to paper The ViT-based compress-process-recover projector can be trained to align ze and mapped codebook distributions.
    Core design premise of VQBridge; the paper provides empirical support but no theoretical guarantee.

pith-pipeline@v1.3.0-alltime-deepseek · 18578 in / 11769 out tokens · 126484 ms · 2026-08-04T18:05:10.504247+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization." pith.science (2026). https://pith.science/paper/YUQEMDTC

@misc{pith2026250910140,
  author       = {Pith},
  title        = {Pith review of: Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YUQEMDTC}},
  note         = {Machine review of arXiv:2509.10140}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vector quantization (VQ) is a key component in discrete tokenizers for image generation, but its training is often unstable due to straight-through estimation bias, one-step-behind updates, and sparse codebook gradients, which lead to suboptimal reconstruction performance and low codebook usage. In this work, we analyze these fundamental challenges and provide a simple yet effective solution. To maintain high codebook usage in VQ networks (VQN) during learning annealing and codebook size expansion, we propose VQBridge, a robust, scalable, and efficient projector based on the map function method. VQBridge optimizes code vectors through a compress-process-recover pipeline, enabling stable and effective codebook training. By combining VQBridge with learning annealing, our VQN achieves full (100%) codebook usage across diverse codebook configurations, which we refer to as FVQ (FullVQ). Through extensive experiments, we demonstrate that FVQ is effective, scalable, and generalizable: it attains 100% codebook usage even with a 262k-codebook, achieves state-of-the-art reconstruction performance, consistently improves with larger codebooks, higher vector channels, or longer training, and remains effective across different VQ variants. Moreover, when integrated with LlamaGen, FVQ significantly enhances image generation performance, surpassing visual autoregressive models (VAR) by 0.5 and diffusion models (DiT) by 0.2 rFID, highlighting the importance of high-quality tokenizers for strong autoregressive image generation.

Figures

Figures reproduced from arXiv: 2509.10140 by Jie Qin, Limeng Qiao, Lin Ma, Xiaofeng Wang, Xingang Wang, Yifan Chang, Zheng Zhu.

Figure 1
Figure 1. Figure 1: Performance comparison and model architecture. Left: Performance comparison showing generation (FID) and reconstruction (rFID) quality across different methods. Right: Architectural comparison between baseline VQVAE/VQGAN and the proposed FVQ model, with VQBridge component for training. ABSTRACT Vector quantization (VQ) is a key component in discrete tokenizers for image generation, but its training is oft… view at source ↗
Figure 2
Figure 2. Figure 2: Toy experiment and t-SNE visualization results. Left: A dynamic vector quantization toy experiment where the target is a fixed optimization objective and the model is optimized with SGD. Using the standard method with only STE leads to a spiral trajectory, while VQBridge produces a piecewise linear trajectory. With VQBridge, zq closely follows ze, reducing their distance already in the early stages of trai… view at source ↗
Figure 3
Figure 3. Figure 3: Codebook usage during training. With a codebook size of 16k, the linear-layer baseline can achieve 100% usage, but usage drops significantly when learning annealing is applied. With a larger codebook size of 262k, even a 5-layer MLP fails to maintain sufficient usage. In contrast, VQBridge rapidly achieves 100% usage under all configurations and with learning annealing, which greatly facilitates VQN optimi… view at source ↗
Figure 4
Figure 4. Figure 4: FVQ framwork overview. VQBridge first applies 1D patchification to the code vectors, then performs global interaction using ViT blocks, and finally restores them to their original size. 3.2 VQBRIDGE DESIGN The design objective of VQBridge is to achieve strong fitting capability, enabling rapid align￾ment between two distributions, while maintaining scalability and training stability. Inspired by the archit… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative results of FVQ. Left: reconstruction performance compared against baseline methods. Right: generation results. Reconstruction Results. In [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: VQBridge desin ablation results. We vary VQBridge configurations including the latent dimension (d ′ ), ViT block depth (N), and patch size (p), as well as codebook parameters such as the vector channel and codebook size. Latent Dimension (d ′ ). In left subfigure, increasing d ′ from 64 to 256 consistently improves the reconstruction quality. Interestingly, the best results appear when the latent dimensio… view at source ↗
Figure 7
Figure 7. Figure 7: Effect of codebook size on training loss Method Codebook Size usage rFID linear 16k 100% 2.10 linear+Annealing 16k 53% 1.52 VQBridge 16k 100% 2.61 VQBridge+Annealing 16k 100% 1.30 MLP-5 262k 20% 2.06 MLP-5+Annealing 262k 22% 1.75 VQBridge 262k 100% 1.40 VQBridge+Annealing 262k 100% 0.95 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: and generation samples in [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: 256 × 256 FVQ-XL (FID=2.07) samples. Classifier-free guidance scale=1.65. 7 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: 256 × 256 FVQ-XL (FID=2.07) samples. Classifier-free guidance scale=1.65. Ballon (417) Beacon (437) Fountain (562) Pier (718) [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: 256 × 256 FVQ-XL (FID=2.07) samples. Classifier-free guidance scale=1.65. 8 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis

    cs.CV 2026-06 unverdicted novelty 6.0

    A masked discrete diffusion model adds token editing at inference and grouped cross-entropy training to reach 0.90 GenEval, 86.9 DPG, and 10.76 HPSv3 scores.

  2. Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis

    cs.CV 2026-06 conditional novelty 6.0

    Masked discrete diffusion with token editing and grouped cross-entropy reaches strong text-to-image generation scores in an 8B decoder-only model, reporting GenEval 0.90, DPG 86.9, HPSv3 10.76.

  3. NSVQ: Mitigating Codebook Collapse by Stabilizing Encoder Drift in Vector Quantization

    cs.CV 2026-06 unverdicted novelty 5.0

    NSVQ mitigates codebook collapse in large-codebook VQ by addressing encoder drift via non-stationary loss, replacement, and staged freezing, improving rFID from 2.39 to 2.10 on ImageNet-1k while achieving 100% utilization.

Reference graph

Works this paper leans on

59 extracted references · 19 linked inside Pith · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Large-dit-imagenet

    Alpha-VLLM. Large-dit-imagenet. https://github.com/Alpha-VLLM/LLaMA2-Accessory/tree/f7fe19834b23e38f333403b91bb0330afe19f79e/Large-DiT-ImageNet , 2024

    2024

  3. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Estimating or propagating gradients through stochastic neurons for conditional computation

    Yoshua Bengio, Nicholas L \'e onard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013

    Pith/arXiv arXiv 2013

  5. [5]

    Large scale gan training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018

    Pith/arXiv arXiv 2018

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    1901

  7. [7]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In CVPR, pp.\ 11305--11315, 2022

    2022

  8. [8]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, pp.\ 1691--1703. PMLR, 2020

    2020

  9. [9]

    Image N et: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Image N et: A large-scale hierarchical image database. In CVPR, pp.\ 248--255, 2009

    2009

  10. [10]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    Pith/arXiv arXiv 2018

  11. [11]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 34: 0 8780--8794, 2021

    2021

  12. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020

    2020

  13. [13]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bj \" o rn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pp.\ 12873--12883, 2021

    2021

  14. [14]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33: 0 6840--6851, 2020

    2020

  16. [16]

    Cascaded diffusion models for high fidelity image generation

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 23 0 (1): 0 2249--2281, 2022

    2022

  17. [17]

    Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks

    Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks. In International Conference on Machine Learning, pp.\ 14096--14113. PMLR, 2023

    2023

  18. [18]

    Phillip Isola, Jun - Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In CVPR, pp.\ 5967--5976, 2017

    2017

  19. [19]

    Scaling up gans for text-to-image synthesis

    Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR, pp.\ 10124--10134, 2023

    2023

  20. [20]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    Pith/arXiv arXiv 2013

  21. [21]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  22. [22]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 128 0 (7): 0 1956--1981, 2020

    1956

  23. [23]

    Laion-coco 600m

    LAION. Laion-coco 600m. https://laion.ai/blog/laion-coco, 2022

    2022

  24. [24]

    Robust training of vector quantized bottleneck models

    Adrian a \'n cucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alum \"a e, and Antoine Laurent. Robust training of vector quantized bottleneck models. In 2020 International Joint Conference on Neural Networks (IJCNN), pp.\ 1--7. IEEE, 2020

    2020

  25. [25]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook - Shin Han. Autoregressive image generation using residual quantization. In CVPR, pp.\ 11513--11522, 2022

    2022

  26. [26]

    Return of unconditional generation: A self-supervised representation generation method

    Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self-supervised representation generation method. NeurIPS, 37: 0 125441--125468, 2024 a

    2024

  27. [27]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024 b

    Pith/arXiv arXiv 2024

  28. [28]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    Open-magvit2: An open-source project toward democratizing auto-regressive visual generation

    Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.04410, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Unitok: A unified tokenizer for visual generation and understanding

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502.20321, 2025

    arXiv 2025

  31. [31]

    Finite scalar quantization: Vq-vae made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Randar: Decoder-only autoregressive visual generation in random orders

    Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. arXiv preprint arXiv:2412.01827, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In CVPR, pp.\ 4195--4205, 2023

    2023

  34. [34]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. article, 2018

    2018

  35. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp.\ 8748--8763. PmLR, 2021

    2021

  36. [36]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), ICML, volume 139, pp.\ 8821--8831, 2021

    2021

  37. [37]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurIPS, volume 32, 2019

    2019

  38. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pp.\ 10684--10695, 2022 a

    2022

  39. [39]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \" o rn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pp.\ 10674--10685, 2022 b

    2022

  40. [40]

    Stylegan-xl: Scaling stylegan to large diverse datasets

    Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In SIGGRAPH, pp.\ 1--10, 2022

    2022

  41. [41]

    Taming scalable visual tokenizer for autoregressive image generation

    Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang, Ying Shan, and Limin Wang. Taming scalable visual tokenizer for autoregressive image generation. arXiv preprint arXiv:2412.02692, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    Pith/arXiv arXiv 2010

  43. [43]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    Conditional image generation with pixelcnn decoders

    Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. NeurIPS, 29, 2016

    2016

  47. [47]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, volume 30, 2017

    2017

  48. [48]

    Vector-quantized image modeling with improved VQGAN

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN . In ICLR, 2022

    2022

  49. [49]

    Language model beats diffusion - tokenizer is key to visual generation

    Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to visual generation. In ICLR, 2024 a

    2024

  50. [50]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. arXiv preprint arXiv:2406.07550, 2024 b

    Pith/arXiv arXiv 2024

  51. [51]

    Regularized vector quantization for tokenized image synthesis

    Jiahui Zhang, Fangneng Zhan, Christian Theobalt, and Shijian Lu. Regularized vector quantization for tokenized image synthesis. In CVPR, pp.\ 18467--18476, 2023

    2023

  52. [52]

    Efros, Eli Shechtman, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pp.\ 586--595, 2018

    2018

  53. [53]

    Online clustered codebook

    Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In ICCV, pp.\ 22798--22807, 2023

    2023

  54. [54]

    Movq: Modulating quantized vectors for high-fidelity image generation

    Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35: 0 23412--23425, 2022

    2022

  55. [55]

    Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99\ arXiv preprint arXiv:2406.11837, 2024 a

    Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99\ arXiv preprint arXiv:2406.11837, 2024 a

    Pith/arXiv arXiv 2024

  56. [56]

    Addressing representation collapse in vector quantized models with one linear layer

    Yongxin Zhu, Bocheng Li, Yifei Xin, and Linli Xu. Addressing representation collapse in vector quantized models with one linear layer. arXiv preprint arXiv:2411.02038, 2024 b

    arXiv 2024

  57. [57]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  58. [58]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  59. [59]

    Left: Performance comparison showing generation (FID) and reconstruction (rFID) quality across different methods

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2017