Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

Preventing Local Pitfalls in Vector Quantization via Optimal Transport

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing the nearest-neighbor search in vector quantization with a Sinkhorn-based optimal transport assignment lets VQNs train stably from scratch, reach 100% codebook utilization, and beat current quantizers on reconstruction quality.

desk verdict A fresh idea in VQ—Sinkhorn-based assignment—with real empirical promise, but the OT formulation is formally wrong in most experiments and the SOTA claim isn't capacity-matched; fixable, so worth refereeing. read the letter →

arxiv 2412.15195 v1 pith:CEA7F2C3 submitted 2024-12-19 cs.CV cs.LG

Borui Zhang , Wenzhao Zheng , Jie Zhou , Jiwen Lu This is my paper · ORCID
classification cs.CVcs.LG MSC 68T0768T45
keywords vectorquantizationoptimaltransportSinkhornalgorithmcodebookutilizationindexcollapseimagereconstructionVQGANdiscretetokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vector-quantized networks assign continuous features to discrete codebook entries by nearest-neighbor search, a greedy rule that routinely strands most entries unused (index collapse) and makes training depend on careful initialization and distillation. This paper argues that the collapse is a local-minimum artifact of the assignment rule itself, not a defect of the codebook, and replaces the search with a global optimal-transport assignment solved by the Sinkhorn algorithm. The resulting quantizer, OptVQ, reaches 100% codebook utilization across codebook sizes and latent dimensions, and in image-reconstruction experiments it improves PSNR, SSIM, LPIPS, and rFID over current vector-quantized models. If the diagnosis is right, a simple swap of the assignment step removes the main source of VQ training instability.

What carries the argument

The load-bearing object is the assignment matrix $A \in R^{l \times n}_+$ defined by the entropy-regularized optimal transport problem $\min_A \mathrm{Tr}(A^T D) - \frac{1}{\epsilon} H(A)$ with row and column sum constraints $A\mathbf{1}_r = \mathbf{1}_r$ and $A^T \mathbf{1}_c = \mathbf{1}_c$, where $D_{ij}$ is the distance between feature $z_i$ and code $c_j$. Instead of the nearest neighbor, the quantizer sends each feature to the code with the largest entry in $A$. The Sinkhorn-Knopp iterations, which alternate row and column normalization starting from $A^0 = e^{-\epsilon D}$, compute this matrix in about five iterations, and a normalization of $D$ decouples the choice of $\epsilon$ from the scale of the data. Because the row and column sums force every code to participate, the argmax over $A$ yields balanced codebook usage.

What would settle it

Run OptVQ on a batch where the number of features $l$ is much larger than the codebook size $n$, record the column sums of the assignment matrix after five Sinkhorn iterations, and count how many codes receive zero argmax picks; if the column sums are far from uniform or any code is never selected, the claimed 100% utilization is not produced by the mechanism the paper describes.

Watch

Extended reading notes

Core claim

The paper's central claim is that the index-collapse failure of vector-quantized networks is caused by the local, greedy nature of nearest-neighbor assignment: once a feature lies inside a code vector's Voronoi cell, the commitment loss keeps pulling it back into that same cell, so badly initialized codes are never repaired. OptVQ replaces the nearest-neighbor rule with a Sinkhorn-based optimal-transport assignment, solving for a soft assignment matrix $A$ that both respects distances and spreads mass over all codes and all features, then choosing the code with the largest entry per feature. The authors report that this removes the need for special initialization and distillation, maintains 100% codebook utilization throughout training, and surpasses current state-of-the-art VQNs on reconstruction quality on ImageNet, MNIST, and CIFAR-10.

Load-bearing premise

The load-bearing premise is that five rounds of the Sinkhorn balancing procedure, run on a transport problem whose exact row and column balance is mathematically impossible whenever the number of features differs from the number of codes, still yield an assignment whose best code per feature keeps every codebook entry in use.

Editorial extensions

If this is right

  • VQNs can be trained from scratch on image reconstruction without the subtle initialization or model distillation that current methods use to avoid collapse.
  • Codebook utilization, which falls below 1% for large codebooks under nearest-neighbor VQ, stays at 100% across codebook sizes from 128 to 16,384 and latent dimensions 8 and 64.
  • Reconstruction quality improves with codebook size under OptVQ, whereas the conventional baseline's reconstruction error does not reliably fall as the codebook grows.
  • The quantization step remains cheap enough for practical training: roughly five Sinkhorn iterations are reported to converge, and a multi-head variant multiplies the effective codebook size to $n^B$.
  • When data and codebook distributions are already similar, OptVQ's assignment agrees with the nearest-neighbor assignment, so the method does not distort the final tokenization once training has stabilized.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not state how the row and column constraints in Eq. (8) are met when the feature count differs from the codebook size; in the reported setups the number of features is typically 16,384 while $n$ ranges down to 128, so the exact problem is infeasible. A useful stress test is to measure how the Sinkhorn column-sum error changes with $n/l$.
  • The experiments stop at reconstruction; if the same stability transfers to token-based generation, OptVQ could serve as a drop-in replacement in generative pipelines, which is the setting where index collapse is most damaging.
  • The manual choice of $\epsilon$ in the paper suggests an interpretable direction: treat $\epsilon$ as a temperature controlling assignment softness and anneal it during training, something the authors leave for future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes OptVQ, a vector quantization method that replaces the conventional nearest-neighbor search with a Sinkhorn-based optimal transport assignment, together with a distance normalization strategy and a multi-head quantizer. The authors argue that nearest-neighbor quantization is prone to local minima and index collapse, and that a globally informed assignment through optimal transport avoids this. They report 100% codebook utilization and improved reconstruction quality over existing VQNs on MNIST, CIFAR-10, and ImageNet, and they release code for reproducibility.

Significance. If the central claims hold, OptVQ would be a practically useful plug-and-play quantizer: it could stabilize VQN training without specialized initialization or distillation, and its 100% codebook utilization together with better reconstruction would address a well-known failure mode. The paper includes a clear motivation, an algorithmic proposal, and a substantial set of experiments, including ablation studies and training statistics. The code release is a further strength. However, the formal optimal-transport problem in Eq. (8) is infeasible in most of the reported experimental settings, and the main comparison in Table 1 is not capacity-matched. These issues affect the interpretation of the central claims, so the current evidence is not yet fully convincing.

major comments (5)
  1. [Sec. 3.2, Eq. (8)] Equation (8) imposes A1_r=1_r and A^T1_c=1_c. Summing these constraints gives total mass l and n respectively, so a feasible A exists only when l=n. The paper never states how l and n are matched. In the ImageNet ablations of Table 3, the default settings (batch size 64, latent 16×16) give l=64×256=16,384 per assignment step, while n ranges from 128 to 16,384; for n<16,384 the constraint set is empty. The sentence in Sec. 3.2 that the constraints 'ensure that each code and feature participate fully' is therefore not supported in the regime where the controlled codebook-size ablations are run. Please either define the actual optimization problem being solved (e.g., an unbalanced OT formulation) or provide an analysis of why 5 Sinkhorn iterations on the infeasible problem produce the desired balanced assignment.
  2. [Sec. 4.4, Table 3] The 100% codebook utilization reported in Table 3 is partly enforced by construction: the Sinkhorn column normalization in Eqs. (11)-(12) forces every code to receive equal total column mass, so the utilization measure is not an independent empirical outcome. To support the claim that OT improves utilization beyond the balancing mechanism, the paper should compare against an alternative assignment that uses only column normalization (or a uniform-prior soft assignment) without the full OT objective, and should report utilization for matched-capacity settings.
  3. [Table 1] The state-of-the-art comparison in Table 1 is not capacity-matched: OptVQ uses B=4 heads with codebook size 16,384 per head, giving an effective codebook of size n^B, whereas most baselines use a single codebook. The reconstruction gains could therefore be attributed to the larger effective codebook rather than to the optimal-transport assignment. Please add an ablation with a single-head OptVQ at comparable effective capacity, or a multi-head baseline with the same B and codebook size, to isolate the effect of the OT mechanism.
  4. [Sec. 3.3, Eqs. (11)-(12)] The proposed normalization replaces the original distance matrix D by D'', a monotone but nonlinear transformation. Because the Sinkhorn soft assignment depends on the magnitudes of the exponentiated costs, solving Eq. (8) with D'' is not equivalent to solving it with D. The paper should justify that this transformed cost preserves the intended global assignment properties, or treat the normalized objective as the actual method and analyze it directly.
  5. [Fig. 9 and Sec. 3.3] The convergence claim that 'no more than 5 iterations are enough' is demonstrated only for a 10×10 instance (l=n=10). In the actual experiments l=16,384 and n varies over several orders of magnitude; with l≠n the alternating row/column normalization does not converge to a doubly stochastic matrix. The paper should either provide convergence experiments at the operating dimensions or restrict the claim to the feasible setting.
minor comments (4)
  1. [Throughout] There are numerous typos and grammatical errors, e.g., 'develope' (Sec. 1), 'ourperforms' (Sec. 4.2), 'substatiating' (Sec. 5), 'obstables' (Sec. 5), and 'pionts' (Fig. 7 caption). A careful proofread is needed.
  2. [Table 1] The 'From Scratch' column uses the symbols " and % without a legend; please clarify what these denote and how they were determined for each baseline.
  3. [Sec. 4.3, Fig. 7] The consistency check would be more informative if it included a quantitative measure of agreement between OptVQ and nearest-neighbor assignments rather than only arrows in a 2D example.
  4. [Appendix C, Fig. 10] The statement that 'the majority of codes are selected between 300 to 600 times' is not supported by the histogram in the figure, which appears to show a broader range; please either adjust the description or report the relevant quantiles.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported 100% codebook utilization is an empirical selection-frequency outcome, not a tautological consequence of the balanced-assignment constraints.

full rationale

The derivation chain in OptVQ does not reduce to its own inputs. The optimal transport objective in Eq. (8) imposes row and column marginal constraints plus an entropy term; these are design choices intended to avoid index collapse. However, the reported 100% codebook utilization is measured as selection frequency, as Appendix C states: "we undertook a comprehensive analysis to ascertain the frequency of selection for each code in the codebook during the quantization process." The hard column constraint A^T 1_c = 1_c only equalizes each code's total assignment mass; it does not logically force every column to be the argmax-selected code of some row. Even a strictly positive doubly stochastic matrix can have a column that is never the row-wise maximum, so the utilization claim is an empirical training outcome under Eq. (9), not a tautology. The reconstruction-quality improvements are comparisons against external baselines under the VQGAN architecture and are not derived from the paper's own assumptions. The formal concern that Eq. (8) is infeasible when the number of features l differs from the codebook size n (e.g., Table 3 with 16,384 features and smaller codebooks) is a correctness or inference gap, not circularity: it questions whether the stated OT problem is actually the one solved, but it does not show that any prediction is equivalent to an input by construction. No load-bearing step relies on self-citation. Therefore, no circular step is exhibited, and the paper's central empirical claims retain independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several assumptions that are not fully justified: the validity of balanced OT as a VQ objective, the use of an infeasible constraint set when l != n, the simplified stability analysis, and the assumption that normalization does not distort the assignment. No new physical entities are introduced.

free parameters (4)
  • epsilon (Sinkhorn entropy coefficient) = 10
    Set by hand in all main experiments; Section 5 admits it requires manual tuning. No sensitivity analysis is reported.
  • Sinkhorn iterations T = 5
    Chosen heuristically; convergence is shown only on a simulated 10x10 case, not on the real feature-code matrix.
  • commitment loss weight beta = 0.25
    Standard VQ-VAE hyperparameter, inherited and fixed; not central to the OT contribution but affects training.
  • number of heads B = 4
    Borrowed from MoVQ; it multiplies effective codebook capacity and confounds the comparison with single-head baselines in Table 1.
assumptions (4)
  • domain assumption The OT problem in Eq. (8) with simultaneous row and column sum constraints is a valid model for VQ assignment, and its solution (after 5 Sinkhorn iterations) yields the quantized tokens via argmax.
    The paper does not justify that a balanced soft assignment is appropriate when features are not uniformly distributed; it is the core modeling choice.
  • ad hoc to paper Sinkhorn row/column normalization schedules remain useful when the number of features l differs from the codebook size n, despite the constraints being infeasible in that case.
    Eq. (8) requires total mass l = n; experiments in Table 3 vary n while l is fixed by batch size, so exact feasibility cannot hold; the algorithm is still applied.
  • domain assumption The convexity argument of Section 3.2 (a feature remains inside its Voronoi cell) captures the training dynamics of a full VQN that also uses reconstruction, perceptual, and adversarial losses.
    The proof only considers the commitment loss and assumes the codebook is slowly varying; the other losses are ignored.
  • ad hoc to paper The normalization in Eqs. (11)-(12) does not change the assignment problem in a way that harms reconstruction; it only rescales distances.
    The normalization is applied to the cost matrix before exponentiation, which changes the OT problem, but the paper asserts it improves numerical stability without analyzing distortion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Preventing Local Pitfalls in Vector Quantization via Optimal Transport." pith.science (2026). https://pith.science/paper/CEA7F2C3

@misc{pith2026241215195,
  author       = {Pith},
  title        = {Pith review of: Preventing Local Pitfalls in Vector Quantization via Optimal Transport},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CEA7F2C3}},
  note         = {Machine review of arXiv:2412.15195}
}
read the original abstract

Vector-quantized networks (VQNs) have exhibited remarkable performance across various tasks, yet they are prone to training instability, which complicates the training process due to the necessity for techniques such as subtle initialization and model distillation. In this study, we identify the local minima issue as the primary cause of this instability. To address this, we integrate an optimal transport method in place of the nearest neighbor search to achieve a more globally informed assignment. We introduce OptVQ, a novel vector quantization method that employs the Sinkhorn algorithm to optimize the optimal transport problem, thereby enhancing the stability and efficiency of the training process. To mitigate the influence of diverse data distributions on the Sinkhorn algorithm, we implement a straightforward yet effective normalization strategy. Our comprehensive experiments on image reconstruction tasks demonstrate that OptVQ achieves 100% codebook utilization and surpasses current state-of-the-art VQNs in reconstruction quality.

Figures

Figures reproduced from arXiv: 2412.15195 by the authors.

Figure 1
Figure 1. Comparison between different VQ methods. (a) Vanilla [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Optimization process of different quantization methods. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training details of OptVQ. (a) The iterative Sinkhorn [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Visualizations of reconstruction results on ImageNet validation set (detailed comparison marked in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Visualizations on MNIST and CIFAR-10. equally at 1. Unless specified otherwise, the quantizer pa￾rameters are configured as follows: the codebook size is set to n = 16384, the feature dimension is d = 64, a sin￾gle fully connected layer serves as the shared affine tran…
Figure 6
Figure 6. Figure 6: The dynamics of the quantization operation in the two-dimensional case. Data points are represented in gray, the codebooks of [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Consistency check between OptVQ and vanilla VQ. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 9
Figure 9. Figure 9: Convergence analysis for Sinkhorn iterations. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Codebook utilization reveal that all codebooks are engaged, exhibiting a relatively consistent pattern of utilization. The least frequently utilized code is selected 138 times, while the most frequently utilized is chosen 1,025 times. Notably, the majority of codes ar…
Figure 11
Figure 11. Figure 11: Details training statistics. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Additional reconstruction results. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Quantize-then-Rectify: Efficient VQ-VAE Training

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A pretrained VAE can be converted into a high-compression VQ-VAE by freezing it and training only a multi-group quantizer plus a post rectifier, cutting training cost by over two orders of magnitude while keeping rFID...

Reference graph

Works this paper leans on

48 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv, abs/2303.08774, 2023. 1, 3

    arXiv 2023

  2. [2]

    Self-labelling via simultaneous clustering and representation learning

    YM Asano, C Rupprecht, and A Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020. 2, 4, 5

    work page 2020

  3. [3]

    Sequential modeling enables scalable learn- ing for large vision models

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learn- ing for large vision models. In CVPR, pages 22861–22872,

  4. [4]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2022. 1, 2, 3

    work page 2022

  5. [5]

    Es- timating or propagating gradients through stochastic neurons for conditional computation

    Yoshua Bengio, Nicholas L ´eonard, and Aaron Courville. Es- timating or propagating gradients through stochastic neurons for conditional computation. arXiv, abs/1308.3432, 2013. 1, 3

    arXiv 2013

  6. [6]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv, abs/2005.14165, 2020. 1, 3

    arXiv 2005

  7. [7]

    Efficient-vqgan: To- wards high-resolution image generation with efficient vision transformers

    Shiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu, Xin Zhao, Deli Zhao, and Kaigi Huang. Efficient-vqgan: To- wards high-resolution image generation with efficient vision transformers. In ICCV, pages 7368–7377, 2023. 1, 2, 3

    work page 2023

  8. [8]

    Deep clustering for unsupervised learning of visual features

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, pages 132–149, 2018. 1, 2, 4

    work page 2018

Show all 48 references
  1. [9]

    Unsupervised learn- ing of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learn- ing of visual features by contrasting cluster assignments. NeurIPS, 33:9912–9924, 2020. 2, 4, 5

    2020

  2. [10]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, pages 11315–11325, 2022. 1, 2, 3, 6

    2022

  3. [11]

    Sinkhorn distances: Lightspeed computation of optimal transport

    Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. NeurIPS, 26, 2013. 2, 4, 5

    2013

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009. 7

    2009

  5. [13]

    Exact penalty methods

    Gianni Di Pillo. Exact penalty methods. Algorithms for con- tinuous optimization: the state of the art , pages 209–253,

  6. [14]

    Generating images with perceptual similarity metrics based on deep networks

    Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. NeurIPS, 29, 2016. 3

    2016

  7. [15]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873–12883, 2021. 1, 2, 3, 5, 6, 7, 11

    2021

  8. [16]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. In ICLR, 2024. 2

    2024

  9. [17]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27,

  10. [18]

    Exact penalty functions in nonlinear programming

    S P Han and Olvi L Mangasarian. Exact penalty functions in nonlinear programming. Mathematical programming, 17: 251–269, 1979. 3

    1979

  11. [19]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. NeurIPS, 30, 2017. 7

    2017

  12. [20]

    Reducing the dimensionality of data with neural networks

    Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504–507, 2006. 3

    2006

  13. [21]

    Straightening out the straight-through estimator: Over- coming optimization challenges in vector quantized net- works

    Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight-through estimator: Over- coming optimization challenges in vector quantized net- works. In ICML, pages 14096–14113, 2023. 1, 2, 3, 4, 6, 7

    2023

  14. [22]

    Batch normalization: Accelerating deep network training by reducing internal co- variate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. In ICML, pages 448–456, 2015. 5

    2015

  15. [23]

    Image-to-image translation with conditional adver- sarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. In CVPR, pages 1125–1134, 2017. 6

    2017

  16. [24]

    Unified language-vision pretraining in llm with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, CHEN Bin, Chengru Song, Di ZHANG, Wenwu Ou, et al. Unified language-vision pretraining in llm with dynamic discrete visual tokenization. In ICLR, 2024. 2

    2024

  17. [25]

    Perceptual losses for real-time style transfer and super-resolution

    Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694–711, 2016. 2, 3

    2016

  18. [26]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv, abs/1312.6114, 2013. 3

    2013 arXiv

  19. [27]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7

    2009

  20. [28]

    Autoencoding beyond pixels using a learned similarity metric

    Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In ICML, pages 1558– 1566, 2016. 2, 3

    2016

  21. [29]

    Gradient-based learning applied to document recog- nition

    Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE , 86(11):2278–2324, 1998. 7

    1998

  22. [30]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, pages 11523–11532, 2022. 2, 6

    2022

  23. [31]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 7

    2019

  24. [32]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zem- ing Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32, 2019. 7

    2019

  25. [33]

    Gen- erating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Gen- erating diverse high-fidelity images with vq-vae-2. NeurIPS, 32, 2019. 2, 3

    2019

  26. [34]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, pages 10684– 10695, 2022. 3 9

    2022

  27. [35]

    Learning internal representations by error prop- agation, parallel distributed processing, explorations in the microstructure of cognition, ed

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error prop- agation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclel- land. vol. 1. 1986. Biometrika, 71(599-607):...

    1986

  28. [36]

    Coding theorems for a discrete source with a fidelity criterion

    Claude E Shannon et al. Coding theorems for a discrete source with a fidelity criterion. IRE Nat. Conv. Rec, 4(142- 163):1, 1959. 5

    1959

  29. [37]

    Super-convergence: Very fast training of neural networks using large learning rates

    Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, pages 369–386, 2019. 7

    2019

  30. [38]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In NeurIPS, 2024. 2

    2024

  31. [39]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv, abs/2302.13971, 2023. 3

    2023 arXiv

  32. [40]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017. 1, 2, 3

    2017

  33. [41]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600–612, 2004. 7

    2004

  34. [42]

    Maskbit: Embedding-free image generation via bit tokens

    Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiao- hui Shen, Daniel Cremers, and Liang-Chieh Chen. Maskbit: Embedding-free image generation via bit tokens. arXiv, abs/2409.16211, 2024. 1, 2

    2024 arXiv

  35. [43]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. In ICLR, 2022. 1, 2, 3, 4, 6

    2022

  36. [44]

    Language model beats diffusion-tokenizer is key to visual generation

    Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. In ICLR, 2024. 2

    2024

  37. [45]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. In NeurIPS,

  38. [46]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586–595,

  39. [47]

    Movq: Modulating quantized vectors for high- fidelity image generation

    Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high- fidelity image generation. NeurIPS, 35:23412–23425, 2022. 2, 5, 6

    2022

  40. [48]

    Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%

    Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%. arXiv, abs/2406.11837, 2024. 1, 2, 4, 5, 6, 7 10 Table of Content for Appendix A . Algorithm Details 11 B . Model Structure 11 C . Codebook Utilization ...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.