Pith. sign in

REVIEW 3 major objections 7 minor 52 references

Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Quantized language-style image models beat diffusion models at equal bit budgets.

desk verdict The qualitative finding — discrete-token vision models quantize more gracefully — is credible and useful, but the headline scaling-law curves are never reported, so the crossover claims can't be checked. read the letter →

arxiv 2501.06218 v1 pith:2N6RFMWM submitted 2025-01-06 cs.CV

Xin Ding , Shijie Cao , Ting Cao , Zhibo Chen This is my paper · ORCID
classification cs.CV
keywords bit-levelscalinglawsquantizationvisiongenerativemodelslanguage-stylediffusionknowledgedistillationdiscreterepresentationspaceTopKLD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that when vision generative models are quantized, the choice of generative paradigm matters as much as the bit width: language-style models that emit discrete tokens from a codebook lose less quality per bit than diffusion-style models that operate on continuous latents. The authors measure Fréchet Inception Distance across model families from roughly 300M to 7B parameters and bit widths from 3 to 16, and find that discrete-token models keep a consistent quality advantage under the same total model bits and compute bits. They trace the advantage to the codebook's discrete reconstruction space, which absorbs small quantization errors instead of letting them accumulate over generation steps. To push the advantage further, they introduce TopKLD, a distillation loss that fits the top-K token probabilities with a mode-seeking term and the tail with a mode-covering term; this restores low-bit scaling so that W3A16 behaves like W4A16 and W4A8 behaves like W8A8 in both integer and floating-point quantization.

What carries the argument

The central objects are the discrete codebook representation space of language-style models—images are tokenized into indices in $\{1,\ldots,K\}$ and reconstructed by table lookup—and the TopKLD distillation loss, $\mathrm{TopKLD}(P_T \| P_S) = \sum_{t,\, y' \in M_s} P_S(y'|x,y_{<t}) \log\frac{P_S(y'|x,y_{<t})}{P_T(y'|x,y_{<t})} + \sum_{t,\, y' \in M_c} P_T(y'|x,y_{<t}) \log\frac{P_T(y'|x,y_{<t})}{P_S(y'|x,y_{<t})}$, where $M_s$ holds the top-K token probabilities and $M_c$ the remaining tail. The codebook space is what makes the model tolerant to quantization noise: reconstruction in discrete space absorbs errors that accumulate in continuous diffusion latents, and the codebook's consistent input features keep activation variance low across inference steps. TopKLD carries the improvement: fitting the top-K tokens with a mode-seeking reverse KL preserves the teacher's confident choices, while fitting the tail with a mode-covering forward KL preserves the less confident 'implicit knowledge' that low-bit students otherwise discard.

What would settle it

Fit the published FID values for V AR at W16A16, W8A16, W4A16, and W3A16 in log-log coordinates and test whether the slopes are equal; if the W3A16 line is not parallel to the others or its confidence interval overlaps W4A16, the claim that TopKLD elevates scaling by one level would be refuted without needing new experiments. Alternatively, repeat the SNR simulation replacing Gaussian noise with correlated noise matched to actual 3-bit quantization error; if the step-like tolerance disappears, the discrete-space explanation would be an artifact of the noise model.

Watch

Extended reading notes

Core claim

Under quantization, language-style visual generative models have superior bit-level scaling laws to diffusion-style models despite comparable full-precision FID. The paper argues the cause is the discrete codebook representation space: when a quantized model maps features back to codebook indices, small perturbations from low-bit weights and activations are corrected rather than compounded, so FID degrades more gracefully as bits are removed. The paper's proposed TopKLD distillation decomposes the teacher's next-token distribution into top-K and tail parts, applying reverse-KL mode-seeking to the former and forward-KL mode-covering to the latter, and shows this balances explicit and implicit knowledge well enough to lift the scaling curve by one level: 3-bit weights match 4-bit behavior under weight-only quantization, and W4A8 surpasses W8A8 under weight-activation quantization, in both integer and floating-point settings.

Load-bearing premise

The headline comparison rests on the assumption that FID falls on a single bivariate power law in model size and bit precision with parallel slopes across precisions, so that one number can rank bit-level scaling; the paper does not report the fitted curve, residuals, or confidence intervals, and if the lines are not parallel the 3-bit-versus-4-bit claims lose their quantitative basis.

Editorial extensions

If this is right

  • Under a fixed total bit budget, a discrete-token autoregressive generator should beat a diffusion transformer of similar full-precision quality, so bit-constrained deployment should prefer the language-style family.
  • Reducing a language-style model's weights from 16 to 4 bits and activations from 16 to 8 bits can improve FID at equal memory and compute, because the discrete space tolerates the loss.
  • Existing post-training quantization methods and plain quantization-aware training do not materially improve bit-level scaling; knowledge distillation is the effective lever.
  • With TopKLD, W3A16 reaches the scaling behavior of W4A16 and W4A8 surpasses W8A8, and the same one-level gain holds for floating-point quantization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the discrete-representation mechanism is the true cause, then any model with discrete tokens—including tokenizer-free bit-token or lookup-free discrete schemes—should show similar quantization tolerance, which is testable without retraining a full model series.
  • The TopKLD split of the probability distribution is not specific to vision; it could be applied to quantized large language models that use top-k sampling, where the tail of the distribution is known to carry calibration-relevant knowledge.
  • The paper's scaling comparisons are relative to FID; an extension to other quality metrics or to latency-constrained serving could change which model is optimal, since bit-level scaling ignores hardware efficiency differences beyond bit count.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper empirically compares how quantization affects the scaling behavior of two families of vision generative models: diffusion-style DiT (458M–7B) and language-style VAR (310M–2B), under weight-only and weight-activation PTQ/QAT settings. It claims that language-style models have better bit-level scaling laws because their discrete token/codebook representation space tolerates quantization noise better than the continuous latent space of DiT, and it proposes a distillation objective, TopKLD, that splits the teacher distribution into top-K mode-seeking and complement mode-covering terms. The authors report that TopKLD improves low-bit scaling, making 3-bit weight-only and W4A8 weight-activation models competitive with 4-bit and W8A8 counterparts, and that the benefit transfers to floating-point quantization. The appendix provides pointwise FID tables for VAR, comparisons with SmoothQuant/GPTQ/GPTVQ/OmniQuant, additional model validation with MAR and LlamaGen, and ablations of the K hyperparameter.

Significance. If the central claims are correct, the paper would provide a practically relevant design guideline: under bit budgets, language-style discrete-token generators are more robust to aggressive quantization, and TopKLD distillation is a simple recipe to shift their bit-level scaling curves. The study is broad in scope—multiple model families, sizes, bit-widths, PTQ/QAT, integer/FP, and additional validation on MAR/LlamaGen—and reports full pointwise FID tables in the appendix. I do not see a circularity problem: TopKLD is evaluated against external baselines and ablations. However, the quantitative evidence for the headline scaling-law conclusion is incomplete: the asserted bivariate power-law fits and their uncertainties are never reported, and the mechanism analysis rests on a Gaussian-noise proxy that is not validated against real quantization error. These issues are fixable with additional analysis and reporting.

major comments (3)
  1. [Section 3, first paragraph; Sections 3.1 and 3.3] The paper states that FID follows "a distinct bivariate power function" of parameter count and bit precision with "nearly parallel" trends, but no fitted equation, coefficients, R², residuals, or confidence intervals are given anywhere in the main text or appendix. As a result, the headline comparisons—language-style models having superior bit-level scaling laws, and TopKLD "elevating the scaling laws by one level"—cannot be checked. Appendix C.3 Tables 2 and 3 show W3A16 FID higher than W4A16, and W4A8 higher than W8A8, at every listed model size; if the authors' "3-bit outperforms 4-bit" and "W4A8 surpasses W8A8" statements are meant at equal total bits through interpolated curves, those curves and their uncertainty intervals must be reported. Without them, the central scaling-law claims are not quantitatively grounded.
  2. [Section 3.2, Figure 3] The tolerance experiments substitute Gaussian noise at controlled SNR for quantization error. Low-bit quantization error is structured, layer-dependent, and input-dependent (as the activation-distribution analysis in the same section shows), so matched-SNR Gaussian noise is not obviously a faithful surrogate. The reported correlations (0.99 vs. 0.86) lack sample sizes and confidence intervals. The mechanistic conclusion that the discrete representation space is the primary reason for superior bit-level scaling therefore needs either actual quantization-error injection experiments or an explicit empirical validation that Gaussian noise matches the effect of quantization error on the reconstruction process.
  3. [Appendix C.3, Tables 2–5; Figures 1, 5–7] All FID values are single runs with no seeds, sample counts, or error bars. Many differences used to support conclusions are small (e.g., W3A16 TopKLD 3.85 vs. Reverse-KLD 4.02 at d16; W8A8 TopKLD 2.75 vs. SmoothQ 3.81 at d16). Without repeated seeds and uncertainty quantification, pointwise comparisons and fitted scaling exponents cannot be distinguished from sampling noise. Please report seeds and, for any fitted power-law parameters, bootstrap or confidence intervals.
minor comments (7)
  1. [Equations (4)–(6)] Equations (4)–(6) contain typographical corruption: the quantizer formula is missing the division by the step size and the zero-point definition is self-referential; please rewrite them cleanly.
  2. [Section 2.2] The sentence defining total model bits and compute bits is incomplete: "MT ∝8" and "CT ∝82" should be written as MT ∝ 8 × N and CT ∝ 8^2 × N with N the parameter count, and the example text should be completed.
  3. [Tables 2–5 in Appendix C.3] The column headers d16, d20, d24, d30 are not defined; clarify that they denote VAR model sizes and map them to parameter counts.
  4. [Figure 3 caption] The caption "A lower SNR indicates a higher noise component" is insufficient; define the x-axis and state whether the plotted "loss" is FID or another metric, and for which model size the data are shown.
  5. [Section 3.3] Section 3.3 refers to "W4A6" when the surrounding experiments concern W4A8; correct the typo.
  6. [Equation (7) and Table 4] Equation (7) should define the decomposition P = [Ms, Mc], specify that Ms/Mc are the top-K and complement token sets, and explain how the chosen K relates to the inference-time top-k of VAR; Table 4 in the appendix should state whether K is selected on a validation split or by matching the teacher's decoding setting.
  7. [Throughout] Minor language and formatting issues: inconsistent "DiT"/"DIT" and "V AR"/"VAR" spacing, "wight W" in Eq. (13), "Institue" in the affiliation, and the use of "loss" versus "FID" in Figure 3.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central comparisons are empirical, the TopKLD loss is evaluated against external baselines, and no load-bearing claim reduces by construction to its inputs.

full rationale

The central claims—that language-style models exhibit better bit-level scaling laws and that TopKLD improves low-bit scaling—are supported by direct FID measurements across model sizes and bit precisions (Figures 1, 5–7; Tables 2–3; Appendix C.3), not derived from the quantities being predicted. TopKLD is defined independently in Eq. 7 as a hybrid Reverse-KL/Forward-KL objective over the top-K and complement probability sets, and its effectiveness is established by comparison with Forward-KLD, Reverse-KLD, and external methods such as GPTQ, GPTVQ, OmniQuant, and SmoothQuant; the ablation of K is reported as a hyperparameter choice, not as a restatement of the target result. The Section 3 assertion that FID follows a bivariate power function is an interpretive summary of measured data rather than a fitted function used to generate the paper's predictions, so the absence of fitted coefficients, residuals, and confidence intervals is a reproducibility limitation, not evidence of circularity. The Gaussian-SNR tolerance experiments in Section 3.2 are an auxiliary mechanistic probe and are not used as an input to the scaling-law comparisons. No load-bearing self-citations or imported uniqueness theorems appear; the reference list is dominated by external prior work. Accordingly, no circular step can be exhibited, and the honest score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central empirical result depends on the assumed power-law scaling form, the Gaussian noise proxy for quantization error, and a causal attribution to discrete representation space; TopKLD contributes one tuned hyperparameter K. No invented entities are introduced.

free parameters (2)
  • Unreported power-law coefficients for the bit-level scaling law
    The paper asserts in Section 3 that FID follows a bivariate power function in parameter count and bit precision, but never reports the fitted coefficients, exponents, or goodness of fit.
  • TopKLD top-k value K = 600 (best in ablation over 400-800)
    Table 4 shows that K influences final generation quality, and the best value matches the top-k sampling used by the model during generation, making K a tuned hyperparameter in the method.
assumptions (3)
  • domain assumption FID follows a bivariate power function in parameter count and bit precision.
    Stated in Section 3 ('we observed ... bivariate power function'), but no fitted equation, residuals, or error bars are reported; all bit-level scaling comparisons rest on this functional form.
  • domain assumption Gaussian noise with a controlled SNR is a faithful proxy for quantization error.
    Section 3.2 uses SNR-controlled Gaussian noise to simulate quantization effects; real low-bit quantization error is structured and depends on activation distributions, so this proxy may not capture the relevant failure modes.
  • domain assumption The difference between model families is attributable to discrete versus continuous representation space rather than to architecture, tokenizer, training data, or sampling procedure.
    The causal claim in Sections 3.2 and C.2 is inferred from only two model pairs (VAR/DiT and LlamaGen/MAR), which differ in many ways besides the discreteness of the representation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models." pith.science (2026). https://pith.science/paper/2N6RFMWM

@misc{pith2026250106218,
  author       = {Pith},
  title        = {Pith review of: Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2N6RFMWM}},
  note         = {Machine review of arXiv:2501.06218}
}
read the original abstract

Vision generative models have recently made significant advancements along two primary paradigms: diffusion-style and language-style, both of which have demonstrated excellent scaling laws. Quantization is crucial for efficiently deploying these models, as it reduces memory and computation costs. In this work, we systematically investigate the impact of quantization on these two paradigms. Surprisingly, despite achieving comparable performance in full precision, language-style models consistently outperform diffusion-style models across various quantization settings. This observation suggests that language-style models have superior bit-level scaling laws, offering a better tradeoff between model quality and total bits. To dissect this phenomenon, we conduct extensive experiments and find that the primary reason is the discrete representation space of language-style models, which is more tolerant of information loss during quantization. Furthermore, our analysis indicates that improving the bit-level scaling law of quantized vision generative models is challenging, with model distillation identified as a highly effective approach. Specifically, we propose TopKLD to optimize the transfer of distilled knowledge by balancing ``implicit knowledge'' and ``explicit knowledge'' during the distillation process. This approach elevates the bit-level scaling laws by one level across both integer and floating-point quantization settings.

Figures

Figures reproduced from arXiv: 2501.06218 by the authors.

Figure 1
Figure 1. Investigation of bit-level scaling laws for VAR (left) and DiT (right) models using standard PTQ and QAT. Left: Quantited VAR exhibits better bit-level scaling laws than full-precision VAR (a shift towards the lower-left region). Right: Quantized DiT shows ”almost” no improvement compared to full precision. 3.2 WHY DO LANGUAGE-STYLE GENERATIVE MODELS HAVE BETTER BIT-LEVEL SCALING LAWS? Both types of generative model… view at source ↗
Figure 2
Figure 2. (a) denotes the generation process of visual generative models. A comparison of time [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Analysis of Fault Tolerance in Representation Space Reconstruction Errors. (A lower SNR [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Visualization of activation values in the 5th, 15th transformer blocks for VAR (top) and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of bit-level scaling laws across various existing superior PTQ methods. Results [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of bit-level scaling laws with distillation applied to QAT under VAR. With [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: (a) Comparison of Reverse KL, Forward KL, and TopKLD when a Gaussian distribution [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Bit-Level scaling laws based on first-order gradient optimization (Omniquant) in PTQ [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Bit-Level scaling laws based on second-order hessian matrix optimization (GPTQ) in PTQ [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Bit-Level scaling laws based on Vector quantization (GPTVQ) in PTQ [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: The visualization for the activation value distributions in the fc1 layers of VAR across the [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: The visualization for the activation value distributions in the fc1 layers of DiT across the [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: The visualization for the activation value distributions in the fc2 layers of VAR across the [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: The visualization for the activation value distributions in the fc2 layers of DiT across the [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: The visualization for the activation value distributions in the qkv layers of VAR across the [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: The visualization for the activation value distributions in the qkv layers of DiT across the [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Investigation of bit-level scaling laws for MAR (left) and LlamaGen (right) models using standard PTQ. right: Quantited LlamaGen exhibits better bit-level scaling laws than full-precision LlamaGen [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: TopKLD provides a stable enhancement to the bit-level scaling ability of LlamaGen, particularly in the low-bit settings of W3A16 and W4A8. C.4 THE ABLATION OF TOPKLD To further investigate the effectiveness of Top KLD, we conducted an ablation study to assess the impa…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 15 canonical work pages

  1. [1]

    Gkd: Generalized knowledge distillation for auto-regressive sequence models

    Rishabh Agarwal, Nino Vieillard, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.13649,

    arXiv

  2. [6]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323,

    arXiv

  3. [9]

    Imagen video: High definition video generation with diffusion models

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a. Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans...

    arXiv

  4. [11]

    Self-conditioned image generation via generating representations

    Tianhong Li, Dina Katabi, and Kaiming He. Self-conditioned image generation via generating representations. arXiv preprint arXiv:2312.03701, 2023a. Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024a. Tianhong Li, Yonglong Tian, He Li, Mingyan...

    arXiv

  5. [12]

    Fq-vit: Post-training quantiza- tion for fully quantized vision transformer

    Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Post-training quantiza- tion for fully quantized vision transformer. arXiv preprint arXiv:2111.13824,

    arXiv

  6. [13]

    Qllm: Accurate and efficient low-bitwidth quantization for large language models

    Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. Qllm: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.08041,

    arXiv

  7. [14]

    Pseudo numerical methods for diffusion models on manifolds

    Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778,

    arXiv

  8. [15]

    Alle- viating distortion in image generation via multi-resolution diffusion models

    Qihao Liu, Zhanpeng Zeng, Ju He, Qihang Yu, Xiaohui Shen, and Liang-Chieh Chen. Alle- viating distortion in image generation via multi-resolution diffusion models. arXiv preprint arXiv:2406.09416, 2024a. Wenxuan Liu and Saiqian Zhang. Hq-dit: Efficient diffusion transformer with fp4 hybrid quantization. arXiv preprint arXiv:2405.19751,

    arXiv

Show all 52 references
  1. [16]

    Enhanced distribution alignment for post-training quantization of diffusion models

    Xuewen Liu, Zhikai Li, Junrui Xiao, and Qingyi Gu. Enhanced distribution alignment for post-training quantization of diffusion models. arXiv preprint arXiv:2401.04585, 2024b. 13 Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solv...

    arXiv

  2. [17]

    A white paper on neural network quantization.arXiv preprint arXiv:2106.08295,

    Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tij- men Blankevoort. A white paper on neural network quantization.arXiv preprint arXiv:2106.08295,

    arXiv

  3. [18]

    Hierarchical text- conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3,

    arXiv

  4. [19]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019a. Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Adv...

    1909 arXiv

  5. [21]

    Omniquant: Omnidirectionally calibrated quantization for large language models

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137,

    arXiv

  6. [24]

    Improved vector quantized diffusion models

    Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. Improved vector quantized diffusion models. arXiv preprint arXiv:2205.16007,

    arXiv

  7. [25]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905,

    arXiv

  8. [26]

    Givt: Generative infinite-vocabulary transformers

    Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite-vocabulary transformers. arXiv preprint arXiv:2312.02116,

    arXiv

  9. [27]

    Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks

    Albert Tseng, Jerry Chee, Qingyao Sun, V olodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396,

    arXiv

  10. [28]

    Gptvq: The blessing of dimensionality for llm quantization

    Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for llm quantization. arXiv preprint arXiv:2402.15319,

    arXiv

  11. [29]

    Maskbit: Embedding-free image generation via bit tokens.arXiv preprint arXiv:2409.16211,

    Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. Maskbit: Embedding-free image generation via bit tokens.arXiv preprint arXiv:2409.16211,

    arXiv

  12. [30]

    Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145,

    arXiv

  13. [31]

    Ptq4dit: Post-training quantization for diffusion transformers

    Junyi Wu, Haoxuan Wang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Ptq4dit: Post-training quantization for diffusion transformers. arXiv preprint arXiv:2405.16005,

    arXiv

  14. [32]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627,

    arXiv

  15. [33]

    Scaling autoregressive models for content- rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5,

    arXiv

  16. [34]

    Magvit: Masked generative video transformer

    15 Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    arXiv

  17. [35]

    Glm-130b: An open bilingual pre-trained model

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414,

    arXiv

  18. [36]

    Fast sampling of diffusion models with exponential integrator

    Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902,

    arXiv

  19. [37]

    A survey on model compression for large language models

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633,

    arXiv

  20. [38]

    and BRECQ (Li et al., 2021), build upon this foundation. However, as the number of model parameters increases, it has been found that these methods cannot be effectively applied to models with billions of parameters due to the challenges in optimizing within the vast solution ...

    2021

  21. [39]

    improves on OBQ by quantizing all weights in a column simultaneously using a shared Hessian H(ℓ) across rows of wight W(ℓ). After quantizing a column q, the remaining columns q′ > qare updated using a Hessian-based rule δ to account for the quantization error in column q, whic...

    2012

  22. [40]

    Llama- Gen is a discrete language model, similar to V AR in terms of its discrete representation space

    MAR represents a continuous language-style model, aligning with the characteristics of DIT. Llama- Gen is a discrete language model, similar to V AR in terms of its discrete representation space. The results, as shown in the figure 17, reveal the following key observations: MA...

    2023

  23. [41]

    554M 10.94 101 2021.07 × Continuous CDM (Ho et al., 2022b) - 4.88 158.7 2021.12 × Continuous LDM-8 (Rombach et al., 2022a) 258M 7.76 209.5 2022.04 × Continuous LDM-4 (Rombach et al., 2022a) 400M 3.6 247.7 × Continuous DiT (Peebles & Xie,

    2021

  24. [42]

    458M 5.02 167.2 2023.03 ✓675M 2.27 278.2 3B 2.1 304.4 7B 2.28 316.2 Continuous MDT (Gao et al.,

    2023

  25. [43]

    676M 1.58 314.7 2024.02 × Continuous DiMR (Liu et al., 2024a) 505M 1.7 289 2024.07 × Discrete VQ-diffusion (Gu et al.,

    2024

  26. [44]

    370M 11.89 - 2022.03 × Discrete VQ-diffusion-V2 (Tang et al.,

    2022

  27. [45]

    370M 7.65 - 2023.02 × L-style Discrete MaskGIT (Chang et al.,

    2023

  28. [46]

    177M 6.18 182.1 2022.02 × Discrete RCG(cond.) (Li et al., 2023a) 502M 3.49 215.5 2023.12 × Discrete MAGVIT-v2 (Yu et al., 2023b) 307M 1.78 319.4 2023.04 × Discrete TiTok (Yu et al.,

    2022

  29. [47]

    287M 1.97 281.8 2024.07 × Discrete MaskBit (Weber et al.,

    2024

  30. [48]

    305M 1.52 328.6 2024.09 × Discrete VQV AE (Razavi et al., 2019a) 13.5B 31.11 45 2019.06 × Discrete VQGAN (Esser et al., 2021a) 1.4B 5.2 175.1 2021.07 × Discrete RQTran (Lee et al.,

    2024

  31. [49]

    3.8B 3.8 323.7 2022.03 × Discrete VITVQ (Yu et al.,

    2022

  32. [50]

    1.7B 3.04 227.4 2022.07 × Discrete V AR (Tian et al.,

    2022

  33. [51]

    310M 3.3 274.4 2024.04 ✓600M 2.57 302.6 1B 2.09 312.9 2B 1.92 323.1 Discrete LlamaGen (Sun et al.,

    2024

  34. [52]

    Our analysis shows that Top KLD consistently achieves the SOTA results across various bit settings

    343M 3.07 256.06 2024.07 ✓775M 2.62 244.1 1.4B 2.34 253.9 3.1B 2.18 263.3 Continuous MAR (Li et al., 2024b) 208M 2.31 281.7 2024.07 ✓479M 1.78 296 943M 1.55 303.7 2023), GPTQ (Frantar et al., 2022), GPTVQ (van Baalen et al., 2024). Our analysis shows that Top KLD consistently ...

    2024

  35. [2004]

    Deep learning scaling is predictable, empirically

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409,

    arXiv

  36. [2014]

    Post-training quantization on diffusion models

    Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1972–1981,

    1972

  37. [2015]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502,

    2010 arXiv

  38. [2017]

    Distilling the knowledge in a neural network

    Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,

    arXiv

  39. [2019]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525,

    arXiv

  40. [2020]

    Ilvr: Con- ditioning method for denoising diffusion probabilistic models

    Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938,

    arXiv

  41. [2021]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  42. [2022]

    Muse: Text-to-image generation via masked generative transformers

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704,

    arXiv

  43. [2023]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023a...

    arXiv

  44. [2024]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee,

    2009

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.