Pith. sign in

REVIEW 4 major objections 4 minor 57 references

HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HMAR claims that next-scale image generation can be made Markovian—each resolution level conditioned only on its immediate predecessor—and that with per-scale masked refinement and loss reweighting this matches or beats VAR-quality images…

desk verdict A credible VAR+MaskGIT hybrid with real kernel engineering, but the 'parameter-matched' claim is contradicted by the paper's own tables and needs re-benchmarking before it can stand. read the letter →

arxiv 2506.04421 v1 pith:7GUQWSX6 submitted 2025-06-04 cs.CV cs.AIcs.LG

Hermann Kumbong , Xian Liu , Tsung-Yi Lin , Ming-Yu Liu , Xihui Liu , Ziwei Liu , Daniel Y. Fu , Christopher Ré
show 1 more author
David W. Romero
This is my paper
classification cs.CVcs.AIcs.LG
keywords hierarchicalmaskedauto-regressivemodelingnext-scalepredictionvisualautoregressiveimagegenerationblock-sparseattentionefficientNetclass-conditionalmulti-scaleVQ-VAE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HMAR tries to establish that coarse-to-fine 'next-scale' image generation can be made both higher in quality and far cheaper than the Visual Auto-Regressive (VAR) approach it builds on. The central move is to declare the prediction of each resolution scale Markovian, conditioned only on the immediately preceding scale, which turns a block-causal attention pattern into a much sparser block-diagonal one and removes the need to cache earlier scales at inference. On top of that, HMAR replaces VAR's single-shot whole-scale sampling with a MaskGIT-style multi-step masked refinement of each scale, and reweights the training loss to favor the coarser scales that carry global structure. The paper reports that the resulting models match or beat parameter-matched VAR, diffusion, and autoregressive baselines in FID on ImageNet 256×256 and 512×512, with substantially higher Inception Scores, while training up to 2.5× faster, sampling up to 1.75× faster, and using about 3× less inference memory. If the claim holds, HMAR is a practical drop-in upgrade for hierarchical autoregressive image models, with the extra ability to change sampling depth at inference and to do zero-shot image editing.

What carries the argument

The load-bearing object is the Markovian reformulation of next-scale prediction, which changes the attention mask from VAR's block-causal pattern to a block-diagonal pattern in which each scale attends only to its immediate predecessor; this is the mechanism that produces the training and inference speedups and the memory savings. The quality side is carried by the intra-scale multi-step masked generation module, a MaskGIT-style refinement head finetuned on top of the next-scale backbone, which removes the per-scale conditional-independence approximation of VAR, and by the log-normal training-loss reweighting that concentrates model capacity on the coarser, structurally important scales. A custom Triton-based, I/O-aware block-sparse attention kernel, extending FlashAttention (which does not support these patterns), is what turns the theoretical sparsity into measured wall-clock gains.

What would settle it

The paper's own ablation supplies the decisive test: with the Markovian conditioning but no masked refinement and no loss reweighting on ImageNet 256×256, FID is 3.76 versus 3.30 for the reported VAR-d16, so the central claim that HMAR matches VAR quality would be falsified if the masked-refinement and loss-reweighting components fail to close that gap at the larger model sizes (d20, d24, d30) where the paper claims parity. An independent check is to measure, at each scale, the held-out predictive loss of conditioning on the immediate predecessor alone versus on all previous scales: if the early scales—which the paper calls the most important—show a large gap, the Markovian equivalence in Eq. 4 is not faithful.

Watch

Extended reading notes

Core claim

The paper's claim is that next-scale prediction is naturally Markovian: because the running reconstruction $\tilde{x}_{1:k-1}$ accumulates all information from earlier stages, the conditional $p(r_k \mid r_1,\dots,r_{k-1})$ equals $p(r_k \mid \tilde{x}_{1:k-1})$, so each resolution scale can be predicted from its immediate predecessor alone (Eq. 4). This equivalence justifies a block-diagonal attention mask whose sparsity grows with resolution, plus inference without prefix computations or key-value caches. The paper then argues that VAR's parallel whole-scale sampling implicitly assumes tokens within a scale are conditionally independent, which degrades quality through oversmoothing and error accumulation across scales; HMAR counters this with an intra-scale multi-step masked generation procedure whose number of steps per scale is an adjustable quality-speed dial, together with a log-normal reweighting of the training loss across scales. Empirically, the paper reports HMAR-d16 reaching FID 3.01 and Inception Score 288.6 versus VAR-d16's 3.36 and 277.8 on ImageNet 256×256, and HMAR-d30 matching VAR-d30's FID 1.95 while raising Inception Score from 303.6 to 334.5, with end-to-end training up to 2.5× faster and inference up to 1.75× faster at roughly 3× lower memory use. At 512×512, the paper's HMAR-d24 is competitive with the much larger VAR-d36 (FID 2.99 vs 2.63, Inception Score 304.1 vs 303.2, at about half the parameters).

Load-bearing premise

The load-bearing assumption is that each resolution level of an image carries enough information by itself to predict the next level up, so a model that only ever looks at the immediately preceding level loses nothing essential.

Editorial extensions

If this is right

  • Parameter-matched HMAR models match or improve on VAR's FID at 256×256 while gaining up to roughly 30 Inception Score points (334.5 vs 303.6 at the d30 scale), so the quality improvement is measurable, not only qualitative.
  • Training at high resolution becomes substantially cheaper: end-to-end speedups reach 2.5× at 1024×1024, and the custom attention kernel alone is up to 15.8× faster forward and 6.4× faster backward, making higher-resolution or larger models more tractable.
  • Inference needs no prefix computation and no key-value cache of earlier scales, giving up to 1.75× faster sampling and about 3× lower memory, which widens the feasible model size on a given GPU.
  • Sampling depth becomes a free parameter: more masked steps at coarse scales improve FID, and more at fine scales improve perceptual detail, all without retraining.
  • The intra-scale masking transfers to zero-shot inpainting, outpainting, and class-conditional editing, extending the model beyond pure class-conditional generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Table 3 indicates the Markovian conditioning by itself loses ground to full-prefix VAR (FID 3.76 vs 3.30 reported), with the headline quality recovered by masked refinement and loss reweighting; an implication the authors leave implicit is that the efficiency win is partly bought by shifting quality work from the backbone to the refinement head, so the two components should be ablated
  • The log-normal loss-difficulty curve suggests a transferable design rule for other hierarchical generative models—pyramidal diffusion and scale-wise text-to-image models could adopt the same difficulty-matched weighting instead of uniform or resolution-proportional weighting.
  • The paper's own codebook analysis shows early scales use highly skewed, low-utilization codebooks while fine scales use nearly the full codebook; a natural testable extension is asymmetric codebooks, small and specialized for coarse scales and larger for fine scales, which the authors flag as future tokenizer work.
  • Because the Markovian equivalence rests on the residual pyramid structure of the multi-scale VQ-VAE, the approach should degrade gracefully if the tokenizer changes; measuring the FID gap between HMAR and VAR under an alternative tokenizer would test how much of the result is the modeling reformulation versus the tokenizer's pyramid property.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces HMAR, a hierarchical masked autoregressive image generation method. HMAR modifies VAR by (i) approximating next-scale prediction as a Markovian process, conditioning each scale only on its immediate predecessor (Sec. 4.1, Eq. 4); (ii) replacing VAR's single-step per-scale sampling with multi-step masked generation (Sec. 4.2, Eq. 6); and (iii) adding scale-wise log-normal loss reweighting (Sec. 4.3, Eq. 8). It also presents custom Triton block-diagonal attention kernels to exploit the resulting sparsity. On ImageNet 256×256 and 512×512, the paper claims that 'parameter-matched' HMAR models match or outperform VAR, diffusion, and autoregressive baselines while achieving faster training and inference and lower memory use, and it demonstrates flexible sampling and zero-shot image editing.

Significance. If the central empirical claim were sustained, HMAR would be a practically valuable improvement over VAR: the Markovian conditioning yields a sparser attention pattern, the masked-refinement module addresses VAR's intra-scale conditional-independence approximation, and the flexible sampling schedule is a genuine algorithmic advantage. The paper also contains a useful analysis of per-scale loss difficulty (Appendix C) and attention patterns. However, the headline comparison is not parameter-matched as claimed, and the 512×512 result contradicts the abstract. Because the quality advantage of the method is confounded with a 20–50% parameter increase, the paper's main conclusion is not supported by the reported evidence.

major comments (4)
  1. [Table 1 and Section 5.1] The abstract and Section 5.1 claim that HMAR models 'match or outperform parameter-matched VAR' on ImageNet 256×256, but Table 1 shows HMAR-d16/d20/d24/d30 have 465M/840M/1.3B/2.4B parameters versus 310M/600M/1.0B/2.0B for VAR-d16/d20/d24/d30. HMAR is 20–50% larger in every allegedly matching row, so the observed FID/IS gains may simply reflect additional capacity. The internal ablation in Table 3 shows that the Markov assumption alone is worse than VAR-d16 (3.76 FID vs 3.30/3.50), and the final HMAR-d16 quality of 3.01 is only reached after adding loss weighting and masked prediction, which are applied in a model with 155M more parameters. Please provide a matched-size comparison (e.g., HMAR with ~310M parameters vs VAR-d16) or explicitly state that quality comparisons are made at larger parameter counts.
  2. [Table 2 and Abstract] The only 512×512 comparison in which HMAR has fewer parameters than VAR directly contradicts the abstract's 'match or outperform' claim: HMAR-d24 (1.3B) is worse than VAR-d36 (2.5B) on FID (2.99 vs 2.63) and only ties on Inception Score (304.1 vs 303.2). The caption says HMAR is 'competitive,' which is a weaker statement, but the abstract and Introduction are not so qualified. The central claim must be revised to reflect the actual reported results, or new experiments must be provided that support the original wording.
  3. [Section 5.2 and Appendix B.4] The training and attention speedups are measured against torch.sdpa block-causal attention, which is a generic fused kernel and is not optimized for VAR's block-causal pattern. The claim of up to 2.5× end-to-end training speedup and 10×–15.8× attention speedup is therefore likely overstated, since a strong baseline (e.g., a FlashAttention-style block-causal implementation, or FlexAttention) is not used. Please benchmark against an optimized VAR attention kernel and report end-to-end training/inference times on matched hardware and model sizes, including the cost of the masked-refinement finetuning phase.
  4. [Section 4.1, Eq. (4)] The Markovian equivalence p(r_k | r_1,...,r_{k-1}) = p(r_k | x~_{1:k-1}) is exact as a mathematical identity, but the implemented attention uses interpolated predecessor-scale tokens rather than the full-resolution reconstruction x~_{1:k-1}. This is an additional approximation, and the paper's own Table 3 shows it degrades FID from 3.30 to 3.76. The paper should state clearly that the block-diagonal condition is an approximation whose cost is only partially recovered by the masked-refinement and loss-weighting components; as written, Section 4.1 presents the reformulation as if it were lossless.
minor comments (4)
  1. [Section 5.1] The text says 'ImageNet at 255×256' but should be 256×256.
  2. [Section 5.3] The reference 'Fig. reffig:masking-quantitative' is not resolved; it should point to the actual figure showing the quantitative effect of masking steps.
  3. [Section 4.2, Eq. (6)] The notation for the number of masking steps is inconsistent: Eq. (6) uses M, while the text and Section 4.3 use M_k. Also, 'Mk=Hk×Wk' should be 'M_k = H_k × W_k' for clarity.
  4. [Appendix B.4] The sentence 'the model dimension (d) of 64' is ambiguous: it likely means the per-head dimension, not the model width; please clarify to avoid confusion with model depth (d-16, d-24, etc.).

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; quality claims rest on external baselines; parameter-count mismatch and benchmark-tuned weights are correctness caveats, not circularity.

full rationale

The derivation chain is self-contained. Eq. 4's equality p(r_k|r_1,...,r_{k-1}) = p(r_k|tilde-x_{1:k-1}) is a mathematical identity conditional on the deterministic multi-scale reconstruction defined in Alg. 2, and the subsequent restriction to block-diagonal attention is an approximation that the paper explicitly ablate (Table 3: Markov Assumption 3.76 vs VAR-d16 3.30/3.50). The central quality claims are evaluated against external VAR checkpoints, DiT, MaskGIT, MAR, and Llamagen numbers; they are not derived from the model's own fitted parameters. Author-overlap citations (FlashAttention [10], Hyperhuman [27], speculative decoding [44]) are peripheral engineering and related-work references, not load-bearing justifications of the HMAR formulation. The log-normal loss weighting and 14-step sampling schedule are selected using benchmark FID, which weakens the independence of the headline numbers but is model selection rather than a fitted parameter renamed as a prediction. The 'parameter-matched' language in the abstract is not supported by Table 1 (HMAR-d16 465M vs VAR-d16 310M; HMAR-d24 1.3B vs VAR-d24 1.0B), and the single 512x512 comparison with fewer parameters loses on FID; these are evidence and claim-precision concerns, not circularity in the derivation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The HMAR quality result rests primarily on two empirically tuned schedules (loss weighting and sampling steps) and on the frozen VAR tokenizer. The Markovian conditioning is an approximation, and the paper's own ablation quantifies its cost. No new physical or architectural entities are postulated beyond a masked-prediction head and a block-diagonal attention kernel.

free parameters (2)
  • Log-normal loss reweighting parameters w(k) = Parameters chosen from per-scale test-loss curve; exact values not given in main text
    Section 4.3 and Appendix C.2: the weighting function was selected among six candidates after inspecting test losses and evaluating FID/IS (Table 4), so the reported quality depends on this fitted schedule.
  • Per-scale masked sampling steps M_k = 14 steps total; more steps at coarse scales, fewer at fine scales
    Appendix D.4: the number of intra-scale sampling steps was tuned on FID after training; Table 1 reports 14 steps as the best configuration.
assumptions (3)
  • domain assumption The running reconstruction x~_{1:k-1} is a sufficient statistic for the sequence r_1,...,r_{k-1} in the conditional distribution p(r_k | r_<k).
    Invoked in Section 4.1 to justify Eq. 4 and the switch to block-diagonal attention; no proof of invertibility of the multi-scale VQ decomposition is given.
  • ad hoc to paper Conditioning only on the immediately preceding scale preserves enough information for high-quality next-scale prediction.
    This is the practical Markovian approximation used in the HMAR architecture (Section 4.1, Fig. 10). Table 3 shows it degrades FID relative to full-context VAR, so it is a design trade-off, not a lossless reduction.
  • domain assumption The frozen multi-scale VQ-VAE tokenizer from VAR provides a sufficient discrete representation for image generation.
    HMAR adopts the VAR tokenizer unchanged (Section 5.1); the paper's own Appendix E reports modest reconstruction quality (PSNR 20.69 dB at 256x256), so generation quality is bounded by this external component.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation." pith.science (2026). https://pith.science/paper/7GUQWSX6

@misc{pith2026250604421,
  author       = {Pith},
  title        = {Pith review of: HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GUQWSX6}},
  note         = {Machine review of arXiv:2506.04421}
}
read the original abstract

Visual Auto-Regressive modeling (VAR) has shown promise in bridging the speed and quality gap between autoregressive image models and diffusion models. VAR reformulates autoregressive modeling by decomposing an image into successive resolution scales. During inference, an image is generated by predicting all the tokens in the next (higher-resolution) scale, conditioned on all tokens in all previous (lower-resolution) scales. However, this formulation suffers from reduced image quality due to the parallel generation of all tokens in a resolution scale; has sequence lengths scaling superlinearly in image resolution; and requires retraining to change the sampling schedule. We introduce Hierarchical Masked Auto-Regressive modeling (HMAR), a new image generation algorithm that alleviates these issues using next-scale prediction and masked prediction to generate high-quality images with fast sampling. HMAR reformulates next-scale prediction as a Markovian process, wherein the prediction of each resolution scale is conditioned only on tokens in its immediate predecessor instead of the tokens in all predecessor resolutions. When predicting a resolution scale, HMAR uses a controllable multi-step masked generation procedure to generate a subset of the tokens in each step. On ImageNet 256x256 and 512x512 benchmarks, HMAR models match or outperform parameter-matched VAR, diffusion, and autoregressive baselines. We develop efficient IO-aware block-sparse attention kernels that allow HMAR to achieve faster training and inference times over VAR by over 2.5x and 1.75x respectively, as well as over 3x lower inference memory footprint. Finally, HMAR yields additional flexibility over VAR; its sampling schedule can be changed without further training, and it can be applied to image editing tasks in a zero-shot manner.

Figures

Figures reproduced from arXiv: 2506.04421 by the authors.

Figure 1
Figure 1. HMAR Samples: Class-conditional ImageNet generated samples at 256×256 (HMAR-d30) and 512×512 (HMAR-d24) resolutions. Abstract Visual Auto-Regressive modeling (VAR) has shown promise in bridging the speed and quality gap between autoregressive image models and diffusion models. VAR reformulates autore￾gressive modeling by decomposing an image into successive resolution scales. During inference, an image is generated … view at source ↗
Figure 2
Figure 2. Illustration of the sequential decoding formulation in different methods. We show the decoding process of next-token prediction [14, 42], parallel masked prediction [6], next-scale prediction [45], and our proposed hierarchical multi-step masked prediction. The dark and light grey squares represent the un-generated and generated tokens, respectively. HMAR generates images in an iterative two-step process by first pr… view at source ↗
Figure 3
Figure 3. Visual Comparisons of Samples from VAR-d16 and HMAR-d16. Selected samples highlighting how HMAR’s multi-step generation at each scale can enhance image quality compared to using only next-scale prediction in VAR. 256x256 512x512 1024x1024 0 5 10 15 20 25 30 Runtime (s) OOM HMAR VAR 256x256 512x512 1024x1024 25 50 75 100 125 150 175 Memory (GB) OOM HMAR VAR (a) Inference Runtime and Memory Footprint vs Resolution, d-… view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: Inference and Training Efficiency. HMAR enables more efficient training and inference compared to VAR, with the efficiency gap becoming more pronounced as we scale to higher resolutions. and quality performance. 5.1. Quality In this section, we evaluate the quality of …
Figure 5
Figure 5. Figure 5: Impact of Masking on Visual Quality HMAR-d16. Increasing masked sampling steps can yield improved visual quality. 5.2. Efficiency We benchmark the training speed, the inference speed, and the memory footprint of HMAR compared to VAR. All benchmarks are on a single A100…
Figure 7
Figure 7. Figure 7: Sequence Length vs Resolution for next-scale (VAR) and next-token (AR) prediction. Next-scale prediction requires longer sequences compared to next-token prediction [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Impact of Additional Sampling Steps on Sequence Length: In VAR compared to HMAR, each additional sampling step leads to longer sequence lengths. We show comparisons for 256×256 length, as shown in [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Comparing Attention Patterns in VAR and HMAR [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Attention masks in VAR and HMAR: Block-diagonal pattern in HMAR (right) enables more sparsity compared to the block-causal pattern in VAR(left). contrast, HMAR adopts a block-diagonal attention mask, where each scale is restricted to attending only to the immediately …
Figure 11
Figure 11. Figure 11: Comparison of forward and backward pass speeds between our block sparse attention kernel in (HMAR) and the PyTorch block-causal attention in (VAR) across different image resolutions (256×256, 512×512, and 1024×1024). Tests were performed on an A100 80GB GPU with batch…
Figure 12
Figure 12. Figure 12: Minimum Test Loss Across Scales We use the minimum test loss at each scale as a proxy for learn￾ing difficulty. We find that this has an approximately log-normal pattern ( [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 14
Figure 14. Figure 14: Total Loss for Different Weighting Functions 0 25k 50k 75k 100k 125k 150k Steps 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Total Accuracy Unweighted Linear Square Root Equal Log Normal Exp. decay [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Total Accuracy for Different Weighting Functions 15 [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Scale-wise Decomposition of Loss and Accuracy. We analyze the impact of loss weighting across scales on generated image quality. By decomposing both loss and accuracy metrics scale-by-scale, we reveal a key insight: prioritizing performance in early and intermediate s…
Figure 18
Figure 18. Figure 18: Image Reconstruction in VQ-VAE, VAR and HMAR [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 17
Figure 17. Figure 17: Error Accumulation in Next-Scale Prediction: Teacher forcing experiment showing error propagation across scales. The top rows show image generation starting from the ground truth at earlier scales, showing lower quality. The bottom rows show image generation starting …
Figure 19
Figure 19. Figure 19: Multi-Scale VQ-VAE [45] Reconstructions: We show some failure cases for the Multi-Scale VQ-VAE [45]. While it can capture general image structures, it struggles to reconstruct fine-grained details, particularly in complex elements such as text and facial features. D.4…
Figure 21
Figure 21. Figure 21: Classifier-free guidance impact on FID and Inception Score E.1. Image Reconstruction We provide PSNR and rFID values for the tokenizer at different image resolutions in [PITH_FULL_IMAGE:figures/full_fig_p018_21.png]
Figure 20
Figure 20. Figure 20: Impact of Extra Masked Sampling Steps on FID for HMAR-d16. Increasing the number of sampling steps at earlier scales leads to better FID. We use classifier-free guidance when sampling and show its effect on FID and Inception score in [PITH_FULL_IMAGE:figures/full_fig…
Figure 22
Figure 22. Figure 22: Comparison of Accuracy Between Next-Scale Prediction and Masked Prediction Across Scales. Masked prediction on residuals is a considerably simpler task to learn compared to next-scale prediction. 0 2000 4000 Sorted Codebook Index 0 5 10 15 Frequency (×100) 1x1 0 2000 …
Figure 23
Figure 23. Figure 23: Analysis of Codebook Usage Patterns Across Scales. We observe distinct patterns in how codebook entries are utilized: early scales show highly skewed distributions with only a small subset of codes being frequently accessed, indicating potential redundancy, while late…
Figure 24
Figure 24. Figure 24: Qualitative Comparisons on ImageNet 256×256 20 [PITH_FULL_IMAGE:figures/full_fig_p020_24.png]
Figure 25
Figure 25. Figure 25: Additional Class-Conditional Image Generation Samples on ImageNet 256×256 21 [PITH_FULL_IMAGE:figures/full_fig_p021_25.png]
Figure 26
Figure 26. Figure 26: Additional Class-Conditional Image Generation Samples on ImageNet 256×256 22 [PITH_FULL_IMAGE:figures/full_fig_p022_26.png]
Figure 27
Figure 27. Figure 27: Additional Class-Conditional Image Generation Samples on ImageNet 512×512 23 [PITH_FULL_IMAGE:figures/full_fig_p023_27.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 35 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 1

    arXiv 2023

  2. [2]

    University of Maryland Computer Science, 1980

    Edward H Adelson and Peter J Burt.Image data compression with the Laplacian pyramid. University of Maryland Computer Science, 1980. 4

    work page 1980

  3. [3]

    Beit: Bert pre-training of image transformers, 2022

    Hangbo Bao, Li Dong, Songhao Piao, and Furu W ei. Beit: Bert pre-training of image transformers, 2022. 12

    work page 2022

  4. [4]

    T om B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, T om Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

    work page 2020

  5. [5]

    The laplacian pyramid as a compact image code

    Peter J Burt and Edward H Adelson. The laplacian pyramid as a compact image code. InReadings in computer vision, pages 671–679. Elsevier, 1987. 4

    work page 1987

  6. [6]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. InPro- ceedings of the IEEE/CVF Conference on Computer V ision and P attern Recognition, pages 11315–11325, 2022. 2, 3, 5, 6, 7, 12

    work page 2022

  7. [7]

    Muse: Text-to-image generation via masked generative transformers.arXiv preprint arXiv:2301.00704, 2023

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Y ang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers.arXiv preprint arXiv:2301.00704, 2023. 3, 12

    arXiv 2023

  8. [8]

    Collaborative decoding makes visual auto-regressive modeling efficient.arXiv preprint arXiv:2411.17787, 2024

    Zigeng Chen, Xinyin Ma, Gongfan Fang, and Xinchao W ang. Collaborative decoding makes visual auto-regressive modeling efficient.arXiv preprint arXiv:2411.17787, 2024. 12

    arXiv 2024

Show all 57 references
  1. [9]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. InInternational Conference on Learning Representations (ICLR), 2024. 3, 4, 5, 12

    2024

  2. [10]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R´e

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R´e. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. 3, 4, 5, 12

    2022

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-W ei Chang, Kenton Lee, and Kristina T outanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. 3, 12

    2019

  4. [12]

    Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021. 1, 3, 12

    2021

  5. [13]

    Noise schedules considered harm- ful

    Sander Dieleman. Noise schedules considered harm- ful. https://sander.ai/2024/06/14/noise- schedules.html, 2024. 5

    2024

  6. [14]

    T aming trans- formers for high-resolution image synthesis, 2021

    Patrick Esser, Robin Rombach, and Bj¨orn Ommer. T aming trans- formers for high-resolution image synthesis, 2021. 1, 2, 3, 6, 12

    2021

  7. [15]

    Depthart: Monocular depth estimation as autoregressive refinement task.arXiv preprint arXiv:2409.15010, 2024

    Bulat Gabdullin, Nina Konovalova, Nikolay Patakin, Dmitry Senushkin, and Anton Konushin. Depthart: Monocular depth estimation as autoregressive refinement task.arXiv preprint arXiv:2409.15010, 2024. 12

    2024 arXiv

  8. [16]

    Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David W arde-Farley, Sherjil Ozair, Aaron Courville, and Y oshua Bengio. Generative adversarial nets.Advances in neural information processing systems, 27, 2014. 12

    2014

  9. [17]

    Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis

    Jian Han, Jinlai Liu, Yi Jiang, Bin Y an, Y uqi Zhang, Zehuan Y uan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprint arXiv:2412.04431, 2024. 12

    2024 arXiv

  10. [18]

    Masked autoencoders are scalable vision learners, 2021

    Kaiming He, Xinlei Chen, Saining Xie, Y anghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. 12

    2021

  11. [19]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 3, 12

    2020

  12. [20]

    Pyramidal flow matching for efficient video generative modeling.arXiv preprint arXiv:2410.05954, 2024

    Y ang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Y ang Song, Y adong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling.arXiv preprint arXiv:2410.05954, 2024. 12

  13. [21]

    Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013

    Diederik P Kingma. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013. 12

    2013 arXiv

  14. [22]

    Autoregressive image generation using residual quanti- zation

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and W ook- Shin Han. Autoregressive image generation using residual quanti- zation. InProceedings of the IEEE/CVF Conference on Computer V ision and P attern Recognition, pages 11523–11532, 2022. 12

    2022

  15. [23]

    Mage: Masked generative encoder to unify representation learning and image synthesis

    Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer V ision and P attern Recognition, pages 2142–2152, 2...

    2023

  16. [24]

    Autoregressive image generation without vector quantization, 2024

    Tianhong Li, Y onglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization, 2024. 2, 6, 7, 12

    2024

  17. [25]

    Imagefolder: Autoregressive image generation with folded tokens.arXiv preprint arXiv:2410.01756, 2024

    Xiang Li, Hao Chen, Kai Qiu, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens.arXiv preprint arXiv:2410.01756, 2024. 12

    2024 arXiv

  18. [26]

    Controlvar: Exploring controllable visual autoregressive modeling.arXiv preprint arXiv:2406.09750,

    Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling.arXiv preprint arXiv:2406.09750,

    arXiv

  19. [27]

    Hyperhuman: Hyper-realistic human generation with latent structural diffusion.arXiv preprint arXiv:2310.08579, 2023

    Xian Liu, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Y anyu Li, Dahua Lin, Xihui Liu, Ziwei Liu, and Sergey Tulyakov. Hyperhuman: Hyper-realistic human generation with latent structural diffusion.arXiv preprint arXiv:2310.08579, 2023. 12

    arXiv

  20. [28]

    Star: Scale-wise text-to-image generation via auto-regressive representations.arXiv preprint arXiv:2406.10797, 2024

    Xiaoxiao Ma, Mohan Zhou, T ao Liang, Y along Bai, Tiejun Zhao, Huaian Chen, and Yi Jin. Star: Scale-wise text-to-image generation via auto-regressive representations.arXiv preprint arXiv:2406.10797, 2024. 12

    2024 arXiv

  21. [29]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. 6, 7

    2023

  22. [30]

    Flexattention: The flexibility of py- torch with the performance of flashattention

    PyT orch Team. Flexattention: The flexibility of py- torch with the performance of flashattention. https : / / pytorch . org / blog / flexattention/, 2024. Accessed: 2024-10-17. 3, 12

    2024

  23. [31]

    Self-attention does not need O(n2)memory.arXiv preprint arXiv:2112.05682, 2021

    Markus N Rabe and Charles Staats. Self-attention does not need O(n2)memory.arXiv preprint arXiv:2112.05682, 2021. 12

    2021 arXiv

  24. [32]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018. 1

    2018

  25. [33]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019. 1, 12

    2019

  26. [34]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022. 3, 12

    2022 arXiv

  27. [35]

    Generating diverse high-fidelity images with vq-vae-2.Advances in neural information processing systems, 32, 2019

    Ali Razavi, Aaron V an den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2.Advances in neural information processing systems, 32, 2019. 12

    2019

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  29. [37]

    Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol A yan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information...

    2022

  30. [38]

    Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P . Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications, 2017. 3, 12

    2017

  31. [39]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision.arXiv preprint arXiv:2407.08608, 2024

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision.arXiv preprint arXiv:2407.08608, 2024. 3, 4, 5, 12

    2024 arXiv

  32. [40]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric W eiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. PMLR, 2015. 3, 12

    2015

  33. [41]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502,

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502,

    2010 arXiv

  34. [42]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Y uan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024. 1, 2, 6, 12

    2024 arXiv

  35. [43]

    Hart: Efficient visual generation with hybrid autoregressive transformer.arXiv preprint arXiv:2410.10812, 2024

    Haotian T ang, Y echeng Wu, Shang Y ang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Y ao Lu, and Song Han. Hart: Efficient visual generation with hybrid autoregressive transformer.arXiv preprint arXiv:2410.10812, 2024. 6, 12, 18

    arXiv

  36. [44]

    Accelerating auto-regressive text-to-image generation with training-free speculative jacobi decoding.arXiv preprint arXiv:2410.01699, 2024

    Y ao Teng, Han Shi, Xian Liu, Xuefei Ning, Guohao Dai, Y u W ang, Zhenguo Li, and Xihui Liu. Accelerating auto-regressive text-to-image generation with training-free speculative jacobi decoding.arXiv preprint arXiv:2410.01699, 2024. 12

    arXiv

  37. [45]

    Visual autoregressive modeling: Scalable image gener- ation via next-scale prediction.arXiv preprint arXiv:2404.02905,

    Keyu Tian, Yi Jiang, Zehuan Y uan, Bingyue Peng, and Liwei W ang. Visual autoregressive modeling: Scalable image gener- ation via next-scale prediction.arXiv preprint arXiv:2404.02905,

    arXiv

  38. [46]

    Triton: an intermediate language and compiler for tiled neural network computations

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International W orkshop on Machine Learning and Programming Languages, pages 10–19, 2019. 5

    2019

  39. [47]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo T ouvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Y asmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 12

    2023 arXiv

  40. [48]

    Pixel recurrent neural networks

    A¨aron V an Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. InInternational conference on machine learning, pages 1747–1756. PMLR, 2016. 3, 12

    2016

  41. [49]

    Conditional image generation with pixelcnn decoders, 2016

    Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelcnn decoders, 2016. 1, 3, 12

    2016

  42. [50]

    Neural discrete representation learning, 2018

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018. 3, 12

    2018

  43. [51]

    Switti: Designing scale-wise transformers for text-to-image synthesis.arXiv preprint arXiv:2412.01819, 2024

    Anton V oronov, Denis Kuznedelev, Mikhail Khoroshikh, V alentin Khrulkov, and Dmitry Baranchuk. Switti: Designing scale-wise transformers for text-to-image synthesis.arXiv preprint arXiv:2412.01819, 2024. 12

    2024 arXiv

  44. [52]

    Maskbit: Embedding-free image generation via bit tokens, 2024

    Mark W eber, Lijun Y u, Qihang Y u, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. Maskbit: Embedding-free image generation via bit tokens, 2024. 3, 12

    2024

  45. [53]

    Scaling autoregressive models for content-rich text-to-image generation,

    Jiahui Y u, Y uanzhong Xu, Jing Y u Koh, Thang Luong, Gunjan Baid, Zirui W ang, Vijay V asudevan, Alexander Ku, Yinfei Y ang, Burcu Karagol A yan, Ben Hutchinson, W ei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Y onghui Wu. Scaling autoregressive models for co...

  46. [54]

    V ar-clip: T ext-to-image generator with vi- sual auto-regressive modeling.arXiv preprint arXiv:2408.01181,

    Qian Zhang, Xiangzi Dai, Ninghua Y ang, Xiang An, Ziyong Feng, and Xingyu Ren. V ar-clip: T ext-to-image generator with vi- sual auto-regressive modeling.arXiv preprint arXiv:2408.01181,

    arXiv

  47. [56]

    Extended Related Work 12 B

    7, 12 10 HMAR Supplementary Material Contents A . Extended Related Work 12 B . Efficient Attention Computation 12 B.1. Long Sequences in Next-Scale Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 B.2. Attention Pattern Analysis . . ....

  48. [57]

    However, these models still suffer from quality, efficiency, and flexibility issues

    for improved image quality, speculative decoding [8, 44], foldable tokens [25] for enhanced efficiency, solidifying its position as a powerful backbone for autoregressive image generation. However, these models still suffer from quality, efficiency, and flexibility issues. Eff...

  49. [2024]

    2, 3, 4, 5, 6, 7, 8, 12, 18

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.