Pith. sign in

REVIEW 3 major objections 7 minor 4 cited by

Next Patch Prediction for Autoregressive Visual Generation

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By predicting a 2x2 patch of image tokens at once, this paper claims to reduce autoregressive training cost to ~0.6x and improve ImageNet FID by up to 1.0 without changing inference.

desk verdict Simple, well-ablated patch-curriculum training for AR image generation with consistent FID gains and ~0.6x cost; the main weakness is single-seed evidence and lack of a compute-matched baseline, not the method itself. read the letter →

arxiv 2412.15321 v3 pith:2SMLEFHB submitted 2024-12-19 cs.CV

Yatian Pang , Peng Jin , Shuo Yang , Bin Lin , Bin Zhu , Zhenyu Tang , Liuhan Chen , Francis E. H. Tay
show 3 more authors
Ser-Nam Lim Harry Yang Li Yuan
This is my paper · ORCID
classification cs.CV
keywords autoregressiveimagegenerationnextpatchpredictiontokenmulti-scaletrainingefficiencyNet256x256VQGANtokenizervisual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that autoregressive image generation is better modeled at the level of patches rather than individual tokens. The authors propose Next Patch Prediction (NPP): group image tokens into non-overlapping 2x2 patches, average their embeddings to form one patch embedding, and train the transformer to predict the whole next patch using a cross-entropy loss over all K ground-truth tokens in that patch. Because the input sequence is K times shorter, training cost drops to about 0.6x. Training is scheduled from coarse to fine, starting with large patches and ending with the standard 1x1 next-token prediction, so the inference procedure is unchanged. On the class-conditional ImageNet 256x256 benchmark, NPP improves FID by up to 1.0 point across model sizes from 111M to 1.4B parameters without adding parameters or a custom tokenizer.

What carries the argument

The load-bearing machinery is the patch-wise cross-entropy loss combined with intra-patch averaging. For each patch $p_i$ containing $K$ token embeddings, the patch embedding is $E(p_i) = \frac{1}{K}\sum_{k=1}^{K} E(x_i^k)$, and the model's prediction for the next patch is scored with $L = -\frac{1}{N}\sum_{i=1}^{N/K}\sum_{k=1}^{K} \log(P^{i})$, where the same predicted distribution is applied to every ground-truth token index in the patch. The Multi-scale Patch Grouping schedule (patch levels $\#L$ and segment factor $\lambda$) starts with a large patch size and shrinks it to $1\times 1$, so NPP gradually turns into vanilla NTP. Rotary positional embeddings are kept at the patch-grid positions rather than averaged positions, a design choice said to come from a pilot study.

What would settle it

Run the same NPP training on a tokenizer designed to be spatially decorrelated (for example, a tokenizer that allocates high-frequency details to adjacent tokens, or on texture-rich data like satellite images) and compare FID against vanilla next-token training at matched compute; if the patch objective fails to improve or hurts FID, the central claim is falsified. An internal check: replace the patch-grid RoPE with averaged patch positions; if that closes the FID gap, the patch-averaging objective is not the whole effect.

Watch

Extended reading notes

Core claim

The central discovery is that the standard next-token objective used in autoregressive image models is suboptimal for image data, because image tokens carry low information density and are spatially redundant. NPP instead defines a patch-level objective: given patch embeddings formed by averaging the token embeddings in each cell, the model predicts the distribution over the next patch, and the loss supervises that single predicted distribution against all K ground-truth token indices inside it. A multi-scale schedule anneals the patch size from 4x4 or 2x2 down to 1x1 during training, which transfers coarse structure learning into fine token-level learning. The resulting models consistently beat the vanilla next-token baseline (LlamaGen) on the same architecture, data, and inference setting, with the largest quality gain at the 343M-parameter scale (FID 2.76 vs 3.80). This is framed as a plug-and-play change to the training objective, not as a new architecture or tokenizer.

Load-bearing premise

The method assumes that nearby image tokens are alike enough that averaging their embeddings and asking for one shared prediction for all of them is a useful curriculum; the paper validates this on ImageNet 256x256 with a single VQGAN tokenizer, so the claim may not transfer to data or tokenizers with weak local correlations.

Editorial extensions

If this is right

  • Training an autoregressive image model with NPP cuts compute to roughly 0.6x of the next-token baseline while increasing throughput by 1.3x-1.8x at matched model size.
  • Image quality improves: NPP reaches FID 2.76 for a 343M-parameter model, beating LlamaGen-L-384 (3.07) and Open-MAGVIT2-B (3.08) with less compute.
  • Because inference and architecture are unchanged, NPP can be applied to existing autoregressive visual generators without retraining the tokenizer or modifying sampling.
  • Scaling behavior holds from 111M to 1.4B parameters: every NPP variant outperforms its LlamaGen counterpart at 0.53x-0.63x of the training cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the patch-averaging objective works because it denoises the training signal, then NPP should be most beneficial early in training, and the gains might persist at higher resolutions or on video where local correlation is even stronger; the paper does not test these settings.
  • The same coarse-to-fine annealing could be applied to other modalities, e.g., predicting averaged embeddings of consecutive tokens in language or audio before fine-token prediction, potentially yielding a similar efficiency-quality trade-off.
  • One testable consequence: during the patch phase, the model's single predicted distribution can in principle be sampled K times to generate all tokens of a patch at once, which would turn the training-time speed-up into a decoding-speed-up; NPP does not exploit this.
  • The RoPE decision to use patch-grid rather than averaged positions is untested; if averaged positions were used, the model might lose the implicit scale signal, which could be probed by an ablation at 384x384 resolution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Next Patch Prediction (NPP), a training-only modification of LlamaGen-style autoregressive image generation. Image token embeddings are grouped into non-overlapping patches and averaged to form shorter patch sequences; the model is trained with a patch-wise cross-entropy loss that scores the predicted patch distribution against the K ground-truth tokens in each patch. A multi-scale schedule starts with a larger patch size and ends with vanilla 1x1 next-token prediction, so inference is identical to standard NTP. On ImageNet 256x256, NPP reports FID improvements over LlamaGen at four model sizes (e.g., 2.76 vs 3.80 for the L model) while reducing training FLOPs to roughly 0.53-0.63x, without adding trainable parameters or modifying the inference procedure.

Significance. If the reported results hold, this is a simple, parameter-free, plug-and-play improvement for autoregressive visual generation: it lowers training cost and improves FID without changing the architecture or the inference-time sampling process. The paper's strengths are its conceptual simplicity, the consistency of the FID gain across four model sizes, and the clean ablations isolating the main design choices. The method is internally coherent: Equation (2) and Equation (3) define a valid patch-level training objective, and the multi-scale schedule ends in vanilla NTP. However, the quantitative headline is currently supported only by single-seed comparisons, with default hyperparameters selected on the same benchmark, so the evidence is not yet as secure as the tables suggest.

major comments (3)
  1. [4.1, 4.2, Table 1] All FID/IS numbers in Tables 1-3 come from a single training run per configuration, and FID is evaluated on 50,000 stochastic samples. With sampling noise and training-seed variation, per-point differences such as 0.74 for NPP-XL vs LlamaGen-XL and 0.56 for NPP-XXL vs LlamaGen-XXL are not clearly outside noise. Please provide at least three training seeds (or, at minimum, multiple evaluation sampling runs) for the headline LlamaGen-vs-NPP comparisons, and report the resulting mean and spread.
  2. [4.5, Table 3] The default settings used in the main tables, specifically lambda=1/2, #L=2, and patch size 2x2, were selected as the best values on the same ImageNet 256x256 benchmark. This is a form of selection on the evaluation set. Please either pre-specify the schedule, select defaults on a validation split, or show that the main FID conclusion is stable across the reasonable range of lambda and #L values reported in Table 3.
  3. [4.3, Table 2] The cost-quality claim compares NPP at 300 epochs against LlamaGen at 300 epochs, not against a compute-matched baseline. Because NPP consumes only 0.53-0.63x the FLOPs, a LlamaGen model trained to the same total FLOP budget (for example, more epochs or a larger model) is needed to establish that the cheaper training curriculum, rather than an under-trained baseline, is what produces the observed FID gain.
minor comments (7)
  1. [3.2, Eq. (3)] In Equation (3), the predicted distribution Pred_i is written as a function of patch i, but the same distribution is scored against each of the K ground-truth tokens; please clarify the notation, for example by writing the loss explicitly as summing K times log P(p_i) or by defining Pred_i as a distribution over the K token indexes.
  2. [4.3, Table 2] For NPP-L, the reported cost ratio is 0.57x, which is lower than the nominal 0.625x implied by lambda=1/2 and a 2x2 patch size. Please explain whether the discrepancy comes from attention FLOPs being super-linear in sequence length or from measurement details.
  3. [3.2, Figure 3] The text says "path average operation" but the intended term is "patch average operation"; the same typo appears in the Figure 3 caption.
  4. [3.1] The sentence "without introducing additional trainable parameters or specifically signing a custom image tokenizer" should read "specifically designing a custom image tokenizer."
  5. [1] The introduction refers to "VQV AE"; this should be "VQVAE."
  6. [Figure 1 caption] The caption states "up to 4.0x training speed up," but Table 2 reports approximately 3.95x for NPP-L vs LlamaGen-L-384; please align the wording to "about 4x" or report the exact number.
  7. [References] The reference list contains duplicate entries (e.g., [36] and [37] are the same paper, and [86] and [87] are the same) and some incomplete entries; please clean up the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported FID and training-cost improvements are empirical comparisons against independent baselines, not identities forced by the method's definitions or fitted parameters.

full rationale

The paper's central claim is that Next Patch Prediction (NPP) improves ImageNet 256x256 generation quality by up to 1.0 FID while reducing training cost to about 0.6x relative to the independent LlamaGen baseline. This claim is empirical: Eq. (2) defines patch embeddings as averages of K token embeddings, and Eq. (3) defines a patch-wise cross-entropy loss supervising the K ground-truth tokens; together they specify a consistent training objective, but no equation in the paper reduces the reported FID or cost to these definitions by construction. The 0.625x cost factor is arithmetic from the schedule lambda = 1/2 and patch size 2x2, not a fitted quantity renamed as a prediction. The default hyperparameters (patch size, lambda, #L) were selected by ablations on the same benchmark, which is benchmark tuning and a validity concern rather than circular derivation, since the reported numbers are measurements of the chosen configuration, not quantities defined by the tuning procedure. Self-citations in the reference list concern related multimodal work and do not carry the load-bearing argument. The RoPE choice is justified by an unreported 'pilot study,' which is an unsupported assertion but not a circular step, as it does not define the headline result in terms of itself. The method is self-contained against the independent LlamaGen and VAR baselines, so no circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or architectural entities; it is a training-time transformation of existing token sequences. The free parameters are three scheduling hyperparameters chosen by ablation on the target benchmark. The key domain assumptions are that average pooling yields useful coarse tokens and that a shared distribution over patch tokens is a valid supervision signal.

free parameters (3)
  • lambda (segment scheduling factor) = 1/2
    Chosen as default after scanning 1/2 to 4/5 (Table 3b). Larger lambda lowers cost to 0.4x but raises FID from 2.76 to 2.92; lambda=1/2 gives the best FID/cost balance. Tuned on the ImageNet benchmark.
  • L (number of patch levels) = 2
    Default after comparing L=1 to 4 (Table 3c). L=2 gives best FID (4.47 for NPP-B, 2.76 for NPP-L); higher L reduces cost slightly but worsens FID. Tuned on the same benchmark.
  • Patch size PS = 2x2
    Default after comparing 1x1, 2x2, and 4x4 (Table 3a). PS=2x2 gives the best FID; PS=4x4 causes a transfer drop of 0.45 FID for NPP-B. Tuned on the same benchmark.
assumptions (4)
  • domain assumption Training cost scales as C ~ 6WN (Kaplan et al. 2020), so shorter sequences proportionally reduce cost
    Invoked in Section 1 and used for the cost tables (Table 2, GFLOPs column).
  • domain assumption Intra-patch average of token embeddings preserves enough information for coarse next-patch prediction
    Eq. 2 in Section 3.2; the whole method rests on this pooling being a useful coarse representation.
  • domain assumption Tokens within a 2x2 patch are correlated enough that one predicted distribution can supervise all K ground-truth tokens
    Eq. 3 and Algorithm 1 (label patchify and repeated pred logits); if false, the patch-wise CE loss would be a misleading training signal.
  • ad hoc to paper RoPE for patch tokens should use patch-grid positions rather than averaged positions
    Section 3.2 paragraph on RoPE; based on a pilot study not shown in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Next Patch Prediction for Autoregressive Visual Generation." pith.science (2026). https://pith.science/paper/2SMLEFHB

@misc{pith2026241215321,
  author       = {Pith},
  title        = {Pith review of: Next Patch Prediction for Autoregressive Visual Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2SMLEFHB}},
  note         = {Machine review of arXiv:2412.15321}
}
abstract

Autoregressive models, built based on the Next Token Prediction (NTP) paradigm, show great potential in developing a unified framework that integrates both language and vision tasks. Pioneering works introduce NTP to autoregressive visual generation tasks. In this work, we rethink the NTP for autoregressive image generation and extend it to a novel Next Patch Prediction (NPP) paradigm. Our key idea is to group and aggregate image tokens into patch tokens with higher information density. By using patch tokens as a more compact input sequence, the autoregressive model is trained to predict the next patch, significantly reducing computational costs. To further exploit the natural hierarchical structure of image data, we propose a multi-scale coarse-to-fine patch grouping strategy. With this strategy, the training process begins with a large patch size and ends with vanilla NTP where the patch size is 1$\times$1, thus maintaining the original inference process without modifications. Extensive experiments across a diverse range of model sizes demonstrate that NPP could reduce the training cost to around 0.6 times while improving image generation quality by up to 1.0 FID score on the ImageNet 256x256 generation benchmark. Notably, our method retains the original autoregressive model architecture without introducing additional trainable parameters or specifically designing a custom image tokenizer, offering a flexible and plug-and-play solution for enhancing autoregressive visual generation.

Figures

Figures reproduced from arXiv: 2412.15321 by the authors.

Figure 1
Figure 1. Comparison of our method and baseline meth￾ods. Our method on a diverse range of models achieves higher FID scores with significantly less training cost on the ImageNet 256×256 generation benchmark. Our method NPP-L achieves up to 4.0× training speed up without performance degradation com￾pared to LlamaGen-L-384. to excel in a variety of natural language processing tasks, exhibiting human-like conversational abiliti… view at source ↗
Figure 2
Figure 2. Motivation of the next patch prediction. a). Illustration of next token prediction. b). Demonstration of the proposed next patch prediction. c). Generation results on the ImageNet benchmark. Please zoom in to view. gies of autoregressive and diffusion models pose significant challenges for developing a unified framework that inte￾grates both language and vision tasks. More recently, a pioneering work LlamaGen [85] a… view at source ↗
Figure 3
Figure 3. Next Patch Prediction. The input image token embeddings are grouped and aggregated into patch embeddings through a path average operation. The autoregressive model is trained to predict the next patch by employing the patch Cross Entropy loss. [x1, x2, ..., xk−1]: max θ pθ(x) = Y K k=1 pθ(xk|c, x1, x2, · · · , xk−1), (1) where pθ represents the token distribution predictor with an autoregressive model parameterized … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Multi-scale Next Patch Prediction. The patch group￾ing function begins with a large patch size, resulting in a short sequence length. As training progresses, the patch size is gradu￾ally reduced to 1 × 1. novel next patch prediction paradigm, we propose a multi￾scale, …
Figure 5
Figure 5. Figure 5: Comparison of our method and baseline methods. The vertical axes are the FID score and IS score. We record the performance curve with the number of epochs as horizontal axes. line LlamaGen [85], we provide a comprehensive study as shown in [PITH_FULL_IMAGE:figures/ful…
Figure 6
Figure 6. Figure 6: Generation results. Please zoom in to view. 4.4. Generation Results In [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. E-4DGS: High-Fidelity Dynamic Reconstruction from the Multi-view Event Cameras

    cs.CV 2025-08 conditional novelty 6.0 of 10

    E-4DGS is a deformable 3D Gaussian Splatting method that reconstructs dynamic scenes directly from multi-view event camera streams, outperforming event-to-image baseline approaches.

  2. Autoregressive Image Generation with Linear Complexity: A Spatial-Aware Decay Perspective

    cs.CV 2025-07 reject novelty 6.0 of 10

    A new linear attention with spatial-aware decay at row boundaries lowers FID for autoregressive image generation on ImageNet relative to the softmax LlamaGen baseline, but the description of the core mask is internall...

  3. AE-NeRF: Augmenting Event-Based Neural Radiance Fields for Non-ideal Conditions and Larger Scene

    cs.CV 2025-01 conditional novelty 5.0 of 10

    AE-NeRF jointly optimizes camera poses and an event-based NeRF with a proposal network and four event-specific losses, improving novel view synthesis under noisy poses and non-uniform motion.

  4. Hierarchical Banzhaf Interaction for General Video-Language Representation Learning

    cs.CV 2024-12 conditional novelty 4.0 of 10

    HBI V2 models video-text alignment as a cooperative game with Hierarchical Banzhaf Interaction plus single/cross-modal representation fusion, improving retrieval, QA, and captioning benchmarks.

Reference graph

Works this paper leans on

120 extracted references · 12 canonical work pages · cited by 4 Pith papers

  1. [1]

    Revisiting neural scaling laws in language and vi- sion

    Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiao- hua Zhai. Revisiting neural scaling laws in language and vi- sion. Advances in Neural Information Processing Systems, 35:22300–22312, 2022. 1

    2022

  2. [2]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. 1

    arXiv 2023

  3. [3]

    Anthropic. Claude. https://www.anthropic.com/ index/introducing-claude, 2023. 1

    2023

  4. [4]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 1

    arXiv 2023

  5. [5]

    Sequential modeling enables scal- able learning for large vision models

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scal- able learning for large vision models. arXiv preprint arXiv:2312.00785, 2023. 3

    arXiv 2023

  6. [6]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 3

    2023

  7. [7]

    Deepseek llm: Scaling open- source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open- source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. 1

    arXiv 2024

  8. [8]

    Large scale gan training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018. 3, 6

    arXiv 2018

Show all 120 references
  1. [9]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neu- ral information processing systems , 33:1877–1901, 2020. 1

    1901

  2. [10]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315– 11325, 2022. 3, 6

    2022

  3. [11]

    Muse: Text- to-image generation via masked generative transformers

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text- to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023. 3

    2023 arXiv

  4. [12]

    Softvq-vae: Efficient 1-dimensional con- tinuous tokenizer, 2024

    Hao Chen, Ze Wang, Xiang Li, Ximeng Sun, Fangyi Chen, Jiang Liu, Jindong Wang, Bhiksha Raj, Zicheng Liu, and Emad Barsoum. Softvq-vae: Efficient 1-dimensional con- tinuous tokenizer, 2024. 3

    2024

  5. [13]

    Pixart: Fast training of diffu- sion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart: Fast training of diffu- sion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 3

    arXiv

  6. [14]

    Gentron: Delving deep into diffusion transformers for image and video generation

    Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xi- ang, and Juan-Manuel Perez-Rua. Gentron: Delving deep into diffusion transformers for image and video generation. arXiv preprint arXiv:2312.04557, 2023. 3

    2023 arXiv

  7. [15]

    Janus- pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus- pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811,

    arXiv

  8. [16]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1–113, 2023. 1

    2023

  9. [17]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems , 36,

  10. [18]

    Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning, 2025. 3

    2025

  11. [19]

    Autoregressive video generation with- out vector quantization, 2024

    Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation with- out vector quantization, 2024. 3

    2024

  12. [20]

    Imagenet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5

    2009

  13. [21]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 1

    2018 arXiv

  14. [22]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 3, 6

    2021

  15. [23]

    DreamLLM: Synergistic multimodal com- prehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. DreamLLM: Synergistic multimodal com- prehension and creation. In The Twelfth International Con- fer...

    2024

  16. [24]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 1, 3, 6

    2021

  17. [25]

    Scaling rectified flow transformers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yan- nik Marek, and Robin Rombach. Scaling rectified flow tr...

    2024

  18. [26]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218 ,

    arXiv

  19. [27]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. 3

    2014

  20. [28]

    Google. Bard. https://bard.google.com/, 2023. 1

    2023

  21. [29]

    Dart: Denoising autoregressive trans- former for scalable text-to-image generation.arXiv preprint arXiv:2410.08159, 2024

    Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, and Shuangfei Zhai. Dart: Denoising autoregressive trans- former for scalable text-to-image generation.arXiv preprint arXiv:2410.08159, 2024. 3

    arXiv

  22. [30]

    Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024

    Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024. 3

    2024

  23. [31]

    Zipar: Accel- erating autoregressive image generation through spatial lo- cality

    Yefei He, Feng Chen, Yuanyu He, Shaoxuan He, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipar: Accel- erating autoregressive image generation through spatial lo- cality. arXiv preprint arXiv:2412.04062, 2024. 3

    2024 arXiv

  24. [32]

    Scaling laws for autoregressive generative modeling

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. 1

    2010 arXiv

  25. [33]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 6

    2017

  26. [34]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 3

    2022 arXiv

  27. [35]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural informa- tion processing systems, 33:6840–6851, 2020. 1

    2020

  28. [37]

    Cascaded diffusion models for high fidelity image generation

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249–2281,

  29. [38]

    Training compute-optimal large language mod- els

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language mod- els. arXiv preprint arXiv:2203.15556, 2022. 1

    2022 arXiv

  30. [39]

    Arflow: Autogressive flow with hybrid linear attention

    Mude Hui, Rui-Jie Zhu, Songlin Yang, Yu Zhang, Zirui Wang, Yuyin Zhou, Jason Eshraghian, and Cihang Xie. Arflow: Autogressive flow with hybrid linear attention. arXiv preprint arXiv:2501.16085, 2025. 3

    2025 arXiv

  31. [40]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13...

    2024

  32. [41]

    Scal- ing up gans for text-to-image synthesis

    Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scal- ing up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124–10134, 2023. 3, 6

    2023

  33. [43]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  34. [44]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019. 3

    2019

  35. [45]

    Auto-encoding vari- ational bayes

    Diederik P Kingma and Max Welling. Auto-encoding vari- ational bayes. arXiv preprint arXiv:1312.6114, 2013. 3

    2013 arXiv

  36. [46]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11523–11532, 2022. 1, 6

    2022

  37. [47]

    Playground v2

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Lin- miao Xu, and Suhail Doshi. Playground v2. 5: Three in- sights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024. 3

    2024 arXiv

  38. [48]

    Autoregressive image generation without vec- tor quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization. arXiv preprint arXiv:2406.11838, 2024. 3

    2024 arXiv

  39. [49]

    Imagefolder: Autoregres- sive image generation with folded tokens

    Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregres- sive image generation with folded tokens. arXiv preprint arXiv:2410.01756, 2024. 3

    2024 arXiv

  40. [50]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 3

    2023 arXiv

  41. [51]

    Open-sora plan: Open-source large video generation model

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00131,

    arXiv

  42. [52]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024. 3

    2024 arXiv

  43. [53]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural infor- mation processing systems, 36, 2024. 3

    2024

  44. [54]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongx- uan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems , 35: 5775–5787, 2022. 3

    2022

  45. [55]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022. 3

    2022 arXiv

  46. [56]

    Unified-io 2: Scaling autoregressive mul- timodal models with vision, language, audio, and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Anirud- dha Kembhavi. Unified-io 2: Scaling autoregressive mul- timodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023. 3

    2023 arXiv

  47. [57]

    Open-magvit2: An open-source project toward democratizing auto-regressive visual gener- ation

    Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual gener- ation. arXiv preprint arXiv:2409.04410, 2024. 3, 6

    2024 arXiv

  48. [58]

    Groma: Localized visual tokenization for grounding multimodal large language models

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. arXiv preprint arXiv:2404.13013, 2024. 3

    2024 arXiv

  49. [59]

    Janusflow: Harmonizing au- toregression and rectified flow for unified multimodal un- derstanding and generation, 2024

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Ji- aying Liu, and Chong Ruan. Janusflow: Harmonizing au- toregression and rectified flow for unified multimodal un- derstanding ...

    2024

  50. [60]

    OpenAI. Chatgpt. https://openai.com/blog/ chatgpt, 2022. 1

    2022

  51. [61]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  52. [62]

    Training lan- guage models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training lan- guage models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35...

    2022

  53. [63]

    Byte la- tent transformer: Patches scale better than tokens, 2024

    Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. Byte la- tent transformer: Patches scale better than tokens, 2024. 3

    2024

  54. [64]

    Freeman, and Yu-Xiong Wang

    Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T. Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. arXiv preprint arXiv:2412.01827, 2024. 3

    2024 arXiv

  55. [65]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 4195– 4205, 2023. 3, 6

    2023

  56. [66]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 3

    2023 arXiv

  57. [67]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 3

    2023 arXiv

  58. [68]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069, 2024. 3

    2024 arXiv

  59. [69]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. article, 2018. 1

    2018

  60. [70]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  61. [71]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Ma- chine Learning Research, 21(1):5485–5551, 2020. 1

    2020

  62. [72]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Interna- tional Conference on Machine Learning, pages 8821–8831. PMLR, 2021. 1, 3

    2021

  63. [73]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022. 3

    2022 arXiv

  64. [74]

    Gener- ating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Gener- ating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019. 1

    2019

  65. [75]

    Flowar: Scale-wise au- toregressive image generation meets flow matching

    Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Flowar: Scale-wise au- toregressive image generation meets flow matching. arXiv preprint arXiv:2412.15205, 2024. 3

    2024 arXiv

  66. [76]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3, 6

    2022

  67. [77]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Sali- mans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural In- forma...

    2022

  68. [78]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information process- ing systems, 29, 2016. 6

    2016

  69. [79]

    Stylegan- xl: Scaling stylegan to large diverse datasets

    Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan- xl: Scaling stylegan to large diverse datasets. In ACM SIG- GRAPH 2022 conference proceedings , pages 1–10, 2022. 6

    2022

  70. [80]

    Patch- level training for large language models

    Chenze Shao, Fandong Meng, and Jie Zhou. Patch- level training for large language models. arXiv preprint arXiv:2407.12665, 2024. 3

    2024 arXiv

  71. [81]

    Taming scalable visual tok- enizer for autoregressive image generation

    Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang, Ying Shan, and Limin Wang. Taming scalable visual tok- enizer for autoregressive image generation. arXiv preprint arXiv:2412.02692, 2024. 3

    2024 arXiv

  72. [82]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 1, 3

    2010 arXiv

  73. [83]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 1, 3

    2019

  74. [84]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024. 5

    2024

  75. [85]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024. 2, 3, 5, 6, 7, 8

    2024 arXiv

  76. [87]

    Generative pretraining in mul- timodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in mul- timodality. arXiv preprint arXiv:2307.05222, 2023

    2023 arXiv

  77. [88]

    Hart: Efficient visual generation with hybrid autoregressive transformer

    Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Jun- song Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint arXiv:2410.10812, 2024. 3

    arXiv

  78. [89]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 ,

    arXiv

  79. [90]

    Gem- ini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gem- ini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 3

    2023 arXiv

  80. [91]

    Internlm: A multilingual language model with progressively enhanced capabilities, 2023

    InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023. 1

    2023

  81. [92]

    Large concept models: Language mod- eling in a sentence representation space

    LCM The, Lo ¨ıc Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R Costa-juss`a, et al. Large concept models: Language mod- eling in a sentence representation space. arXiv preprint arX...

    2024 arXiv

  82. [93]

    Visual autoregressive modeling: Scalable im- age generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable im- age generation via next-scale prediction. arXiv preprint arXiv:2404.02905, 2024. 3, 6, 7

    2024 arXiv

  83. [94]

    Metamorph: Multimodal understanding and generation via instruction tuning, 2024

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning, 2024. 3

    2024

  84. [95]

    Llama: Open and efficient foundation language mod- els

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language mod- els. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  85. [96]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  86. [97]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 1, 3

    2017

  87. [98]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 1

    2017

  88. [99]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 3

    2024 arXiv

  89. [100]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 3

    2024 arXiv

  90. [101]

    Parallelized autoregressive visual genera- tion

    Yuqing Wang, Shuhuai Ren, Zhijie Lin, Yujin Han, Haoyuan Guo, Zhenheng Yang, Difan Zou, Jiashi Feng, and Xihui Liu. Parallelized autoregressive visual genera- tion. arXiv preprint arXiv:2412.15119, 2024. 3

    2024 arXiv

  91. [102]

    Maskbit: Embedding-free image generation via bit tokens

    Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xi- aohui Shen, Daniel Cremers, and Liang-Chieh Chen. Maskbit: Embedding-free image generation via bit tokens. arXiv:2409.16211, 2024. 3

    2024 arXiv

  92. [103]

    Emer- gent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Bar- ret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emer- gent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022. 1

    2022 arXiv

  93. [104]

    Bloom: A 176b-parameter open-access multi- lingual language model

    BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´c, Daniel Hess- low, Roman Castagn´e, Alexandra Sasha Luccioni, Franc ¸ois Yvon, et al. Bloom: A 176b-parameter open-access multi- lingual language model. arXiv preprint arXiv:2211.05100,

    arXiv

  94. [105]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024. 3

    arXiv

  95. [106]

    Liq- uid: Language models are scalable multi-modal generators

    Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Heng- shuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liq- uid: Language models are scalable multi-modal generators. arXiv preprint arXiv:2412.04332, 2024. 3

    2024 arXiv

  96. [107]

    Omnigen: Unified image generation

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340, 2024. 3

    2024 arXiv

  97. [108]

    Show-o: One single transformer to unify multimodal under- standing and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal under- standing and generation. arXiv preprint arXiv:2408.12528,

    arXiv

  98. [109]

    Raphael: Text-to- image generation via large mixture of diffusion paths.arXiv preprint arXiv:2305.18295, 2023

    Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to- image generation via large mixture of diffusion paths.arXiv preprint arXiv:2305.18295, 2023. 3

    2023 arXiv

  99. [110]

    Baichuan 2: Open large-scale language models

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. 1

    2023 arXiv

  100. [111]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024. 3

    2024 arXiv

  101. [112]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021. 1, 6

    2021 arXiv

  102. [113]

    Scaling autore- gressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autore- gressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022. 1, 3

    2022 arXiv

  103. [114]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2023

  104. [115]

    Lan- guage model beats diffusion–tokenizer is key to visual gen- eration

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Lan- guage model beats diffusion–tokenizer is key to visual gen- eration. arXiv preprint arXiv:2310.05737, 2023. 3

    arXiv

  105. [116]

    Randomized autoregressive visual gen- eration

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Randomized autoregressive visual gen- eration. 2024. 3

    2024

  106. [117]

    An image is worth 32 tokens for reconstruction and generation

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. NeurIPS,

  107. [118]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 1

    2022 arXiv

  108. [119]

    Gpt4roi: Instruction tuning large language model on region-of- interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of- interest. arXiv preprint arXiv:2307.03601, 2023. 3

    2023 arXiv

  109. [120]

    Qlip: Text-aligned visual tokenization unifies auto- regressive multimodal understanding and generation

    Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Kr ¨ahenb¨uhl, and De-An Huang. Qlip: Text-aligned visual tokenization unifies auto- regressive multimodal understanding and generation. arXiv preprint arXiv:2502.05178, 2025. 3

    2025 arXiv

  110. [121]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 3

    2024 arXiv

  111. [122]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023. 3

    2023

  112. [123]

    Stabilize the latent space for image au- toregressive modeling: A unified perspective, 2024

    Yongxin Zhu, Bocheng Li, Hang Zhang, Xin Li, Linli Xu, and Lidong Bing. Stabilize the latent space for image au- toregressive modeling: A unified perspective, 2024. 3

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.