{"id":"e1940fc9-8679-4553-b356-0f7ca61a764f","arxiv_id":"2412.15321","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Averaging neighboring image tokens into patches during training lets autoregressive image models train faster and generate higher-quality images, with inference unchanged.","lead":"This paper proposes a training method for autoregressive image models that groups image tokens into patches and predicts the next patch, gradually switching to single-token prediction. It reports better image quality and about 40 percent lower training cost than the LlamaGen baseline on ImageNet, with no changes to the model architecture.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-seed FID comparisons and benchmark-tuned defaults leave open that part of the reported 0.56–1.04 FID gain is seed or schedule noise; the central quality-at-0.6x-cost claim needs a multi-seed, compute-matched control.","rationale":"I agree with the reader's CONDITIONAL assessment: the paper presents a simple, well-ablated method with consistent gains across four model sizes, and I found no derivation error or circular reasoning. My principal concern is not the transferability of the local-correlation assumption, though that is a legitimate secondary worry; it is the evidentiary basis for the headline quantitative claim. The reader's rationale also mentioned the absence of error bars and post-hoc hyperparameter selection, and that is the more load-bearing issue for the specific claim of 'improving FID by up to 1.0 at 0.6x training cost.' A multi-seed, compute-matched control would settle whether the claimed gains are robust or partially an artifact of single-run evaluation and benchmark tuning. Because this concern does not move me away from the reader's CONDITIONAL verdict, I set verdict_should_be to UNCHANGED and mark agreement as partial: the weakest_assumption in the reader's verdict focuses on token correlation and transfer, whereas my load-bearing concern is the statistical reliability and fairness of the comparison.","tokens_in":18079,"tokens_out":16843,"duration_ms":155059,"concrete_test":"Run NPP-L and LlamaGen-L with 5 seeds each under identical settings (same data order, learning-rate schedule, batch size, and CFG protocol) and report mean +/- std FID/IS for the final checkpoints. Additionally train a LlamaGen-L control at roughly the same total GFLOPs as NPP-L (about 172 full-sequence epochs, matching NPP-L's ~47.95 GFLOPs) and evaluate it with the same sampling protocol. If the FID gap between NPP-L and LlamaGen-L collapses to within one standard deviation, or the compute-matched LlamaGen-L reaches the same FID as NPP-L, the central quality-at-reduced-cost claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method is internally coherent: Eq. 2 and Eq. 3 define a consistent patch-level training objective, the multi-scale schedule ends in vanilla NTP, and the reported FLOP/throughput measurements are plausible. The load-bearing issue is evidential rather than derivational. Every FID number in Tables 1–3 comes from a single training run, and the default settings (#L=2, lambda=1/2, PS=2x2) were selected on the same ImageNet 256 benchmark. The headline 'up to 1.0 FID' is the maximum over model sizes of a single-seed difference; with 50k-sample FID evaluation plus training-seed variation, margins on the order of 0.3–0.5 at L/XL sizes are not clearly outside noise. The cost-quality claim is also supported mainly by comparing NPP at 300 epochs (47.95 GFLOPs for NPP-L) against LlamaGen at 300 epochs (83.54 GFLOPs), not against a LlamaGen model trained at the same total FLOP budget. A compute-matched baseline is needed to confirm that the cheaper curriculum, rather than some incidental advantage, is what delivers the FID gain. This does not invalidate the paper, but it makes the central quantitative claim less secure than the current tables suggest.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Next Patch Prediction (NPP), a training-only modification of LlamaGen-style autoregressive image generation. Image token embeddings are grouped into non-overlapping patches and averaged to form shorter patch sequences; the model is trained with a patch-wise cross-entropy loss that scores the predicted patch distribution against the K ground-truth tokens in each patch. A multi-scale schedule starts with a larger patch size and ends with vanilla 1x1 next-token prediction, so inference is identical to standard NTP. On ImageNet 256x256, NPP reports FID improvements over LlamaGen at four model sizes (e.g., 2.76 vs 3.80 for the L model) while reducing training FLOPs to roughly 0.53-0.63x, without adding trainable parameters or modifying the inference procedure.","tokens_in":18426,"tokens_out":6515,"duration_ms":51865,"significance":"If the reported results hold, this is a simple, parameter-free, plug-and-play improvement for autoregressive visual generation: it lowers training cost and improves FID without changing the architecture or the inference-time sampling process. The paper's strengths are its conceptual simplicity, the consistency of the FID gain across four model sizes, and the clean ablations isolating the main design choices. The method is internally coherent: Equation (2) and Equation (3) define a valid patch-level training objective, and the multi-scale schedule ends in vanilla NTP. However, the quantitative headline is currently supported only by single-seed comparisons, with default hyperparameters selected on the same benchmark, so the evidence is not yet as secure as the tables suggest.","major_comments":[{"comment":"All FID/IS numbers in Tables 1-3 come from a single training run per configuration, and FID is evaluated on 50,000 stochastic samples. With sampling noise and training-seed variation, per-point differences such as 0.74 for NPP-XL vs LlamaGen-XL and 0.56 for NPP-XXL vs LlamaGen-XXL are not clearly outside noise. Please provide at least three training seeds (or, at minimum, multiple evaluation sampling runs) for the headline LlamaGen-vs-NPP comparisons, and report the resulting mean and spread.","section":"4.1, 4.2, Table 1"},{"comment":"The default settings used in the main tables, specifically lambda=1/2, #L=2, and patch size 2x2, were selected as the best values on the same ImageNet 256x256 benchmark. This is a form of selection on the evaluation set. Please either pre-specify the schedule, select defaults on a validation split, or show that the main FID conclusion is stable across the reasonable range of lambda and #L values reported in Table 3.","section":"4.5, Table 3"},{"comment":"The cost-quality claim compares NPP at 300 epochs against LlamaGen at 300 epochs, not against a compute-matched baseline. Because NPP consumes only 0.53-0.63x the FLOPs, a LlamaGen model trained to the same total FLOP budget (for example, more epochs or a larger model) is needed to establish that the cheaper training curriculum, rather than an under-trained baseline, is what produces the observed FID gain.","section":"4.3, Table 2"}],"minor_comments":[{"comment":"In Equation (3), the predicted distribution Pred_i is written as a function of patch i, but the same distribution is scored against each of the K ground-truth tokens; please clarify the notation, for example by writing the loss explicitly as summing K times log P(p_i) or by defining Pred_i as a distribution over the K token indexes.","section":"3.2, Eq. (3)"},{"comment":"For NPP-L, the reported cost ratio is 0.57x, which is lower than the nominal 0.625x implied by lambda=1/2 and a 2x2 patch size. Please explain whether the discrepancy comes from attention FLOPs being super-linear in sequence length or from measurement details.","section":"4.3, Table 2"},{"comment":"The text says \"path average operation\" but the intended term is \"patch average operation\"; the same typo appears in the Figure 3 caption.","section":"3.2, Figure 3"},{"comment":"The sentence \"without introducing additional trainable parameters or specifically signing a custom image tokenizer\" should read \"specifically designing a custom image tokenizer.\"","section":"3.1"},{"comment":"The introduction refers to \"VQV AE\"; this should be \"VQVAE.\"","section":"1"},{"comment":"The caption states \"up to 4.0x training speed up,\" but Table 2 reports approximately 3.95x for NPP-L vs LlamaGen-L-384; please align the wording to \"about 4x\" or report the exact number.","section":"Figure 1 caption"},{"comment":"The reference list contains duplicate entries (e.g., [36] and [37] are the same paper, and [86] and [87] are the same) and some incomplete entries; please clean up the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is evidential rather than derivational: the central cost-quality claim rests on single-seed FID numbers and on defaults selected on the same benchmark. I would not reject the paper, but the authors should be asked to provide multi-seed results or confidence intervals for the headline comparisons, and to add a compute-matched LlamaGen baseline. If those experiments confirm the reported effect, the paper would be a solid contribution to the autoregressive visual generation literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Next Patch Prediction is a simple, well-ablated training recipe for autoregressive image generation, and the empirical story is consistent: across four model sizes (111M to 1.4B), the multi-scale patch curriculum improves FID by 0.56 to 1.04 while cutting training FLOPs to roughly 0.53–0.63x, with no inference change and no extra parameters. That is a practically useful result, and I believe it is genuinely new relative to the cited patch-level LLM training and VAR, because the specific combination—average-pooled patch embeddings on an off-the-shelf VQGAN, coarse-to-fine schedule ending in vanilla NTP—is not in the prior work.\n\nThe paper does the right things methodologically. The objective in Eq. 2–3 is coherent: average K token embeddings, supervise all K ground-truth tokens with the same predicted distribution. The multi-scale schedule ends at 1x1, so inference is untouched. Ablations on patch size, lambda, and #L are clean and show the defaults are reasonable, not knife-edge. The FLOP and throughput numbers in Table 2 are plausible and clearly reported. Code is promised.\n\nThe soft spots are evidential, not derivational. Every FID in Tables 1–3 comes from a single training run, and the default settings (lambda=1/2, #L=2, PS=2) were selected on the same ImageNet 256 benchmark. The margins at L/XL/XXL (0.76–1.04) are large enough that I doubt they are pure noise, but the paper would be much stronger with multi-seed means and error bars. The cost-quality claim would also be cleaner with a compute-matched LlamaGen control—train LlamaGen for the same total FLOPs as NPP (roughly 170 epochs at the L size) and show NPP still wins. Right now the comparison is NPP at 300 epochs vs LlamaGen at 300 epochs, which bundles the curriculum with the lower per-epoch cost. The RoPE choice (patch-grid positions rather than averaged positions) is justified only by an unpublished pilot study; that is a minor weakness, but an easy one to fix in revision. Transfer to other tokenizers or datasets is untested, so the method's breadth is open.\n\nOverall, this is a solid, honest empirical paper. The central claim likely holds, but the exact magnitude of the gain and the attribution to the curriculum (vs. just the FLOP savings) deserve referee scrutiny. I'd send it to review and ask for the compute-matched and multi-seed controls. The paper is primarily for people working on AR image generation and training efficiency; it is a useful datapoint for that community.","headline":"Simple, well-ablated patch-curriculum training for AR image generation with consistent FID gains and ~0.6x cost; the main weakness is single-seed evidence and lack of a compute-matched baseline, not the method itself.","tokens_in":19019,"tokens_out":4149,"would_cite":true,"duration_ms":35803,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By predicting a 2x2 patch of image tokens at once, this paper claims to reduce autoregressive training cost to ~0.6x and improve ImageNet FID by up to 1.0 without changing inference.","keywords":["autoregressive image generation","next patch prediction","next token prediction","multi-scale training","training efficiency","ImageNet 256x256","VQGAN tokenizer","visual generation"],"falsifier":"Run the same NPP training on a tokenizer designed to be spatially decorrelated (for example, a tokenizer that allocates high-frequency details to adjacent tokens, or on texture-rich data like satellite images) and compare FID against vanilla next-token training at matched compute; if the patch objective fails to improve or hurts FID, the central claim is falsified. An internal check: replace the patch-grid RoPE with averaged patch positions; if that closes the FID gap, the patch-averaging objective is not the whole effect.","tokens_in":17884,"feed_emoji":"🖼️","tokens_out":5157,"duration_ms":37424,"temperature":0.7,"pith_summary":"This paper claims that autoregressive image generation is better modeled at the level of patches rather than individual tokens. The authors propose Next Patch Prediction (NPP): group image tokens into non-overlapping 2x2 patches, average their embeddings to form one patch embedding, and train the transformer to predict the whole next patch using a cross-entropy loss over all K ground-truth tokens in that patch. Because the input sequence is K times shorter, training cost drops to about 0.6x. Training is scheduled from coarse to fine, starting with large patches and ending with the standard 1x1 next-token prediction, so the inference procedure is unchanged. On the class-conditional ImageNet 256x256 benchmark, NPP improves FID by up to 1.0 point across model sizes from 111M to 1.4B parameters without adding parameters or a custom tokenizer.","feed_headline":"Next-patch prediction makes image autoregression cheaper and better","feed_subtitle":"Averaging 2x2 token embeddings into patch targets improves FID up to 1.0 at 0.6x training cost on ImageNet.","key_machinery":"The load-bearing machinery is the patch-wise cross-entropy loss combined with intra-patch averaging. For each patch $p_i$ containing $K$ token embeddings, the patch embedding is $E(p_i) = \\frac{1}{K}\\sum_{k=1}^{K} E(x_i^k)$, and the model's prediction for the next patch is scored with $L = -\\frac{1}{N}\\sum_{i=1}^{N/K}\\sum_{k=1}^{K} \\log(P^{i})$, where the same predicted distribution is applied to every ground-truth token index in the patch. The Multi-scale Patch Grouping schedule (patch levels $\\#L$ and segment factor $\\lambda$) starts with a large patch size and shrinks it to $1\\times 1$, so NPP gradually turns into vanilla NTP. Rotary positional embeddings are kept at the patch-grid positions rather than averaged positions, a design choice said to come from a pilot study.","core_discovery":"The central discovery is that the standard next-token objective used in autoregressive image models is suboptimal for image data, because image tokens carry low information density and are spatially redundant. NPP instead defines a patch-level objective: given patch embeddings formed by averaging the token embeddings in each cell, the model predicts the distribution over the next patch, and the loss supervises that single predicted distribution against all K ground-truth token indices inside it. A multi-scale schedule anneals the patch size from 4x4 or 2x2 down to 1x1 during training, which transfers coarse structure learning into fine token-level learning. The resulting models consistently beat the vanilla next-token baseline (LlamaGen) on the same architecture, data, and inference setting, with the largest quality gain at the 343M-parameter scale (FID 2.76 vs 3.80). This is framed as a plug-and-play change to the training objective, not as a new architecture or tokenizer.","pith_inferences":["If the patch-averaging objective works because it denoises the training signal, then NPP should be most beneficial early in training, and the gains might persist at higher resolutions or on video where local correlation is even stronger; the paper does not test these settings.","The same coarse-to-fine annealing could be applied to other modalities, e.g., predicting averaged embeddings of consecutive tokens in language or audio before fine-token prediction, potentially yielding a similar efficiency-quality trade-off.","One testable consequence: during the patch phase, the model's single predicted distribution can in principle be sampled K times to generate all tokens of a patch at once, which would turn the training-time speed-up into a decoding-speed-up; NPP does not exploit this.","The RoPE decision to use patch-grid rather than averaged positions is untested; if averaged positions were used, the model might lose the implicit scale signal, which could be probed by an ablation at 384x384 resolution."],"forward_implications":["Training an autoregressive image model with NPP cuts compute to roughly 0.6x of the next-token baseline while increasing throughput by 1.3x-1.8x at matched model size.","Image quality improves: NPP reaches FID 2.76 for a 343M-parameter model, beating LlamaGen-L-384 (3.07) and Open-MAGVIT2-B (3.08) with less compute.","Because inference and architecture are unchanged, NPP can be applied to existing autoregressive visual generators without retraining the tokenizer or modifying sampling.","Scaling behavior holds from 111M to 1.4B parameters: every NPP variant outperforms its LlamaGen counterpart at 0.53x-0.63x of the training cost."],"supporting_citations":[{"why":"Supplies the autoregressive baseline (LlamaGen), model sizes, training recipe, and the FID/throughput numbers that NPP is compared against.","marker":"[85]"},{"why":"Provides the VQGAN image tokenizer whose discrete tokens are grouped into patches and reused by NPP without modification.","marker":"[24]"},{"why":"Justifies the central efficiency claim by establishing that transformer training cost scales linearly with input sequence length.","marker":"[42]"},{"why":"Defines the RoPE positional embedding whose placement on patch-grid versus averaged positions is a design choice studied in the method.","marker":"[84]"},{"why":"Represents the competing next-scale prediction approach that requires a custom multi-scale tokenizer, which NPP is contrasted against.","marker":"[93]"},{"why":"Provides a strong autoregressive baseline (Open-MAGVIT2) whose FID NPP surpasses, showing the gain is not achieved against a weak comparison.","marker":"[57]"}],"fun_headline_variants":["Patch-level prediction cuts training cost and boosts image generation FID","Predict next patch instead of next token to slash training cost and improve FID","Next-Patch Prediction: cheaper training, better autoregressive images","Patch targets beat token targets for image autoregression at 0.6x cost","Plug-and-play patch prediction: same model, 0.6x training, better FID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that nearby image tokens are alike enough that averaging their embeddings and asking for one shared prediction for all of them is a useful curriculum; the paper validates this on ImageNet 256x256 with a single VQGAN tokenizer, so the claim may not transfer to data or tokenizers with weak local correlations.","fun_headline_variants_meta":{"raw":{"variants":["Patch-level prediction cuts training cost and boosts image generation FID","Predict next patch instead of next token to slash training cost and improve FID","Next-Patch Prediction: cheaper training, better autoregressive images","Patch targets beat token targets for image autoregression at 0.6x cost","Plug-and-play patch prediction: same model, 0.6x training, better FID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000893,"raw_usage":{"total_tokens":3875,"prompt_tokens":998,"completion_tokens":2877,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":2777}},"tokens_in":614,"tokens_out":2877,"duration_ms":16679,"temperature":1.0,"reasoning_tokens":2777,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:32:10.551487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same NPP training on a tokenizer designed to be spatially decorrelated (for example, a tokenizer that allocates high-frequency details to adjacent tokens, or on texture-rich data like satellite images) and compare FID against vanilla next-token training at matched compute; if the patch objective fails to improve or hurts FID, the central claim is falsified. An internal check: replace the patch-grid RoPE with averaged patch positions; if that closes the FID gap, the patch-averaging objective is not the whole effect.","supporting_citations":[{"cited_title":"Roformer: Enhanced transformer with rotary position embedding","cited_arxiv_id":null,"evidence_quote":"Defines the RoPE positional embedding whose placement on patch-grid versus averaged positions is a design choice studied in the method."}],"review_version":1}