{"id":"f343d7d4-331e-428f-82aa-f2ed4fc1a750","arxiv_id":"2412.18390","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A recurrent token-prediction model that adds noise during quantization and generates images by predicting discrete codes over ten steps, reaching FID 2.56 on ImageNet 256x256.","lead":"RDPM generates images by predicting quantized codes for ten recurrent steps, using the same cross-entropy loss as language models. On ImageNet 256x256 it reports FID 2.56 with 602M parameters at 10 steps, close to much slower diffusion models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The generation loop hinges on predicting codes whose targets depend on the unobserved latent v; the paper never shows this conditional distribution is learnable, so the method's central mechanism is unverified.","rationale":"The paper is well-engineered and the reported FID results are encouraging, but the central mechanism rests on a learning problem that the paper does not analyze. The reader identified the same load-bearing assumption: the training target in Eq. (6) depends on the original latent v, which is absent from the conditioning inputs. I agree with that assessment and sharpen it by noting that the issue is not merely identifiability but the magnitude of the irreducible conditional entropy. If that entropy is high, the transformer's code predictions are near-random and the recurrent accumulation in Algorithm 2 will not track any real image's tokenization. A teacher-forced accuracy measurement, or an estimate of the conditional entropy, would directly settle whether the concern lands. The existing FID numbers suggest the model learns something, but they do not demonstrate that the code-prediction mechanism is functioning as the paper claims. I would keep the verdict conditional: the authors should release code and report the predictability analysis before the central claim is accepted.","tokens_in":13667,"tokens_out":11161,"duration_ms":107997,"concrete_test":"Use the released tokenizer and generative model (or retrain a small version) on a held-out ImageNet split. For each step t, compute the top-1/top-5 code prediction accuracy of RDPM's transformer under teacher forcing, and compare it to the accuracy of a trivial constant or prior predictor. Also estimate the empirical conditional entropy H(C_t | epsilon_t, y, t, z'_{t-1}) by evaluating the model's predictive distribution on held-out codes. If the accuracy is close to 1/4096 or the entropy is close to log 4096, the target codes are effectively unpredictable from the available conditioning, and RDPM's recurrent generation cannot be carrying the information the decoder needs; if accuracy is substantially above chance, the reader's concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2, Eq. (6) defines the generative predictor as v'_t = f(epsilon_t, y, t, z'_{t-1}), but the training target v'_t = Q(alpha_t(v - z'_{t-1}) + beta_t * epsilon_t) always contains the original latent v, which is not available to f. At fixed inputs, v varies over the class-conditional distribution, so C_t is not a deterministic function of the inputs; the cross-entropy loss in Eq. (7) is therefore minimizing an irreducible conditional entropy. The paper provides no measurement of this entropy or of the model's teacher-forced code accuracy, and no argument that the conditioned code distribution is concentrated. If the conditional entropy is close to log K (4096), the transformer cannot make informative code predictions, and the recurrent accumulation z'_t = z'_{t-1} + v'_t in Algorithm 2 will drift away from any real image's tokenization. The success of the entire framework depends on this learnability, yet the only evidence is the aggregate FID; no teacher-forced versus free-running comparison is reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RDPM, a framework for class-conditional image generation on ImageNet 256×256. A VAE is trained with a diffusion-based quantization scheme (Algorithm 1) that encodes the latent into T=10 discrete code maps by progressively mixing Gaussian noise with residual vectors and quantizing. A transformer is then trained (Eq. 6) to predict the next code map from the current noise, class label, timestep, and accumulated quantized vectors, using a cross-entropy loss (Eq. 7). At inference, Algorithm 2 recurrently predicts code maps starting from zero accumulation and decodes the final sum. The method reports FID 2.56 at 602M parameters with 10 sampling steps, comparable to recent VAR models and better than several prior discrete-tokenizer baselines.","tokens_in":1763,"tokens_out":1674,"duration_ms":65809,"significance":"The empirical results are noteworthy: they demonstrate that a discrete-token generative model trained with a GPT-style cross-entropy loss can reach FID levels close to continuous diffusion models on ImageNet 256×256 while using only 10 steps. The paper also promises code and weight release, which would support reproducibility. However, the theoretical framing as a “diffusion” process is not established, and the central learnability assumption of the recurrent predictor is not examined. If the authors can provide diagnostic evidence for the conditional predictability of the codes and temper the diffusion claims, this would be a useful empirical contribution to discrete-token generation.","major_comments":[{"comment":"The training target v'_t = Q(alpha_t(v - z'_{t-1}) + beta_t epsilon_t) depends on the original latent v, which is not an input to the predictor f(epsilon_t, y, t, z'_{t-1}). Since z'_{t-1} is a lossy function of v (it is a sum of quantized residuals), multiple source latents can map to the same z'_{t-1}, and for fixed inputs (epsilon_t, y, t, z'_{t-1}) the target code is not uniquely determined. The cross-entropy loss in Eq. (7) is therefore minimizing an irreducible conditional entropy, but the paper provides no measurement of this entropy or of the model's teacher-forced code accuracy. Without such diagnostics, it is unclear whether the free-running accumulation in Algorithm 2 stays on the training manifold. Please report teacher-forced top-1 accuracy, an estimate of the conditional entropy of target codes given the inputs, and a comparison of teacher-forced versus free-running FID or reconstruction quality.","section":"Section 3.2, Eq. (6) and Algorithm 2"},{"comment":"The repeated claim that RDPM “solves diffusion probabilistic models” or performs a “diffusion process on discrete-value domains” is not supported by the methodology. Algorithm 2 is an autoregressive code predictor: there is no reverse Markov chain, no SDE discretization, no ELBO, and no derivation that iterating Eq. (6) maps standard Gaussian noise to the data distribution. The forward process in Algorithm 1 is not inverted in any formal sense; the model simply learns a conditional distribution over codes. The authors should either provide a formal connection between the recurrent prediction objective and a diffusion process, or revise the terminology to describe an autoregressive model over residual quantized codes with noise-conditioned inputs.","section":"Abstract, Sections 1, 3.2, 4.5"},{"comment":"The noise schedule hyperparameter phi is selected by evaluating FID-50K on the same metric used for the final reported performance. The pow schedule with phi=0.75 is chosen because it yields the best FID-50K, which introduces selection bias into the headline numbers. Please report results with a fixed, untuned schedule (e.g., a standard linear or cosine schedule) and clearly state which hyperparameters were tuned on the evaluation metric, or use a validation split for hyperparameter selection.","section":"Section 4.3, Figure 5, Table 4"}],"minor_comments":[{"comment":"The paper claims to pioneer “Discrete Diffusion” but does not cite prior work on discrete diffusion models such as D3PM (Austin et al., 2021) or CDCD (Hoogeboom et al., 2021). Please add references and position RDPM relative to these lines of work.","section":"Introduction / Related Work"},{"comment":"The text says alpha_t increases from small to large and reaches 1 at t=T, while beta_t^2 + alpha_t^2 = 1. This is the opposite of the standard diffusion direction, where noise increases over time. Please clarify the intended direction and justify why this is called a diffusion process.","section":"Section 3.1, Eq. (2)"},{"comment":"The quantization operation Q(·) is described as nearest-neighbor search, but the loss in Eq. (4) requires gradients to flow through the quantization step. Please specify whether a straight-through estimator or other gradient approximation is used, as is standard in VQ-VAE and VQGAN.","section":"Section 3.1, Algorithm 1"},{"comment":"The caption says rows 2 and 3 represent discrete quantized tokens, but the figure itself does not label which row corresponds to which method. Please add explicit labels to the row illustrations.","section":"Figure 3"},{"comment":"The row labeled “Raw” is unclear; please define what “raw” means (e.g., no CFG, no Gumbel noise, greedy decoding) in the caption or text.","section":"Table 3"},{"comment":"The displayed cross-entropy formula contains an extra “log” inside the softmax expression; it should be Ct,i,j log( e^{bv'_{t,i,j}} / sum_k e^{bv'_{t,i,k}} ) or equivalent. Please correct the notation.","section":"Eq. (7)"},{"comment":"The phrase “predict the direction of the next data distribution” is unclear; consider replacing with “predict the next quantized vector v'_t” or “estimate the next code map”.","section":"Algorithm 2, line 3"},{"comment":"Reference [38] contains a typo: “Tian et at.” should be “Tian et al.” Please fix.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper’s central claim is overstated relative to the actual method, which is essentially an autoregressive model over residual quantized codes. The main technical risk is the learnability of the code prediction from partial state; the authors should provide direct evidence (teacher-forced accuracy, conditional entropy, free-running diagnostics) rather than relying only on aggregate FID. The related work also omits the existing discrete diffusion literature, which may be intentional to highlight novelty but should be addressed. The empirical comparison is useful and the method appears competitive, so a major revision with the requested diagnostics and a reframed narrative could make this acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core of this paper is a new recipe: instead of quantizing an image latent in one shot or in deterministic residuals, they inject Gaussian noise at each residual step, quantize the noisy residual, and then train a transformer to predict the next code recurrently from the accumulated codes plus the noise. That specific combination is not in RQ-VAE, RQ-Transformer, MaskGIT, VAR, or MAR, as far as I can tell. And it works: 10-step generation on ImageNet 256 gives FID 2.56 at 602M parameters, close to DiT-XL/2's 2.27, with far fewer forward passes. The ablations in Tables 2-4 are useful, and the paper is clearly written.\n\nThe soft spot is the packaging, not the engineering. Calling this a diffusion model is a stretch. Algorithm 2 is an autoregressive code predictor with added noise; there is no diffusion SDE, no reverse Markov chain, no ELBO, and no derivation that iterating Eq. (6) maps pure noise to the data distribution. The \"discrete diffusion\" framing is at best a loose analogy.\n\nOn the specific learnability worry: the stress-test note says the prediction target v'_t depends on the unobserved latent v, so the model is asked to predict an essentially random code. That's only half right. The model also conditions on z'_{t-1}, the accumulated codes from previous steps, which is a function of v. So the conditional distribution is over residuals given previous residual codes, much like any hierarchical VQ generative model. The real gap is that the paper never measures teacher-forced code accuracy or the conditional entropy, so we don't know how concentrated that distribution is. If it's close to uniform over 4096 codes, the whole approach would fall apart; if it's concentrated, the method is fine. They should report that number. This is a missing experiment, not a demonstrated flaw.\n\nOther smaller issues: no error bars on the FID numbers, no comparison to RQ-Transformer at a matched parameter budget, and no baseline from the established discrete diffusion literature (e.g., D3PM). The noise schedule parameter phi is tuned on FID-50K, which is a minor circularity for the headline number. They also promise code and weights; that should be enforced.\n\nOverall, this is a solid empirical contribution that deserves a serious referee. I'd send it to review; with the framing fixed and the missing accuracy numbers added, it's a publishable contribution for the fast token-based generation community.","headline":"A genuinely novel discrete-token generation recipe with competitive ImageNet FID, but the 'diffusion' framing outstrips the method; the learnability worry is real but missing evidence, not a demonstrated flaw.","tokens_in":14478,"tokens_out":4284,"would_cite":false,"duration_ms":36447,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that diffusion probabilistic models can be solved by recurrent token prediction, converting the denoising process into a sequence of GPT-style classification steps over a vector-quantized codebook.","keywords":["recurrent diffusion probabilistic model","discrete diffusion","token prediction","vector quantization","image generation","GPT-style loss","ImageNet"],"falsifier":"Fix an image and a step t, then draw many noise samples epsilon_t while keeping z'_{t-1} fixed, and record the empirical distribution of the ground-truth code C_t. If the RDPM transformer's predicted distribution over the codebook at that (z'_{t-1}, epsilon_t, y, t) diverges sharply from this empirical distribution, the recurrent chain is not learning the intended denoising transition, and the method's claim of solving diffusion via token prediction would be falsified.","tokens_in":13397,"feed_emoji":"🖼️","tokens_out":6627,"duration_ms":57660,"temperature":0.7,"pith_summary":"RDPM claims a new way to run diffusion: instead of denoising continuous latents, it converts an image into a stack of vector-quantized codes, one per timestep, and trains a transformer to predict the next timestep's code from the noise, class label, timestep, and previously accumulated codes. Because the loss is cross-entropy over codebook entries, the optimization exactly matches GPT-style next-token prediction, which the authors argue opens the door to unified multimodal models. On ImageNet 256x256, RDPM reaches FID 2.56 in only 10 sampling steps with a 602M-parameter transformer, outperforming prior discrete-token methods and approaching continuous diffusion models. The paper frames its contribution as a new paradigm—'discrete diffusion'—rather than a marginal accuracy gain.","feed_headline":"Diffusion done as next-token prediction in 10 steps","feed_subtitle":"Encoding images into quantized codes lets a transformer denoise step by step with a GPT-style loss.","key_machinery":"The central object is the diffusion-based image tokenizer of Algorithm 1 combined with the recurrent token predictor of Algorithm 2. The tokenizer mixes Gaussian noise into the latent over T steps, each time quantizing the residual against a shared codebook, so an image becomes T code maps. The generator is a transformer f that maps (epsilon_t, y, t, z'_{t-1}) to the next code map, with the accumulated quantized vectors z'_{t-1} carrying information forward; the cross-entropy loss in Eq. (7) is identical in form to language-model next-token prediction, which is what the paper says unifies discrete and continuous signal generation.","core_discovery":"The central discovery is that a diffusion process can be 'solved' by recurrent token prediction: the forward process adds Gaussian noise to the VAE latent and quantizes the residual each step, producing T discrete codes; the reverse process is a transformer that, at each step, predicts the code for that step given the noise, class condition, timestep, and the codes accumulated so far. This turns denoising into a sequence of classification problems with a GPT-style cross-entropy loss, and the paper demonstrates that ten such recurrent steps are enough to reach FID 2.56 on ImageNet 256x256 at 602M parameters.","pith_inferences":["If the learnability assumption holds at scale, RDPM suggests a path toward a single model that handles text and continuous modalities with one cross-entropy objective, potentially simplifying multimodal training recipes.","The T-step tokenization increases the number of tokens per image by a factor of T over a single VQ code map; the paper's success at T=10 implies the transformer can exploit temporal structure across timesteps, which might transfer to video generation where the timestep axis and the frame axis could be merged.","A direct comparison with a version using a standard VQ-VAE tokenizer (single code map) under the same recurrent predictor would isolate how much of the gain comes from the diffusion-based tokenization versus the recurrent prediction itself; the paper does not report that ablation.","Because inference injects fresh Gaussian noise at every step, the model's diversity may be tunable by the noise schedule and Gumbel temperature; this suggests a testable knob for controlling the precision-diversity trade-off in discrete diffusion."],"forward_implications":["Only 10 recurrent steps are needed for high-quality generation, far fewer than the hundreds of steps typical of continuous diffusion, while preserving quality.","The loss function matches GPT-style next-token prediction, so the same optimization framework used for text can be applied to image, video, and audio generation.","The diffusion-based image tokenizer reduces discretization information loss compared to single-step VQ, as shown by reconstruction FID dropping to 1.52 with the weight and bias tricks.","RDPM outperforms prior methods that use discrete visual tokenizers on ImageNet 256x256, including MaskGIT and VAR at comparable scales.","The approach is not limited to images; the tokenization applies to any continuous signal domain."],"supporting_citations":[{"why":"Supplies the diffusion forward/noise paradigm that RDPM's tokenization and generation build on.","marker":"[16]"},{"why":"Grounds the latent-space diffusion formulation and deterministic sampling perspective that motivates the recurrent denoising view.","marker":"[35]"},{"why":"Provides the VQGAN-style composite loss (reconstruction, perceptual, adversarial) used for training the tokenizer.","marker":"[43]"},{"why":"Supplies the residual quantization mechanism that RDPM generalizes to T diffusion steps.","marker":"[23]"},{"why":"Supplies the Gumbel-noise decoding and linearly increasing guidance used in sampling, and the MaskGIT baseline as comparison.","marker":"[6]"},{"why":"Supplies the transformer backbone and AdaLN-Zero conditioning used by the recurrent predictor.","marker":"[29]"},{"why":"Supplies classifier-free guidance used in the conditional generation.","marker":"[15]"}],"fun_headline_variants":["Diffusion as next-token prediction, 10 steps to FID 2.56","RDPM: recurrent token prediction solves diffusion in 10 steps","GPT-style loss for image diffusion with 10-step inference","Discrete diffusion by predicting token codes, fast and accurate","Diffusion denoising reimagined as token classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a model can learn to predict the next quantized code from the noise, the class, the timestep, and the codes already produced, even though the original image encoding that determined that code is never shown to it.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion as next-token prediction, 10 steps to FID 2.56","RDPM: recurrent token prediction solves diffusion in 10 steps","GPT-style loss for image diffusion with 10-step inference","Discrete diffusion by predicting token codes, fast and accurate","Diffusion denoising reimagined as token classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1501,"prompt_tokens":921,"completion_tokens":580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":491}},"tokens_in":537,"tokens_out":580,"duration_ms":5511,"temperature":1.0,"reasoning_tokens":491,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:43:31.701595+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix an image and a step t, then draw many noise samples epsilon_t while keeping z'_{t-1} fixed, and record the empirical distribution of the ground-truth code C_t. If the RDPM transformer's predicted distribution over the codebook at that (z'_{t-1}, epsilon_t, y, t) diverges sharply from this empirical distribution, the recurrent chain is not learning the intended denoising transition, and the method's claim of solving diffusion via token prediction would be falsified.","supporting_citations":[{"cited_title":"Maskgit: Masked generative image transformer","cited_arxiv_id":null,"evidence_quote":"Supplies the Gumbel-noise decoding and linearly increasing guidance used in sampling, and the MaskGIT baseline as comparison."},{"cited_title":"Autoregressive image generation using residual quantization","cited_arxiv_id":null,"evidence_quote":"Supplies the residual quantization mechanism that RDPM generalizes to T diffusion steps."}],"review_version":1}