{"id":"6a7f61d6-348d-4162-a4bb-b1c8ff250755","arxiv_id":"2412.06787","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Discrete Interpolants frames image generation, segmentation, and video generation as unmasking discrete [MASK] tokens, connecting masked generative models and discrete diffusion models.","lead":"A generative modeling paper proposes \"Discrete Interpolants\", a discrete-state framework that treats both image generation and segmentation as progressive unmasking of [MASK] tokens. The authors report strong results on MS-COCO, ImageNet 256, and FaceForensics, while claiming to unify masked generative and diffusion models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Conditional segmentation sampling uses clean condition tokens, but Eq. (5) trains with both modalities masked by the same κt; the inference input distribution is not supported by the training loss, so the 'one training, flexible conditioning' claim is unproven.","rationale":"The reader's weakest assumption concerns the implicit timestep argument in Section 3.2. That concern is plausible but is substantially supported by prior work on time-agnostic masked diffusion (Zheng et al. 2024; Sahoo et al. 2024), which the paper cites. I therefore do not regard the implicit timestep premise as the single most load-bearing risk. The more concrete gap is the conditional-sampling protocol for the joint modality model: the training loss in Eq. (5) always corrupts both modalities, while the Cityscapes sampling shown in Figs. 15-16 appears to keep one modality clean. If the condition is clean throughout sampling, the model is being queried at input combinations that were essentially never seen in training. This directly threatens the paper's second major contribution — segmentation as unmasking with 'only training once' — because the reported mIoU may reflect an unstated or ad hoc inference protocol rather than the proposed framework. This does not overturn the reader's conditional verdict: the generation results and ablations may still be valuable, but the paper needs a corrected or clarified conditioning protocol, and ideally a matched comparison, before the joint-training claim can be accepted. The SOTA overclaim on MS-COCO noted by the reader is also real, but it is a framing issue; the conditional-sampling gap is more central to the scientific contribution.","tokens_in":21962,"tokens_out":14943,"duration_ms":168069,"concrete_test":"On Cityscapes, retrain the same joint model and compare three inference protocols: (A) condition clean, target unmasked from [M] (the paper's apparent default); (B) condition corrupted at the same κt as the target at each sampling step; (C) train with condition always clean and target masked, then sample as in (A). Report mIoU for A, B, and C. Also compute the model's cross-entropy on held-out input pairs of type (x_1 clean, y_t fully masked) versus matched training-distribution pairs. If mIoU(A) is substantially worse than mIoU(B), or the held-out loss is much higher, the claimed 'one training, flexible conditional sampling' is not validated as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 defines the joint training loss (Eq. 5) over z_t = x_t ⊕ y_t, where both modalities are corrupted by the same scheduler: pt|1(xt|x1)pt|1(yt|y1). For any t>0, a training example contains a mixture of clean and [M] tokens in both modalities. The conditional sampling used for the Cityscapes experiments (Sec. 3.5, Figs. 15-16) instead feeds one modality as a clean condition while sampling the other from [M] throughout the unmasking chain. The probability that a training sample ever contains all condition tokens clean while all target tokens are masked is (1-κt)^{Lx} κt^{Ly}, which is negligible for typical token counts (Lx, Ly ≈ 256-1024) except near t=0 or t=1. Hence the model is evaluated on inputs far from its training distribution, and the mIoU numbers in Table 5 do not establish that segmentation-as-unmasking works under the stated 'train once' recipe. The paper neither masks the condition at inference nor trains with an unmasked-condition protocol; the flexible conditional sampling claim (abstract, Sec. 1) is therefore not supported by the loss in Eq. (5).","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces \"Discrete Interpolants,\" a discrete-state flow-matching framework for vision. It defines masking schedules κt that interpolate between fully masked tokens and data tokens, trains a model with masked cross-entropy to predict original tokens, and supports explicit-timestep, implicit-timestep, and MaskGIT-style sampling. It also proposes training once on image–segmentation-mask pairs to model a joint distribution and then sampling conditionally in either direction, framing semantic segmentation as an unmasking process. Experiments are reported on ImageNet 256, MS-COCO, FaceForensics, and Cityscapes, with the central claims being a unified design-space analysis of masked generative and discrete diffusion models and a demonstration that discriminative tasks can be recast as unmasking.","tokens_in":22203,"tokens_out":10391,"duration_ms":104255,"significance":"If the claims are supported, the paper would provide a useful single discrete-token recipe for generation, conditional generation, and dense prediction, and its explicit-versus-implicit timestep analysis is a practical contribution. The paper ships extensive ablations of sampling steps, softmax temperature, CFG scale, schedulers, Gumbel noise, and argmax churning, and it is transparent that several of these are empirical tuning findings rather than theoretical predictions. The theoretical scaffolding is largely imported from discrete flow matching and masked diffusion literature, so the novelty lies in the vision-domain unification and empirical study rather than in new theory. The results are potentially valuable but currently contain an overstatement about MS-COCO state-of-the-art and a training–inference mismatch in the conditional segmentation protocol that undermines the strongest new claim.","major_comments":[{"comment":"The sentence \"Our method achieves state-of-the-art performance compared to both continuous-state and discrete-state models\" is contradicted by Table 2: U-ViT attains FID 5.48 while the Implicit Timestep Model attains 5.65, so the continuous-state comparison is not state-of-the-art, and the Explicit Timestep Model is 6.03. The defensible claim is state-of-the-art among the discrete-state baselines listed (VQ-Diffusion at 19.75). Because the gap to U-ViT is 0.17 FID and no error bars or multiple seeds are reported, the present wording overstates the result; please restrict the claim to discrete-state models or reword it as competitive, and report variance or at least note single-run metrics.","section":"§4.2.1, Table 2"},{"comment":"Eq. (5) trains on z_t = x_t ⊕ y_t with both modalities corrupted by the same masking schedule, but the conditional sampling used for Table 5 and Figs. 15–16 conditions on one clean modality while unmasking the other. For typical token counts (Lx, Ly on the order of 256–1024), a training example in which all condition tokens are clean and all target tokens are masked occurs with probability (1−κt)^{Lx} κt^{Ly}, which is negligible except at t near 0 or 1. The model is therefore evaluated on inputs far outside the training distribution, so the mIOU/FID numbers in Table 5 do not establish the claimed \"train once, flexible conditional sampling\" recipe. The paper should train with an unmasked-condition protocol, mask the condition at inference according to κt, or provide an explicit ablation justifying the mismatch. In addition, §4.1 mentions a 0.1 conditional dropout for classifier-free guidance, but Eq. (5) contains no condition term, so it is unclear how the unconditional branch in Eq. (6) was trained.","section":"§3.4–§3.5, Eq. (5), Table 5"},{"comment":"The abstract and contribution list claim \"competitive\" results on FaceForensics compared to counterpart discrete-state models, but Table 4 compares only against Latte, a continuous-state model. No discrete-state video baseline is included, so the comparison class stated in the claims is not actually evaluated. Please add discrete-state video baselines (for example, MAGVIT or a discrete adaptation of Latte) or explicitly restrict the claim to the continuous-state comparison shown.","section":"§4.2.1, Table 4"},{"comment":"Table 5 reports mIOU of 89.1 and 90.1 and FID of 34.4 and 33.8 on Cityscapes without any baseline segmentation method, so the discriminative value of the segmentation-as-unmasking framework is not quantified relative to existing approaches. Even if the training–inference mismatch above is resolved, the reader cannot tell whether these mIOU numbers are strong, and the statement in §4.2.2 that the [MASK] token can be leveraged to reframe discriminative tasks should be supported by a comparison to at least one standard segmentation model or to prior discrete diffusion discriminative work.","section":"§4.2.2, Table 5"}],"minor_comments":[{"comment":"The heading \"Classifier-free Gudiance\" contains a typo and should read \"Classifier-free Guidance.\"","section":"Section 3.5"},{"comment":"The opening sentence \"Our work Stochastic Interpolant extends discrete flow matching theory to vision tasks\" is imprecise; the theory is from Gat et al. [22] and the paper builds on and applies it, so the sentence should say \"builds on\" or \"applies\" rather than \"extends,\" or it should clearly specify what is newly extended.","section":"Section 5"},{"comment":"The notation in Algorithm 1 is informal: the line `p(x1|xt+Δt, t+Δt; θ) ← Cat[δxt(t + Δt) + ut(xt)Δt]` conflates a distribution with a sampling rule, uses `δxt(t + Δt)` ambiguously, and the pseudocode should explicitly state that only masked positions are resampled and unmasked positions are carried over, matching the prose description.","section":"Algorithm 1"},{"comment":"The paper should state whether code and checkpoints are released; the FID, FVD, and mIOU tables cannot otherwise be independently reproduced, especially given the sensitivity of FID to the evaluation protocol.","section":"Section 4.1"},{"comment":"The sentence \"we share the mask schedule between two different modalities, we find it empirically works well\" is a run-on sentence, and the sharing assumption is presented only as an empirical finding; a brief explanation of why a shared schedule is expected to work would help readers assess the design choice.","section":"Section 3.4"},{"comment":"Scheduler names are not fully harmonized: Table 1 lists \"Arccos,\" Figure 2 uses \"Arcsine\" and \"Arccos,\" and Figure 3 uses \"ArcCos\"; please unify the naming convention across tables, figures, and text.","section":"Table 1 and Figures 2–3"}],"recommendation":"major_revision","confidential_remarks":"The paper has useful empirical content and a clear presentation of a unified design space, but the central MS-COCO state-of-the-art claim is contradicted by the paper's own Table 2, and the conditional segmentation protocol in §3.4–§3.5 is not supported by the training loss in Eq. (5). Both issues are fixable within the manuscript's scope, but they need to be addressed before acceptance. The authors should also be asked to add discrete video baselines and at least one segmentation baseline, and to provide error bars or clearly label single-run results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere is my read on [MASK] is All You Need. The paper is a legitimate extension of discrete flow matching to large-scale vision. The core equations (1)-(4) are already in Gat et al. and the masked diffusion works, so the theoretical novelty is thin. What is genuinely new: the large-scale experiments on ImageNet256, MS-COCO, FaceForensics, and Cityscapes, plus the systematic design-space analysis across schedules, temperature, CFG, NFE, and implicit-vs-explicit timestep. The finding that an implicit timestep model matches the explicit one at scale is useful, and the ablation on masking cross-entropy with w(t)=1 rather than ELBO weight is worth reporting.\n\nBut there are soft spots, and one is load-bearing. First, the headline SOTA claim on MS-COCO is contradicted by their own Table 2: U-ViT gets FID 5.48, their best Implicit Timestep Model gets 5.65, and the text says \"state-of-the-art performance compared to both continuous-state and discrete-state models.\" That is simply false as written. No error bars anywhere, and FID differences around 0.2 are not meaningful without them. Second, the stress-test concern about conditional segmentation sampling is correct. Equation (5) trains on z_t where both modalities are masked by the same κt. At inference (Sec 3.5, Figures 15-16) they feed one modality clean and sample the other from full mask. For typical token counts, the probability of seeing all condition tokens clean while all target tokens are masked is negligible under the training distribution except near t=0 or t=1. So the model is evaluated out-of-distribution, and the mIoU numbers do not establish the \"train once, flexible conditioning\" claim. The paper never masks the condition at inference, nor trains with an unmasked-condition protocol. This is fixable, but as written the claim is unproven.\n\nThere are also reproducibility issues: no code release, and a long list of per-dataset tuned hyperparameters (scheduler, temperature, CFG, Gumbel noise, top-p, argmax). The design-space study is the paper's real contribution, and that stands despite the overclaiming.\n\nWho is this for? People working on discrete-state generative models and their connection to masked models. The ablation study is a useful reference, and the generation results are competitive even if not SOTA. I would send this to a serious referee, but with a clear request: fix the SOTA claim, add error bars, and either change the training to support unmasked-condition inference or explicitly weaken the conditional sampling claim. With those changes it could be a solid paper.\n\nRecommendation: engage, but require revision.","headline":"A useful large-scale empirical study of discrete flow matching with an overclaimed SOTA headline and a real train/inference gap in the segmentation-as-unmasking claim.","tokens_in":22791,"tokens_out":3257,"would_cite":true,"duration_ms":35578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One masked-token recipe unifies diffusion, masked generation, and segmentation in a single model.","keywords":["masked generative models","discrete diffusion models","discrete flow matching","implicit timestep","image segmentation as unmasking","token-based image generation","classifier-free guidance","video generation"],"falsifier":"Probe an implicit-timestep model's ability to read corruption level from the mask alone: train it on masked inputs generated by two different schedulers or timesteps that produce the same visible [MASK] pattern for the same clean sequence, then see whether supplying the true timestep changes the predicted clean-token distribution. If the timestep helps, the mask pattern alone does not encode the corruption level; equivalently, if a probe classifier cannot recover the remaining mask ratio from $x_t$ accurately, the implicit model has no channel for that information and the central premise fails.","tokens_in":21692,"feed_emoji":"🧩","tokens_out":9191,"duration_ms":89630,"temperature":0.7,"pith_summary":"This paper tries to establish that a single discrete-token recipe, built on progressively unmasking [MASK] tokens, is enough to cover what are usually treated as separate model families: masked generative models, discrete diffusion models, and a classic discriminative task such as semantic segmentation. The recipe, named Discrete Interpolants, interpolates between a fully masked token sequence and a real token sequence through a masking schedule, and a network learns to predict the clean tokens from whatever tokens remain. The authors argue that the timestep can be removed from the network, because the pattern of remaining [MASK] tokens already reveals the corruption level, and this is what makes the model simultaneously a diffusion model and a masked generative model. They further show that image segmentation can be recast as the same unmasking process, so one joint training on image-mask pairs yields both image-conditioned segmentation and mask-conditioned image generation. If these claims hold, the payoff is a shared architecture and training objective for generation, conditional generation, and dense prediction, with reported state-of-the-art results on MS-COCO and competitive results on ImageNet 256 and FaceForensics.","feed_headline":"One unmasking model handles generation and segmentation","feed_subtitle":"A single discrete-token model trained once can generate images, complete masked tokens, and segment scenes.","key_machinery":"Discrete interpolants driven by a masking schedule $\\kappa_t$ are the central object: the interpolated state is $p_{t|0,1}(x|x_0,x_1) = (1-\\kappa_t)\\delta_{[M]}(x) + \\kappa_t\\delta_{x_1}(x)$, and the vector field to learn is $u_t(x_t) = \\frac{\\dot\\kappa_t}{1-\\kappa_t}[p_{1|t}(x_1|x_t,t;\\theta) - \\delta_{x_t}(x)]$. The schedule ($\\kappa_t = t$ linear, cosine, quadratic, and others) controls how gradually real tokens replace the [MASK] token. The training signal is masked cross-entropy with a weighting $w(t)$, where only positions that are [MASK] in $x_t$ contribute; the paper finds this masking necessary to avoid overfitting in vision and finds $w(t)=1$ better than the ELBO-derived weight $\\dot\\kappa_t/(1-\\kappa_t)$. The other load-bearing design choice is the implicit-timestep network $p(x_1|x_t;\\theta)$, which omits $t$ and therefore behaves like a masked generative model during sampling; a final argmax over logits at the last step, called churning, removes leftover [MASK] tokens and fixes scheduler misalignment.","core_discovery":"The central claim is that unmasking is a common language for generation and dense prediction. Concretely, the paper defines a discrete interpolant under a masking schedule $\\kappa_t$, with $p_{t|0,1}(x|x_0,x_1) = (1-\\kappa_t)\\delta_{x_0}(x) + \\kappa_t\\delta_{x_1}(x)$, where $x_0$ is the all-[MASK] state and $x_1$ is the clean token sequence. A network trained with masked cross-entropy to predict $x_1$ from $x_t$ acts as the vector field of a discrete flow, and the same network can be sampled as an explicit-timestep diffusion model, as an implicit-timestep model, or with masked-generative greedy unmasking. The paper's distinctive move is dropping the timestep: because the scheduler is monotone, the pattern of [MASK] tokens in $x_t$ already encodes how corrupted the input is. That is the step that connects discrete diffusion to masked generative models and lets segmentation be framed as unmasking. With image and segmentation-mask tokens concatenated and masked under a shared schedule, one training run serves image-conditioned segmentation, mask-conditioned image generation, and joint modeling of the two modalities.","pith_inferences":["If the implicit-timestep premise holds for discrete tokens, the same time-agnostic idea is worth testing in continuous-state diffusion for editing and arbitrary-order sampling; the paper notes this possibility but does not demonstrate it.","Segmentation-as-unmasking should extend to any tokenizable dense prediction target, such as depth, surface normals, or object detection, since the paper's argument only relies on a shared discrete vocabulary and a masked joint distribution.","A practical testable extension is to use the residual [MASK]-token rate after sampling as a proxy for scheduler misalignment: if churning helps, the failure mode is mostly leftover masked tokens, and monitoring that rate could predict when a new scheduler is safe to use.","Because the paper identifies irreversible unmasking as the main limitation, adding a smoothing or corrector term to allow remasking could convert the framework into a fully reversible stochastic interpolant; this is a future direction the paper only sketches."],"forward_implications":["A single trained discrete model can be sampled in explicit-timestep diffusion style, implicit-timestep diffusion style, or masked-generative greedy style, using the same weights.","Segmentation becomes an unmasking task: after joint training on image and mask token pairs, one checkpoint can return segmentation masks conditioned on images or images conditioned on masks.","Removing the explicit timestep yields simpler, order-flexible sampling that is upper-bounded by the token length and can handle row-by-row or editing-style schedules where a global timestep is hard to define.","Masked cross-entropy with a constant weight plus a final argmax step improves low-step sampling and corrects the mismatch when sampling with a scheduler different from training.","The same discrete recipe scales from images to video; adapting the continuous-state Latte model to discrete tokens gives a better FVD on FaceForensics."],"supporting_citations":[{"why":"Defines MaskGIT, the masked generative baseline whose greedy confidence-based sampling the paper re-uses with diffusion-trained weights.","marker":"[9]"},{"why":"Discrete flow matching supplies the scheduler/interpolant formulation and the conditional coupling that the paper generalizes to large-scale vision.","marker":"[22]"},{"why":"Provides the Kolmogorov-equation discrete-state flow theory that justifies training a network as the vector field.","marker":"[8]"},{"why":"Shows absorbing discrete diffusion secretly models conditional distributions of clean data, supporting the removal of the explicit timestep.","marker":"[57]"},{"why":"Simplifies and generalizes masked diffusion, another basis for the implicit-timestep model design.","marker":"[63]"},{"why":"Supplies the U-ViT backbone and the baseline results that the MS-COCO comparison is built on.","marker":"[6]"},{"why":"Provides the SD-VQ-F8 tokenizer that turns images into the discrete tokens used across all experiments.","marker":"[60]"},{"why":"Supplies the ELBO weighting $w(t)$ that the paper tests and then replaces with $w(t)=1$ for visual quality.","marker":"[42]"}],"fun_headline_variants":["Unmasking: one model for both generation and segmentation","Single unmasking model unifies generation and segmentation","One training run: generate images and segment with unmasking","Discrete interpolants: unmasking bridges diffusion and masked models","From masked tokens to images and segmentation in one pass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a masked token sequence itself reveals how much corruption has been applied, so the network can safely ignore the timestep; if two different timesteps can produce indistinguishable masked inputs, the implicit model conflates corruption levels and the claimed bridge to masked generative models weakens.","fun_headline_variants_meta":{"raw":{"variants":["Unmasking: one model for both generation and segmentation","Single unmasking model unifies generation and segmentation","One training run: generate images and segment with unmasking","Discrete interpolants: unmasking bridges diffusion and masked models","From masked tokens to images and segmentation in one pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000988,"raw_usage":{"total_tokens":4224,"prompt_tokens":1018,"completion_tokens":3206,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":3127}},"tokens_in":634,"tokens_out":3206,"duration_ms":25468,"temperature":1.0,"reasoning_tokens":3127,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:21:28.683155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Probe an implicit-timestep model's ability to read corruption level from the mask alone: train it on masked inputs generated by two different schedulers or timesteps that produce the same visible [MASK] pattern for the same clean sequence, then see whether supplying the true timestep changes the predicted clean-token distribution. If the timestep helps, the mask pattern alone does not encode the corruption level; equivalently, if a probe classifier cannot recover the remaining mask ratio from $x_t$ accurately, the implicit model has no channel for that information and the central premise fails.","supporting_citations":[{"cited_title":"Discrete flow matching","cited_arxiv_id":null,"evidence_quote":"Discrete flow matching supplies the scheduler/interpolant formulation and the conditional coupling that the paper generalizes to large-scale vision."},{"cited_title":"Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2024","cited_arxiv_id":null,"evidence_quote":"Shows absorbing discrete diffusion secretly models conditional distributions of clean data, supporting the removal of the explicit timestep."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the SD-VQ-F8 tokenizer that turns images into the discrete tokens used across all experiments."}],"review_version":1}