Pith. sign in

REVIEW 18 cited by

Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.02908 v6 pith:JHF2OAQS submitted 2024-09-04 cs.LG cs.AIcs.CL

Kaiwen Zheng , Yongxin Chen , Hanzi Mao , Ming-Yu Liu , Jun Zhu , Qinsheng Zhang This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords modelsdiffusionsamplingmaskedmdmscategoricalgenerationtheoretically
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Masked diffusion models (MDMs) have emerged as a popular research topic for generative modeling of discrete data, thanks to their superior performance over other discrete diffusion models, and are rivaling the auto-regressive models (ARMs) for language modeling tasks. The recent effort in simplifying the masked diffusion framework further leads to alignment with continuous-space diffusion models and more principled training and sampling recipes. In this paper, however, we reveal that both training and sampling of MDMs are theoretically free from the time variable, arguably the key signature of diffusion models, and are instead equivalent to masked models. The connection on the sampling aspect is drawn by our proposed first-hitting sampler (FHS). Specifically, we show that the FHS is theoretically equivalent to MDMs' original generation process while significantly alleviating the time-consuming categorical sampling and achieving a 20$\times$ speedup. In addition, our investigation raises doubts about whether MDMs can truly beat ARMs in text generation. We identify, for the first time, an underlying numerical issue, even with the commonly used 32-bit floating-point precision, which results in inaccurate categorical sampling. We show that it lowers the effective temperature both theoretically and empirically, and the resulting decrease in token diversity makes previous evaluations, which assess the generation quality solely through the incomplete generative perplexity metric, somewhat unfair.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Masked diffusion enables coherent beat tracking

    cs.SD 2026-08 conditional novelty 6.0 of 10

    Masked diffusion with independent beat/downbeat masking, a balanced unmasking schedule, and inter-step peak picking reduces incoherent beat-tracking outputs and improves GTZAN scores.

  2. From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Masked diffusion language models fail to exploit their any-order interface because of positional uncertainty; insertion-based (FlexMDM) and latent-segment (LatentMDM) variants recover distinct any-order inference beha...

  3. Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes

    cs.LG 2026-07 conditional novelty 6.0 of 10

    CO2Jump couples text and image denoising through cross-modal attention and remasking, achieving best joint accuracy on three concurrent-generation tasks.

  4. CaRE Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models

    cs.AI 2026-06 reject novelty 6.0 of 10

    CaRE shows that uncontrolled temperature and compute explain most MAUVE differences between MDLM remasking strategies, with high-entropy remasking reducing MAUVE under stochastic unmasking.

  5. Token Time Continuous Diffusion for Language Modeling

    cs.CL 2026-05 conditional novelty 6.0 of 10

    A continuous diffusion language model where each token denoises at its own rate—sure tokens first—improves few-step generation over discrete samplers and roughly matches global-time continuous models.

  6. Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Under an exact oracle denoiser derived from a ground-truth Markov chain, SEDD, MDLM, LLaDA, and ReMDM samplers show large transition-level error at few diffusion steps; GenPPL/MAUVE can improve or stay flat while tran...

  7. CANDI: Hybrid Discrete-Continuous Diffusion Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    CANDI combines masked and Gaussian corruption in one noising process, letting discrete diffusion models use continuous gradients for joint updates and guidance.

  8. Fine-Tuning Masked Diffusion for Provable Self-Correction

    cs.LG 2025-10 conditional novelty 6.0 of 10

    PRISM fine-tunes any masked diffusion model with a binary-cross-entropy loss so its new head provably estimates per-token quality p(x_i=y_i|y⊕m_i) and can remask low-quality tokens at inference.

  9. Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    TraceRL aligns the RL objective with the DLM's actual step-by-step decoding, producing TraDo-4B/8B models that beat autoregressive baselines on math reasoning.

  10. Any-Order Flexible Length Masked Diffusion

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlexMDM is a discrete diffusion model that provably supports any-order generation over variable-length sequences by learning an insertion expectation alongside the unmasking posterior, validated by length-fidelity, ma...

  11. Neuro-Symbolic Generative Diffusion Models for Physically Grounded, Robust, and Safe Generation

    cs.LG 2025-06 conditional novelty 6.0 of 10

    NSD adds iterative constraint projection to both continuous and discrete diffusion sampling, achieving near-zero constraint violations across molecular, robotic, material, and language generation tasks.

  12. Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking

    cs.LG 2025-05 conditional novelty 6.0 of 10

    EB-Sampler dynamically unmasks multiple low-entropy tokens per function evaluation, accelerating masked diffusion model sampling by 2-3x with negligible accuracy loss.

  13. Discrete Markov Bridge

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Discrete Markov Bridge learns the forward rate matrix and the reverse score in a continuous-time Markov chain, achieving BPC 1.38 on Text8 and FID 11.63 on CIFAR-10.

  14. Theoretical Benefit and Limitation of Diffusion Language Model

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Masked diffusion language models have a metric-dependent efficiency tradeoff: near-optimal perplexity in constant steps, but sequence-level correctness needs linearly many steps in the worst case.

  15. Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Masked diffusion models trained order-agnostically can solve puzzles better than autoregressive models when inference unmasking order is chosen adaptively by confidence.

  16. TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Trajectory-aware commit gating (TILG + History Gate + capped extra promotion) improves or preserves DLLM accuracy while reducing steps and raising tokens-per-forward without retraining.

  17. Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Training on the best of K generated candidates improves image, video, and language generative models, with the reported gains growing with scale and enabling single-pass end-to-end generation.

  18. DLM-One: Diffusion Language Models for One-Step Sequence Generation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    DLM-One distills a continuous diffusion language model into a one-step student, achieving roughly 500x inference speedup while staying within a few percent of the teacher on BLEU, ROUGE, and BERTScore, with substantia...

Pith tools