Pith. sign in

REVIEW 17 cited by

Simple and Effective Masked Diffusion Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.07524 v2 pith:YMK5LGUW submitted 2024-06-11 cs.CL cs.AIcs.LG

Subham Sekhar Sahoo , Marianne Arriola , Yair Schiff , Aaron Gokaslan , Edgar Marroquin , Justin T Chiu , Alexander Rush , Volodymyr Kuleshov This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords diffusionlanguagemodelsmaskedmodelingsimpleeffectiveobjective
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

    cs.AI 2026-05 conditional novelty 7.0 of 10

    Masked diffusion language models, not larger autoregressive LLMs, are the better building block for text-based world models in agentic RL, improving rollout fidelity, diversity, and downstream task success.

  2. DODO: Discrete OCR Diffusion Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Block-based discrete diffusion can transcribe documents in parallel, roughly matching autoregressive OCR accuracy while cutting inference time by up to about 3x in a lower-accuracy fast variant.

  3. Just on Time: Token-Level Early Stopping for Diffusion Language Models

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Jot, a token-level early stopping rule using a top-2 confidence ratio and spatial context, speeds up diffusion language model decoding by up to 19.6x with minor quality loss.

  4. DSB: Dynamic Sliding Block Scheduling for Diffusion LLMs

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Dynamic Sliding Block scheduling adapts the decoding block size and position in diffusion LLMs based on confidence, improving quality and speed over fixed blocks.

  5. CANDI: Hybrid Discrete-Continuous Diffusion Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    CANDI combines masked and Gaussian corruption in one noising process, letting discrete diffusion models use continuous gradients for joint updates and guidance.

  6. Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows

    cs.LG 2025-07 conditional novelty 6.0 of 10

    TarFlowLM models language in a continuous latent space with transformer-based autoregressive normalizing flows, using mixture-CDF and Rosenblatt couplings, and reports competitive NELBO on TEXT8 and OpenWebText.

  7. Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Diffusion Tree Sampling is a Monte Carlo tree search over denoising trajectories that propagates terminal rewards backward to sample from reward-aligned distributions, showing up to 10x compute savings on tested benchmarks.

  8. Neuro-Symbolic Generative Diffusion Models for Physically Grounded, Robust, and Safe Generation

    cs.LG 2025-06 conditional novelty 6.0 of 10

    NSD adds iterative constraint projection to both continuous and discrete diffusion sampling, achieving near-zero constraint violations across molecular, robotic, material, and language generation tasks.

  9. Learning Distributions over Permutations and Rankings with Factorized Representations

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Factorized codes for permutations let standard transformers learn arbitrary permutation distributions with guaranteed validity, and the paper adds a fast insertion-vector decoding theorem plus two new benchmarks.

  10. Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Adaptive Classifier-Free Guidance (A-CFG) re-masks low-confidence tokens in the unconditional input at each generation step, improving reasoning and planning accuracy for masked diffusion language models.

  11. Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    In overparameterized diffusion models, generalization happens first and memorization starts later, with the memorization time growing linearly with dataset size.

  12. Enabling Autoregressive Models to Fill In Masked Tokens

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Combining a frozen MLM and AR model with a trained linear layer enables autoregressive models to perform masked token infilling with KV-cached inference.

  13. TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Trajectory-aware commit gating (TILG + History Gate + capped extra promotion) improves or preserves DLLM accuracy while reducing steps and raising tokens-per-forward without retraining.

  14. On the Design Space of Discrete Diffusion Online Adaptation for Molecular Optimization

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Online fine-tuning of discrete diffusion models with complementary acquisition, CVaR shaping, density-entropy debiasing, replay, and validity control finds better molecules under fixed oracle budgets than offline fine...

  15. Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Training on the best of K generated candidates improves image, video, and language generative models, with the reported gains growing with scale and enabling single-pass end-to-end generation.

  16. T$^\star$: Progressive Block Scaling for Masked Diffusion Language Models Through Trajectory Aware Reinforcement Learning

    cs.CL 2026-01 conditional novelty 5.0 of 10

    A progressive RL-based curriculum, T⋆, doubles block sizes in masked diffusion LMs while keeping or improving math reasoning accuracy.

  17. Masked Diffusion Language Models with Frequency-Informed Training

    cs.CL 2025-09 conditional novelty 4.0 of 10

    Masked diffusion language models trained on 100M words match a hybrid GPT-BERT baseline on BabyLM tests, with a rare-word-focused masking variant.

Pith tools