REVIEW 16 cited by
Simple and Effective Masked Diffusion Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm
Forward citations
Cited by 16 Pith papers
-
Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL
Masked diffusion language models, not larger autoregressive LLMs, are the better building block for text-based world models in agentic RL, improving rollout fidelity, diversity, and downstream task success.
-
DODO: Discrete OCR Diffusion Models
Block-based discrete diffusion can transcribe documents in parallel, roughly matching autoregressive OCR accuracy while cutting inference time by up to about 3x in a lower-accuracy fast variant.
-
Just on Time: Token-Level Early Stopping for Diffusion Language Models
Jot, a token-level early stopping rule using a top-2 confidence ratio and spatial context, speeds up diffusion language model decoding by up to 19.6x with minor quality loss.
-
DSB: Dynamic Sliding Block Scheduling for Diffusion LLMs
Dynamic Sliding Block scheduling adapts the decoding block size and position in diffusion LLMs based on confidence, improving quality and speed over fixed blocks.
-
CANDI: Hybrid Discrete-Continuous Diffusion Models
CANDI combines masked and Gaussian corruption in one noising process, letting discrete diffusion models use continuous gradients for joint updates and guidance.
-
Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows
TarFlowLM models language in a continuous latent space with transformer-based autoregressive normalizing flows, using mixture-CDF and Rosenblatt couplings, and reports competitive NELBO on TEXT8 and OpenWebText.
-
Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models
Diffusion Tree Sampling is a Monte Carlo tree search over denoising trajectories that propagates terminal rewards backward to sample from reward-aligned distributions, showing up to 10x compute savings on tested benchmarks.
-
Neuro-Symbolic Generative Diffusion Models for Physically Grounded, Robust, and Safe Generation
NSD adds iterative constraint projection to both continuous and discrete diffusion sampling, achieving near-zero constraint violations across molecular, robotic, material, and language generation tasks.
-
Learning Distributions over Permutations and Rankings with Factorized Representations
Factorized codes for permutations let standard transformers learn arbitrary permutation distributions with guaranteed validity, and the paper adds a fast insertion-vector decoding theorem plus two new benchmarks.
-
Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking
Adaptive Classifier-Free Guidance (A-CFG) re-masks low-confidence tokens in the unconditional input at each generation step, improving reasoning and planning accuracy for masked diffusion language models.
-
Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models
In overparameterized diffusion models, generalization happens first and memorization starts later, with the memorization time growing linearly with dataset size.
-
TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding
Trajectory-aware commit gating (TILG + History Gate + capped extra promotion) improves or preserves DLLM accuracy while reducing steps and raising tokens-per-forward without retraining.
-
On the Design Space of Discrete Diffusion Online Adaptation for Molecular Optimization
Online fine-tuning of discrete diffusion models with complementary acquisition, CVaR shaping, density-entropy debiasing, replay, and validity control finds better molecules under fixed oracle budgets than offline fine...
-
Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
Training on the best of K generated candidates improves image, video, and language generative models, with the reported gains growing with scale and enabling single-pass end-to-end generation.
-
T$^\star$: Progressive Block Scaling for Masked Diffusion Language Models Through Trajectory Aware Reinforcement Learning
A progressive RL-based curriculum, T⋆, doubles block sizes in masked diffusion LMs while keeping or improving math reasoning accuracy.
-
Masked Diffusion Language Models with Frequency-Informed Training
Masked diffusion language models trained on 100M words match a hybrid GPT-BERT baseline on BabyLM tests, with a rare-word-focused masking variant.
Discussion (0). Continue with ORCID to comment.