REVIEW 5 cited by
Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The recent surge of generative AI has been fueled by the generative power of diffusion probabilistic models and the scalable capabilities of large language models. Despite their potential, it remains elusive whether diffusion language models can solve general language tasks comparable to their autoregressive counterparts. This paper demonstrates that scaling diffusion models w.r.t. data, sizes, and tasks can effectively make them strong language learners. We build competent diffusion language models at scale by first acquiring knowledge from massive data via masked language modeling pretraining thanks to their intrinsic connections. We then reprogram pretrained masked language models into diffusion language models via diffusive adaptation, wherein task-specific finetuning and instruction finetuning are explored to unlock their versatility in solving general language tasks. Experiments show that scaling diffusion language models consistently improves performance across downstream language tasks. We further discover that instruction finetuning can elicit zero-shot and few-shot in-context learning abilities that help tackle many unseen tasks by following natural language instructions, and show promise in advanced and challenging abilities such as reasoning.
Forward citations
Cited by 5 Pith papers
-
Sentence Curve Language Models
Predicting a B-spline 'sentence curve' target instead of static word embeddings improves diffusion language models on translation benchmarks.
-
A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models
Visual token pruning causes severe loss in discrete diffusion MLLMs; only from-scratch models on long-answer tasks recover via late denoising, so redundancy is recoverability, not dispensability.
-
ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute
ParaThinker trains LLMs for native parallel reasoning and reports 7 to 12 percent higher accuracy on math benchmarks over sequential thinking with modest latency overhead.
-
Theoretical Benefit and Limitation of Diffusion Language Model
Masked diffusion language models have a metric-dependent efficiency tradeoff: near-optimal perplexity in constant steps, but sequence-level correctness needs linearly many steps in the worst case.
-
A Survey on Diffusion Language Models
A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.
Discussion (0). Continue with ORCID to comment.