Pith. sign in

REVIEW 12 cited by

Energy-Based Diffusion Language Models for Text Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.21357 v4 pith:ODNYPODD submitted 2024-10-28 cs.CL cs.LG

Minkai Xu , Tomas Geffner , Karsten Kreis , Weili Nie , Yilun Xu , Jure Leskovec , Stefano Ermon , Arash Vahdat This is my paper · ORCID
classification cs.CLcs.LG
keywords modelsdiffusiongenerationautoregressivelanguagemodelenergy-basedsampling
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Despite remarkable progress in autoregressive language models, alternative generative paradigms beyond left-to-right generation are still being actively explored. Discrete diffusion models, with the capacity for parallel generation, have recently emerged as a promising alternative. Unfortunately, these models still underperform the autoregressive counterparts, with the performance gap increasing when reducing the number of sampling steps. Our analysis reveals that this degradation is a consequence of an imperfect approximation used by diffusion models. In this work, we propose Energy-based Diffusion Language Model (EDLM), an energy-based model operating at the full sequence level for each diffusion step, introduced to improve the underlying approximation used by diffusion models. More specifically, we introduce an EBM in a residual form, and show that its parameters can be obtained by leveraging a pretrained autoregressive model or by finetuning a bidirectional transformer via noise contrastive estimation. We also propose an efficient generation algorithm via parallel important sampling. Comprehensive experiments on language modeling benchmarks show that our model can consistently outperform state-of-the-art diffusion models by a significant margin, and approaches autoregressive models' perplexity. We further show that, without any generation performance drop, our framework offers a 1.3$\times$ sampling speedup over existing diffusion models. Reproduced code is available at https://github.com/MinkaiXu/Energy-Diffusion-LLM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Gumbel Distillation for Parallel Text Generation

    cs.CL 2026-03 conditional novelty 6.0 of 10

    Conditioning parallel decoders on Gumbel noise sampled from an autoregressive teacher's Gumbel-Max process improves generation quality on LM1B and OpenWebText.

  2. IDLM: Inverse-distilled Diffusion Language Models

    cs.LG 2026-02 reject novelty 6.0 of 10

    IDLM distills pretrained discrete diffusion language models into few-step generators, cutting inference steps by 4–64× with roughly matched GenPPL and entropy.

  3. Fine-Tuning Masked Diffusion for Provable Self-Correction

    cs.LG 2025-10 conditional novelty 6.0 of 10

    PRISM fine-tunes any masked diffusion model with a binary-cross-entropy loss so its new head provably estimates per-token quality p(x_i=y_i|y⊕m_i) and can remask low-quality tokens at inference.

  4. Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows

    cs.LG 2025-07 conditional novelty 6.0 of 10

    TarFlowLM models language in a continuous latent space with transformer-based autoregressive normalizing flows, using mixture-CDF and Rosenblatt couplings, and reports competitive NELBO on TEXT8 and OpenWebText.

  5. Theoretical Benefit and Limitation of Diffusion Language Model

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Masked diffusion language models have a metric-dependent efficiency tradeoff: near-optimal perplexity in constant steps, but sequence-level correctness needs linearly many steps in the worst case.

  6. Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Masked diffusion models trained order-agnostically can solve puzzles better than autoregressive models when inference unmasking order is chosen adaptively by confidence.

  7. Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Diffusion models trained with denoising score matching can follow coarse image rules but cannot reliably reproduce fine-grained inter-feature rules, and a two-layer network analysis shows a constant error for such rules.

  8. Categorical Schr\"odinger Bridge Matching

    cs.LG 2025-02 conditional novelty 6.0 of 10

    The paper proves that discrete-time iterative Markovian fitting converges to the Schrödinger Bridge on finite discrete spaces and introduces CSBM, a practical matching algorithm for categorical data.

  9. Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers

    cs.LG 2026-07 conditional novelty 5.5 of 10

    SNLP reduces symbolic FHE bootstraps from 53 to 20 on a 0.5B model with +1.2% PPL degradation and lower polynomial-error amplification than sequential inference.

  10. On the Separability of Information in Diffusion Models

    cs.LG 2025-09 conditional novelty 5.0 of 10

    Diffusion models devote most of their information budget to class-agnostic texture, and the small class-relevant slice is what classifier-free guidance amplifies.

  11. A Survey on Diffusion Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.

  12. Generative AI for Autonomous Driving: A Review

    cs.CV 2025-05 conditional novelty 2.0 of 10

    A review of generative models (VAEs, GANs, diffusion, transformers, LLMs) applied to map generation, scenario generation, trajectory prediction, and motion planning for autonomous driving.

Pith tools