pith. machine review for the scientific record. sign in

arxiv: 2605.07933 · v1 · submitted 2026-05-08 · 💻 cs.CL

Recognition: no theorem link

How to Train Your Latent Diffusion Language Model Jointly With the Latent Space

Viacheslav Meshchaninov , Alexander Shabalin , Egor Chimbulatov , Nikita Gushchin , Ilya Koziev , Alexander Korotin , Dmitry Vetrov
Authors on Pith no claims yet

Pith reviewed 2026-05-11 03:25 UTC · model grok-4.3

classification 💻 cs.CL
keywords latent diffusionlanguage modelsnon-autoregressive generationtext generationjoint trainingdiffusion modelscontinuous latent space
0
0 comments X

The pith

Joint training of latent encoder, diffusion model, and decoder yields faster non-autoregressive text generation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Latent diffusion models generate text by denoising continuous representations rather than discrete tokens, allowing parallel processing of entire sequences. The central challenge is building a latent space that is both easy to denoise and easy to turn back into readable tokens. This paper shows that training the encoder that creates the latent space together with the diffusion denoiser and the decoder produces stronger results than prior diffusion approaches for language. A four-part recipe avoids the collapse that occurs with naive joint training and delivers better text quality on standard benchmarks while running several times faster.

Core claim

By reshaping representations from a pre-trained language model with a trainable encoder and jointly optimizing it with the diffusion model and decoder using an MSE decoder loss, a diffusion-to-encoder warmup phase, adaptive timestep sampling, and decoder-input noise, the resulting model generates higher-quality text than existing discrete and continuous diffusion language models while being 2-13 times faster on OpenWebText and LM1B.

What carries the argument

The joint-training recipe of MSE decoder loss, diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise that shapes pre-trained representations into a latent space easy to denoise and decode.

If this is right

  • Non-autoregressive text generation via latent diffusion can outperform prior diffusion baselines in both quality and speed.
  • Each element of the training recipe contributes measurably to final generation performance.
  • Continuous latent spaces derived from pre-trained language models become practical for diffusion when the encoder is learned jointly.
  • Joint learning of the latent space is presented as a key step for making latent diffusion competitive with other text-generation approaches.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same joint-training pattern could be tested on other modalities that already use pre-trained encoders and latent diffusion.
  • Adaptive timestep sampling may reduce the need for manual hyperparameter schedules in future diffusion language models.
  • If the recipe generalizes, it could allow larger diffusion language models to be trained without separate pre-training stages for the latent space.

Load-bearing premise

The four-part training recipe is both necessary and sufficient to produce a latent space that supports high-quality joint training of encoder, diffusion model, and decoder.

What would settle it

Removing any one recipe component and retraining on OpenWebText, then measuring whether perplexity or generation metrics degrade relative to the full recipe.

Figures

Figures reproduced from arXiv: 2605.07933 by Alexander Korotin, Alexander Shabalin, Dmitry Vetrov, Egor Chimbulatov, Ilya Koziev, Nikita Gushchin, Viacheslav Meshchaninov.

Figure 1
Figure 1. Figure 1: Quality-diversity trade-off in text generation. Pareto frontiers are obtained by sweeping NFEs. The marker size denotes the generation time, and the yellow star shows the statistics of real texts. The proposed LDLM achieves the best trade-off between Gen. PPL (↓) and entropy (↑) on both OpenWebText and LM1B, while remaining faster than competing baselines. 1 Constructor University, Bremen, Germany; 2 HSE U… view at source ↗
Figure 2
Figure 2. Figure 2: The proposed joint training framework. Diffusion latents z0 are produced by applying a frozen pre-trained token encoder Eh followed by a trainable latent encoder E θ z to the input tokens w. The latents are decoded back to text through a trainable latent decoder Dθ h and token decoder Dθ w. Two-stage decoding. We also decode latents in two stages. During training, we perturb the input of the latent decoder… view at source ↗
Figure 3
Figure 3. Figure 3: (Left): Latent space smoothness for different decoder losses measured by Gen. PPL of decoded interpolated latents. (Middle and Right): Training loss dynamics for varying diffusion-to￾encoder warmup size Swu. To test this hypothesis more directly, we measure latent-space smoothness following prior work on latent diffusion for text [57, 29]: we interpolate between two random latent representations of real te… view at source ↗
Figure 4
Figure 4. Figure 4: Effect of decoder noise σdec on the model quality. (Left): Latent space state reflected by the diffusion loss and coordinate-wise latent standard deviation. (Middle): Decoder loss and reconstruction accuracy. (Right): Latent space smoothness measured as Gen. PPL of the decoded mean of two random latents. Adaptive timestep sampling. We compare uniform timestep sampling with the adaptive sampler, keeping all… view at source ↗
Figure 5
Figure 5. Figure 5: Diffusion loss w.r.t timestep for the uniform time t and adapted time u = F −1 (t). Decoder input noise. Finally, we study the effect of Gaussian noise injected into the latent decoder input. We compare several val￾ues of σdec and provide the training results in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Diffusion-to￾encoder warmup schedule. The coefficient γ(s) controls the strength of the diffusion gradient passed to the encoder during the first Swu training steps. This appendix gives the implementation details for the diffusion-to￾encoder warmup used in Section 5.2. We warm up only the gradient of the diffusion objective with respect to the latent encoder output. This is implemented with a stop-gradient… view at source ↗
Figure 7
Figure 7. Figure 7: Samples from LDLM trained on LM1B for varying number of steps. [PITH_FULL_IMAGE:figures/full_fig_p025_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Samples from LDLM trained on OWT (NFE: 128 and 256). [PITH_FULL_IMAGE:figures/full_fig_p026_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Samples from LDLM trained on OWT (NFE: 512 and 1024). [PITH_FULL_IMAGE:figures/full_fig_p027_9.png] view at source ↗
read the original abstract

Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation by operating on continuous text representations and denoising entire sequences in parallel. The major challenge in latent diffusion modeling is constructing a suitable latent space. In this work, we present the Latent Diffusion Language Model (LDLM), in which the latent encoder, diffusion model, and decoder are trained jointly. LDLM builds its latent space by reshaping the representations of a pre-trained language model with a trainable encoder, yielding latents that are easy to both denoise and decode into tokens. We show that naive joint training produces a low-quality diffusion model, and propose a simple training recipe consisting of an MSE decoder loss, diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise. Ablations show that each component substantially impacts generation performance. On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous diffusion language models while being $2{\text -}13\times$ faster, indicating that jointly learning the latent space is a key step toward making latent diffusion competitive for text generation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces the Latent Diffusion Language Model (LDLM), in which a latent encoder, diffusion model, and decoder are trained jointly to produce continuous representations suitable for non-autoregressive text generation. The authors observe that naive joint training yields poor diffusion models and therefore propose a four-part training recipe (MSE decoder loss, diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise). Ablations are reported to show that each component matters, and the resulting model is claimed to outperform prior discrete and continuous diffusion language models on OpenWebText and LM1B while delivering 2–13× faster sampling.

Significance. If the reported gains prove robust and reproducible, the work would be a meaningful step toward practical latent diffusion for language, by demonstrating that a jointly learned latent space can be both easy to denoise and easy to decode. The explicit ablation of the training recipe is a positive feature that helps isolate which design choices enable joint optimization.

major comments (2)
  1. [§4] §4 (Experiments): the central performance claims rest on benchmark numbers that are summarized qualitatively in the abstract but whose concrete values, baseline implementations, and statistical significance are not visible in the provided description; without these, the magnitude of the improvement and the 2–13× speedup cannot be assessed.
  2. [§3.3] §3.3 (Training recipe): the adaptive timestep sampling and decoder-input noise are described at a high level; the precise functional forms, schedules, and hyper-parameter ranges are needed to verify that the recipe is both necessary and sufficient for the claimed latent-space properties.
minor comments (2)
  1. [Abstract] The abstract would benefit from one or two concrete metric values (e.g., MAUVE or perplexity deltas) to make the performance claim immediately evaluable.
  2. [§2] Notation for the latent variable z and the encoder/decoder mappings should be introduced once in §2 and used consistently thereafter.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and have revised the manuscript to provide the requested details on experimental results and the training recipe.

read point-by-point responses

  1. Referee: [§4] §4 (Experiments): the central performance claims rest on benchmark numbers that are summarized qualitatively in the abstract but whose concrete values, baseline implementations, and statistical significance are not visible in the provided description; without these, the magnitude of the improvement and the 2–13× speedup cannot be assessed.

    Authors: We agree that the experimental section would benefit from greater explicitness. The manuscript already contains tables reporting the concrete perplexity, MAUVE, and diversity numbers on OpenWebText and LM1B together with the 2–13× wall-clock sampling times. In the revision we have added (i) explicit citations and hyper-parameter tables for every baseline implementation, (ii) standard deviations computed over three independent runs for all main results, and (iii) a short paragraph clarifying how the speedup was measured (average time to generate 1 024 tokens on a single A100). These changes make the magnitude of the gains directly verifiable without altering any claims. revision: yes

  2. Referee: [§3.3] §3.3 (Training recipe): the adaptive timestep sampling and decoder-input noise are described at a high level; the precise functional forms, schedules, and hyper-parameter ranges are needed to verify that the recipe is both necessary and sufficient for the claimed latent-space properties.

    Authors: We accept that the original description of the two components was insufficiently precise. In the revised Section 3.3 we now give the exact functional forms: adaptive timestep sampling draws t ∼ p(t) ∝ (T − t + 1)^−α with α = 0.8 and T = 1 000; decoder-input noise adds isotropic Gaussian noise whose variance schedule is σ(t) = 0.05 · (1 − t/T). The hyper-parameter ranges explored during development are listed in Appendix D, and we have inserted a compact algorithm box that shows the full joint-training loop. These additions allow readers to reproduce the recipe and to assess its necessity via the existing ablations. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper presents an empirical method for jointly training a latent encoder, diffusion model, and decoder for text generation. Its core claims rest on benchmark results (OpenWebText, LM1B) and ablations showing that the proposed recipe (MSE decoder loss, warmup, adaptive sampling, input noise) improves over naive joint training and prior discrete/continuous diffusion LMs. No derivation chain, first-principles prediction, or fitted parameter is redefined as an output; the latent space quality is validated externally rather than by construction from its own inputs. No self-citation is load-bearing for the central result.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on standard diffusion modeling assumptions plus the empirical effectiveness of the four training heuristics; no new physical or mathematical entities are introduced.

axioms (2)
  • domain assumption Pre-trained language model representations can be reshaped into a latent space suitable for diffusion
    Invoked in the description of how the encoder builds the latent space.
  • domain assumption Standard Gaussian diffusion process assumptions hold in the learned latent space
    Implicit in the use of a diffusion model on the continuous latents.

pith-pipeline@v0.9.0 · 5517 in / 1275 out tokens · 40654 ms · 2026-05-11T03:25:24.745358+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 58 canonical work pages · 4 internal anchors

  1. [1]

    Alayrac, J

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. a. Bi ´nkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: a visual langu...

    work page 2022

  2. [2]

    One billion word benchmark for measuring progress in statistical language modeling

    C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One billion word benchmark for measuring progress in statistical language modeling.arXiv preprint arXiv:1312.3005, 2013

    work page arXiv 2013

  3. [3]

    B. Chen, S. Bi, H. Tan, H. Zhang, T. Zhang, Z. Li, Y . Xiong, J. Zhang, and K. Zhang. Aligning visual foundation encoders to tokenizers for diffusion models.arXiv preprint arXiv:2509.25162, 2025

    work page arXiv 2025

  4. [4]

    T. Chen. On the importance of noise scheduling for diffusion models, 2023

    work page 2023

  5. [5]

    T. Chen, R. Zhang, and G. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning, 2023

    work page 2023

  6. [6]

    Deschenaux, C

    J. Deschenaux, C. Gulcehre, and S. S. Sahoo. The diffusion duality, chapter ii: Ψ-samplers and efficient curriculum.arXiv preprint arXiv:2602.21185, 2026

    work page arXiv 2026

  7. [7]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo- gies, volume 1 (long and short papers), pages 4171–4186, 2019

    work page 2019

  8. [8]

    Dieleman

    S. Dieleman. Generative modelling in latent space, 2025

    work page 2025

  9. [9]

    Dieleman, L

    S. Dieleman, L. Sartran, A. Roshannai, N. Savinov, Y . Ganin, P. H. Richemond, A. Doucet, R. Strudel, C. Dyer, C. Durkan, C. Hawthorne, R. Leblond, W. Grathwohl, and J. Adler. Continuous diffusion for categorical data, 2022

    work page 2022

  10. [10]

    Durkan, A

    C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural spline flows. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    work page 2019

  11. [11]

    Z. Gao, J. Guo, X. Tan, Y . Zhu, F. Zhang, J. Bian, and L. Xu. Empowering diffusion models on the embedding space for text generation. In K. Duh, H. Gomez, and S. Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 46...

    work page 2024

  12. [12]

    Gokaslan, V

    A. Gokaslan, V . Cohen, E. Pavlick, and S. Tellex. Openwebtext corpus.URL http://Skylion007.github.io/OpenWebTextCorpus, 2019

    work page 2019

  13. [13]

    S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong. Diffuseq: Sequence to sequence text generation with diffusion models. InThe Eleventh International Conference on Learning Representations, 2023

    work page 2023

  14. [14]

    Gulrajani and T

    I. Gulrajani and T. B. Hashimoto. Likelihood-based diffusion language models.Advances in Neural Information Processing Systems, 36:16693–16715, 2023. 10

    work page 2023

  15. [15]

    D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y . Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Ding, H. Gao, H. Qu, H. Li, J. G...

    work page 2025

  16. [16]

    X. Han, S. Kumar, and Y . Tsvetkov. SSD-LM: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 11575–11596, Toronto, Canada, July 202...

    work page 2023

  17. [17]

    Havasi, B

    M. Havasi, B. Karrer, I. Gat, and R. T. Q. Chen. Edit flows: Variable length discrete flow matching with sequence-level edit operations. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    work page 2025

  18. [18]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020

    work page 2020

  19. [19]

    Holtzman, J

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi. The curious case of neural text degeneration. InInternational Conference on Learning Representations, 2020

    work page 2020

  20. [20]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023

    work page 2023

  21. [21]

    W. Kang, K. Galim, S. Oh, M. Lee, Y . Zeng, S. Zhang, C. R. C. Hooper, Y . Hu, H. I. Koo, N. I. Cho, and K. Lee. Parallelbench: Understanding the trade-offs of parallel decoding in diffusion LLMs. InThe Fourteenth International Conference on Learning Representations, 2026

    work page 2026

  22. [22]

    Karimi Mahabadi, H

    R. Karimi Mahabadi, H. Ivison, J. Tae, J. Henderson, I. Beltagy, M. Peters, and A. Cohan. TESS: Text-to-text self-conditioned simplex diffusion. In Y . Graham and M. Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 2347–2361, St. Julian’s, Malta, M...

    work page 2024

  23. [23]

    Karras, M

    T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors,Advances in Neural Information Processing Systems, 2022

    work page 2022

  24. [24]

    Boosting generative image modeling via joint image-feature synthesis.arXiv preprint arXiv:2504.16064, 2025

    T. Kouzelis, E. Karypidis, I. Kakogeorgiou, S. Gidaris, and N. Komodakis. Boosting generative image modeling via joint image-feature synthesis.arXiv preprint arXiv:2504.16064, 2025

    work page arXiv 2025

  25. [25]

    I. Labs, S. Khanna, S. Kharbanda, S. Li, H. Varma, E. Wang, S. Birnbaum, Z. Luo, Y . Miraoui, A. Palrecha, S. Ermon, A. Grover, and V . Kuleshov. Mercury: Ultra-fast language models based on diffusion, 2025. 11

    work page 2025

  26. [26]

    C. Lee, J. Yoo, M. Agarwal, S. Shah, J. Huang, A. Raghunathan, S. Hong, N. M. Boffi, and J. Kim. Flow map language models: One-step language modeling via continuous denoising. arXiv preprint arXiv:2602.16813, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  27. [27]

    X. Li, J. Thickstun, I. Gulrajani, P. S. Liang, and T. B. Hashimoto. Diffusion-lm improves controllable text generation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems, volume 35, pages 4328–4343. Curran Associates, Inc., 2022

    work page 2022

  28. [28]

    Lovelace, V

    J. Lovelace, V . Kishore, C. Wan, E. Shekhtman, and K. Q. Weinberger. Latent diffusion for language generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 56998–57025. Curran Associates, Inc., 2023

    work page 2023

  29. [29]

    Meshchaninov, E

    V . Meshchaninov, E. Chimbulatov, A. Shabalin, A. Abramov, and D. Vetrov. Compressed and smooth latent space for text diffusion modeling. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    work page 2025

  30. [30]

    Meshchaninov, E

    V . Meshchaninov, E. Shibaev, A. Makoian, I. Klimov, D. Sheshenya, A. Malinin, N. Balagansky, D. Gavrilov, A. Alanov, and D. Vetrov. Guided star-shaped masked diffusion.arXiv preprint arXiv:2510.08369, 2025

    work page arXiv 2025

  31. [31]

    Mu and P

    J. Mu and P. Viswanath. All-but-the-top: Simple and effective postprocessing for word repre- sentations. InInternational Conference on Learning Representations, 2018

    work page 2018

  32. [32]

    S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. ZHOU, Y . Lin, J.-R. Wen, and C. Li. Large language diffusion models. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    work page 2025

  33. [33]

    Achiam, S

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V . Balcom, P. Bal- tescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brun...

    work page 2024

  34. [34]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4195–4205, October 2023

    work page 2023

  35. [35]

    Pillutla, S

    K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y . Choi, and Z. Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 4816–4828. Curran Associa...

    work page 2021

  36. [36]

    Discrete Flow Maps

    P. Potaptchik, J. Yim, A. Saravanan, P. Holderrieth, E. Vanden-Eijnden, and M. S. Albergo. Discrete flow maps.arXiv preprint arXiv:2604.09784, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  37. [37]

    Pynadath, J

    P. Pynadath, J. Shi, and R. Zhang. Candi: Hybrid discrete-continuous diffusion models, 2025

    work page 2025

  38. [38]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    work page 2019

  39. [39]

    D. Roos, O. Davis, F. Eijkelboom, M. Bronstein, M. Welling,˙I. ˙I. Ceylan, L. Ambrogioni, and J.-W. van de Meent. Categorical flow maps.arXiv preprint arXiv:2602.12233, 2026

    work page arXiv 2026

  40. [40]

    S. S. Sahoo, M. Arriola, Y . Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V . Kuleshov. Simple and effective masked diffusion language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 130136–130184. Curran Associates, Inc., 2024

    work page 2024

  41. [41]

    S. S. Sahoo, J. Deschenaux, A. Gokaslan, G. Wang, J. T. Chiu, and V . Kuleshov. The diffusion duality. InForty-second International Conference on Machine Learning, 2025

    work page 2025

  42. [42]

    Shabalin, S

    A. Shabalin, S. Elistratov, V . Meshchaninov, I. Sadrtdinov, and D. Vetrov. Why gaussian diffusion models fail on discrete data?, 2026

    work page 2026

  43. [43]

    Shabalin, V

    A. Shabalin, V . Meshchaninov, E. Chimbulatov, V . Lapikov, R. Kim, G. Bartosh, D. Molchanov, S. Markov, and D. Vetrov. Tencdm: Understanding the properties of the diffusion model in the space of language model encodings.Proceedings of the AAAI Conference on Artificial Intelligence, 39(23):25110–25118, Apr. 2025

    work page 2025

  44. [44]

    Shariatian, A

    D. Shariatian, A. Durmus, U. Simsekli, and S. Peluchetti. Latent-augmented discrete diffusion models, 2026

    work page 2026

  45. [45]

    J. Shen, J. Zhao, Z. He, and Z. Lin. Codar: Continuous diffusion language models are more powerful than you think, 2026

    work page 2026

  46. [46]

    J. Shi, K. Han, Z. Wang, A. Doucet, and M. Titsias. Simplified and generalized masked diffusion for discrete data. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  47. [47]

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2011

  48. [48]

    Y . Song, Z. Zhang, C. Luo, P. Gao, F. Xia, H. Luo, Z. Li, Y . Yang, H. Yu, X. Qu, Y . Fu, J. Su, G. Zhang, W. Huang, M. Wang, L. Yan, X. Jia, J. Liu, W.-Y . Ma, Y .-Q. Zhang, Y . Wu, and H. Zhou. Seed diffusion: A large-scale diffusion language model with high-speed inference, 2025

    work page 2025

  49. [49]

    Strudel, C

    R. Strudel, C. Tallec, F. Altché, Y . Du, Y . Ganin, A. Mensch, W. S. Grathwohl, N. Savinov, S. Dieleman, L. Sifre, and R. Leblond. Self-conditioned embedding diffusion for text generation, 2023. 13

    work page 2023

  50. [50]

    Y . Su, T. Lan, Y . Wang, D. Yogatama, L. Kong, and N. Collier. A contrastive framework for neural text generation. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors,Advances in Neural Information Processing Systems, 2022

    work page 2022

  51. [51]

    Score- based generative modeling in latent space

    A. Vahdat, K. Kreis, and J. Kautz. Score-based generative modeling in latent space, 2021.URL https://arxiv. org/abs/2106.05931, 2021

    work page arXiv 2021

  52. [52]

    G. Wang, Y . Schiff, S. S. Sahoo, and V . Kuleshov. Remasking discrete diffusion models with inference-time scaling. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    work page 2025

  53. [53]

    C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P. Luo, S. Han, and E. Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618, 2025

    work page arXiv 2025

  54. [54]

    J. Ye, J. Gao, S. Gong, L. Zheng, X. Jiang, Z. Li, and L. Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning.arXiv preprint arXiv:2410.14157, 2024

    work page arXiv 2024

  55. [55]

    J. Ye, Z. Zheng, Y . Bao, L. Qian, and M. Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises.Transactions of the Association for Computational Linguistics, 2024

    work page 2024

  56. [56]

    H. Yuan, Z. Yuan, C. Tan, F. Huang, and S. Huang. Seqdiffuseq: Text diffusion with encoder- decoder transformers.ArXiv, abs/2212.10325, 2022

    work page arXiv 2022

  57. [57]

    Zhang, J

    Y . Zhang, J. Gu, Z. Wu, S. Zhai, J. M. Susskind, and N. Jaitly. PLANNER: Generating diversified paragraph via latent language diffusion model. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  58. [58]

    Diffusion Transformers with Representation Autoencoders

    B. Zheng, N. Ma, S. Tong, and S. Xie. Diffusion transformers with representation autoencoders. arXiv preprint arXiv:2510.11690, 2025. 14 Appendix A Additional method details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 A.1 ELBO interpretation of the joint objective . . . . . . . . . . . . . . . . . . . . . . 16 A.2 Diffusion-to-enco...

    work page internal anchor Pith review arXiv 2025