Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

Theoretical Benefit and Limitation of Diffusion Language Model

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Masked diffusion language models hit near-optimal perplexity in a constant number of steps, yet worst-case sequence correctness requires a number of steps that grows linearly with sequence length.

desk verdict A genuinely useful positive TER bound for MDMs, but the headline linear-steps SER lower bound is not proved as written due to a false monotonicity claim and an impossible numerical bound. read the letter →

arxiv 2502.09622 v2 pith:4KJPN6ET submitted 2025-02-13 cs.LG cs.AIcs.CLstat.ML

Guhao Feng , Yihan Geng , Jian Guan , Wei Wu , Liwei Wang , Di He This is my paper · ORCID
classification cs.LGcs.AIcs.CLstat.ML
keywords maskeddiffusionmodelsdiscretesamplingstepsperplexitysequenceerrorratehiddenMarkovn-gramlanguageefficiency-accuracytrade-off
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether masked diffusion language models (MDMs) actually deliver the parallel-sampling efficiency they seem to promise, and answers that it depends entirely on what "good" means. For token-level quality, measured by perplexity or token error rate, the paper proves that a well-trained MDM can reach near-optimal TER with a number of sampling steps that is independent of sequence length $L$, so long sequences do not require more model executions. For sequence-level correctness, measured by sequence error rate, the metric that matters for reasoning chains, the paper proves a worst-case lower bound: there is an HMM with a 16-symbol vocabulary for which any MDM satisfying the uniform learning assumption needs at least $N = C L$ steps to push SER below $1/2$. The intended practical conclusion is that MDMs are efficient for fluent generation but lose their efficiency advantage on tasks that require exact reasoning-style outputs.

What carries the argument

The argument runs through the factorized reverse model $p_\theta(\mathbf{x}^0|\mathbf{x}_t) = \prod_i p_\theta(x_i^0|\mathbf{x}_t)$, which samples each position independently at each diffusion step. For the upper bound, the key object is a separator: a run of $n-1$ already-sampled consecutive tokens that splits the sequence into independent intervals under an $n$-gram model; bounding the number of dependencies $\mathrm{DEP}_n(\tau)$ across the reverse process controls the KL divergence between generated and true distributions. For the lower bound, the key object is an interval HMM in which each interval of length $l$ stores pairwise-correlated bits, so that sampling two positions in the same interval simultaneously has probability at least roughly $1/2 - 2\sqrt{2\epsilon_{\mathrm{learning}}}$ of producing an inconsistent sequence; summing this over $L/l$ intervals with $N = C L$ steps forces SER above $1/2$.

What would settle it

Train an MDM on the paper's 16-symbol interval HMM with interval length $l=5$, verify per-state KL errors below $1/128$, and measure SER at $N = C L$ for fixed $C$ and increasing $L$; if SER stays above $1/2$ as $L$ grows, the lower bound holds, while a failure to stay above $1/2$ would refute the theorem's counterexample. Independently, recomputing the probability bound in the proof of Theorem E.9 for simultaneous sampling of two positions in one interval would settle the correctness of the claimed $p_e$ and the constant $C$.

Watch

Extended reading notes

Core claim

The central discovery is a metric-dependent split in the efficiency-accuracy trade-off of MDMs. The positive result, Theorem 4.2, states that for any n-gram language, a reverse model satisfying Assumption 4.1 with an appropriately chosen masking schedule can achieve $\log \mathrm{TER}(p) \le \log \mathrm{TER}(q) + \epsilon_{\mathrm{learning}} + 4\epsilon \log|V|$ using $N = O((n-1)/\epsilon^n)$ steps, a count with no dependence on $L$. The negative result, Theorem 4.4, constructs an HMM over 16 symbols such that for any reverse model with $\epsilon_{\mathrm{learning}} < 1/128$ and any masking schedule, there is a constant $C$ with $\mathrm{SER}(p) > 1/2$ whenever $N = C L$. The proof isolates the mechanism: TER tolerates per-token mistakes as long as marginals are right, while SER demands every token in the sequence be jointly correct, and parallel unmasking of multiple positions in the same dependency interval creates irreducible errors.

Load-bearing premise

The theorems inherit Assumption 4.1, which requires the factorized reverse model's per-token KL error to be below $\epsilon_{\mathrm{learning}}$ at every diffusion time and every partially masked state; if real training only satisfies the bound on average, neither the constant-step TER guarantee nor the linear-step SER barrier is established.

Editorial extensions

If this is right

  • With a well-learned reverse model, MDMs can reach near-optimal perplexity in a number of model executions that stays constant as sequence length grows, unlike autoregressive models that need $L$ executions.
  • For exact sequence correctness, worst-case MDM sampling needs a number of steps proportional to $L$, so MDMs lose their parallel-efficiency advantage over autoregressive models on reasoning-style tasks.
  • Remasking strategies and cached-output samplers do not escape the linear-step lower bound, because the bound depends only on which positions are unmasked together.
  • The practical recommendation flips with the metric: prefer MDMs for fluency-oriented generation and autoregressive models for correctness-oriented generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear-step barrier transfers to real transformers, benchmark gains measured by perplexity alone should not be read as evidence that diffusion LMs are ready for chain-of-thought reasoning, since the same model can look excellent on perplexity while failing exactness.
  • The lower-bound HMM is a deliberately adversarial toy language; a natural stress test is to check whether MDMs trained on code or math corpora show SER degrading with length even when perplexity plateaus, and whether the degradation slope matches the predicted linear threshold.
  • The worst-case result leaves open the possibility that most natural languages are far from the adversarial interval HMM, so an average-case or data-dependent version of Theorem 4.4 could determine whether the linear barrier is common or rare.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper analyzes the sampling-step efficiency of masked diffusion language models (MDMs) under two evaluation metrics. It proves (Theorem 4.2, Appendix D) that for n-gram languages, under a uniform per-token KL learning bound (Assumption 4.1), MDMs can match near-optimal perplexity/TER in a number of steps independent of sequence length, via an explicit masking schedule. It then claims (Theorem 4.4, Appendix E.2) a worst-case negative result: there is an HMM over a vocabulary of size 16 such that, for any sufficiently well-trained factorized reverse model and any masking schedule, SER(p) > 1/2 whenever N = CL; i.e., linear-in-L steps are necessary for low sequence error rate. The proof is based on an interval construction in which simultaneous sampling of two tokens in the same interval causes error with probability bounded below. Synthetic experiments on n-gram and HMM languages are reported as supporting the TER/SER dichotomy.

Significance. If the results held as stated, this would be a valuable conceptual contribution: it would give the first rigorous explanation of why MDMs can appear efficient under perplexity yet lose their parallel-sampling advantage under sequence-level accuracy, with direct implications for reasoning tasks. The positive TER analysis is substantive: it constructs a masking schedule and uses a separator/dependency decomposition, rather than fitting constants. The negative direction is also well chosen: an explicit adversarial HMM counterexample is more informative than a generic lower bound, and the synthetic experiments mirror the claimed dichotomy. However, the central negative theorem is not proved as written: the key final inequality in Theorem E.9 rests on a false monotonicity claim and an impossible numerical lower bound. The construction appears plausible and likely repairable, but the manuscript's central claim is currently blocked. The strength of both theorems is also conditional on Assumption 4.1, a strong uniform-learning assumption whose practical verification is not discussed.

major comments (3)
  1. [Appendix E.2, Theorem E.9] The final step of the proof of Theorem E.9 is mathematically incorrect. After Lemma E.6 the proof has SER(p) ≥ 1 − [(1 − 1/y)^y]^{p_e/(Cl)} with y = CL/p_e, and then asserts that (1 − 1/y)^y is decreasing, concluding SER(p) ≥ p_e/(Cl). For y > 1, the function (1 − 1/y)^y is strictly increasing and tends to e^{−1}; the inequality therefore goes in the opposite direction, so the claimed constant lower bound on SER does not follow from the displayed expressions. Because this is the only step that converts the linear step budget N = CL into a constant SER lower bound, Theorem 4.4 is not established as written.
  2. [Appendix E.2, Theorem E.9 (constants)] The numerical instantiation is internally inconsistent. With l = 5 and ε_learning < 1/128, the proof gives p_e > 1/4. If the constant C is p_e/(l+1), i.e., the value used to obtain C ≥ 1/24, then p_e/(Cl) = (l+1)/l = 6/5 > 1, which is impossible as a lower bound on a probability. The proof also writes C = 2p_e/(l+1) in one place while the computation uses C = p_e/(l+1). This is a second defect in the same calculation, independent of the monotonicity error.
  3. [Section 4, Assumption 4.1] Both main theorems are conditioned on Assumption 4.1, a uniform per-token KL bound that must hold for every time t and every partially masked state x_t. This is a strong learning-quality assumption for real factorized denoising models, and the paper does not discuss any procedure for checking or enforcing it. I do not claim the assumption is circular, but its strength should be stated clearly as a scope condition on the claimed TER/SER dichotomy, especially in the abstract and Section 6.
minor comments (5)
  1. [Appendix D, Lemmas D.8 and D.12] These proofs refer to 'Theorem 4.1' when the cited statement is Assumption 4.1; the main text contains no Theorem 4.1.
  2. [Appendix D, Lemma D.12] In the proof of Lemma D.12, the first-step contribution is displayed as δ_1 log|V|, whereas the preceding derivation gives (δ_1 L − 1) log|V| up to sign conventions; a factor L appears to be dropped. The final bound of 4ε should be re-derived with the corrected first-step term.
  3. [Appendix E.2, Example E.7] The example cites 'Theorem C.3' for Pinsker's inequality; the correct cross-reference is Lemma C.3.
  4. [Section 5.2] The text says 'The experiment results are presented in Section 5' and later 'As Section 5 shows', but the results are in Table 1 (with details in Appendix F); the cross-references should point to the table.
  5. [Section 6] The conclusion contains 'SEDD-unform', a typo for 'SEDD Uniform' (Lou et al., 2024).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the masking schedule is constructed rather than fitted, the adversarial HMM is an explicit counterexample, and the experiments are validation, not inputs to the theory.

full rationale

The derivation chain is self-contained. The positive TER result (Theorem 4.2 / Lemma D.12) constructs an explicit masking schedule, δ1 = ε and δi = δ = ε^n/(C(n−1)), rather than fitting it to the target conclusion, and the final bound is obtained by summing per-step dependency estimates. The negative SER result (Theorem 4.4 / Theorem E.9) is built on an explicit adversarial example: the interval language in Example E.7 and its HMM realization in Note E.8 are constructed distributions, not restatements of the theorem's conclusion. Assumption 4.1 (uniform learning quality) is a stated premise shared by both theorems, and the theorems are conditional on it; no equation assumes the target result as an input. The empirical study in Section 5 uses freshly generated n-gram and HMM datasets and is reported as corroboration, not as a source of fitted parameters appearing in the proofs. I also checked the citation chain: the paper relies on standard prior work for the MDM formalism and training objective, but the load-bearing arguments are derived in the appendix rather than imported from self-citations, and there are no author-overlapping 'uniqueness' citations invoked to force the conclusions. The main caveat I found is a mathematical error in the proof of Theorem E.9, not a circularity: the claim that (1 − 1/y)^y is decreasing is false (it is increasing for y > 1), so the displayed lower bound SER(p) ≥ pe/(Cl) does not follow as written, and the numerical instantiation yields pe/(Cl) = 6/5 > 1, an impossible lower bound on a probability. That is a correctness defect in the proof as written, and it is repairable via 1 − exp(−pe/(Cl)), which gives roughly 0.699 > 1/2 for the stated constants, but it does not make the argument circular. For these reasons the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the learning-quality assumption, on the n-gram/HMM modeling assumption, on the factorized reverse model, and on standard probability inequalities. No data-fitted free parameters are used in the theorems; the schedule constants and adversarial HMM are constructed as part of the proofs, so they are not invented entities in the sense of new physical or mathematical objects.

assumptions (4)
  • domain assumption Assumption 4.1: the trained reverse model p_theta satisfies DKL(q_{0|t}(x_i^0|x_t) || p_theta(x_i^0|x_t)) < epsilon_learning for all t and x_t.
    This is the learning-quality premise. It is reasonable for idealized training but is not proven; both main theorems inherit it.
  • domain assumption The target language can be modeled as an n-gram or HMM, and the reverse model is factorized as in Equation (3).
    The positive theorem is only for n-gram languages; the negative theorem uses an HMM construction. Real language is not exactly n-gram or HMM, so the scope is limited.
  • standard math Standard probabilistic tools: Chernoff bound, Pinsker's inequality, Jensen's inequality, and the chain rule of KL divergence.
    These are used throughout Appendices D and E and are accepted background results.
  • domain assumption In the interval setting (Definition E.4), tokens in different intervals are independent, and simultaneous sampling in the same interval causes error with probability at least p_e.
    This is the structure of the adversarial example in Theorem E.7, converted to HMM form in Note E.8. It is a construction used to establish the lower bound, not a general fact about all distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Theoretical Benefit and Limitation of Diffusion Language Model." pith.science (2026). https://pith.science/paper/4KJPN6ET

@misc{pith2026250209622,
  author       = {Pith},
  title        = {Pith review of: Theoretical Benefit and Limitation of Diffusion Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4KJPN6ET}},
  note         = {Machine review of arXiv:2502.09622}
}
read the original abstract

Diffusion language models have emerged as a promising approach for text generation. One would naturally expect this method to be an efficient replacement for autoregressive models since multiple tokens can be sampled in parallel during each diffusion step. However, its efficiency-accuracy trade-off is not yet well understood. In this paper, we present a rigorous theoretical analysis of a widely used type of diffusion language model, the Masked Diffusion Model (MDM), and find that its effectiveness heavily depends on the target evaluation metric. Under mild conditions, we prove that when using perplexity as the metric, MDMs can achieve near-optimal perplexity in sampling steps regardless of sequence length, demonstrating that efficiency can be achieved without sacrificing performance. However, when using the sequence error rate--which is important for understanding the "correctness" of a sequence, such as a reasoning chain--we show that the required sampling steps must scale linearly with sequence length to obtain "correct" sequences, thereby eliminating MDM's efficiency advantage over autoregressive models. Our analysis establishes the first theoretical foundation for understanding the benefits and limitations of MDMs. All theoretical findings are supported by empirical studies.

Figures

Figures reproduced from arXiv: 2502.09622 by the authors.

Figure 1
Figure 1. Efficiency and accuracy of MDMs on GSM8K (8-shot): The left figure uses Qwen2.5-7B [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Efficiency and accuracy of MDMs on MBPP (3-shot): Similar to the GSM8K experiment, [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the example for computing dependencies in the [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    CreditDecoding accelerates parallel decoding in diffusion LLMs by fusing accumulated Trace Credit with current logits to accept early-correct tokens sooner, yielding up to 5.48x speedup and accuracy gains.

  2. Sentence Curve Language Models

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Predicting a B-spline 'sentence curve' target instead of static word embeddings improves diffusion language models on translation benchmarks.

  3. ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Parallel decoding in diffusion LLMs degrades quality on token-dependent tasks, and existing strategies cannot adapt parallelism to preserve accuracy, as shown by the new 17-task ParallelBench benchmark.

  4. ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute

    cs.CL 2025-08 conditional novelty 6.0 of 10

    ParaThinker trains LLMs for native parallel reasoning and reports 7 to 12 percent higher accuracy on math benchmarks over sequential thinking with modest latency overhead.

  5. dKV-Cache: The Cache for Diffusion Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    dKV-Cache reuses cached key and value states of decoded tokens during diffusion LM denoising, delivering 2-10x faster inference with near-lossless quality on several benchmarks.

  6. Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding

    cs.LG 2026-01 conditional novelty 5.0 of 10

    A training-free inference framework prunes suffix masks, adapts confidence thresholds, and early-exits at EOS to speed up diffusion LLM decoding by up to 68×.

Reference graph

Works this paper leans on

71 extracted references · 28 canonical work pages · cited by 6 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    D., Ho, J., Tarlow, D., and van den Berg, R

    Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, 2021 a

    work page 2021

  4. [4]

    Program synthesis with large language models, 2021 b

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models, 2021 b . URL https://arxiv.org/abs/2108.07732

    arXiv 2021

  5. [5]

    Dirichlet diffusion score model for biological sequence generation

    Avdeyev, P., Shi, C., Tan, Y., Dudnyk, K., and Zhou, J. Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning, pp.\ 1276--1301. PMLR, 2023

    work page 2023

  6. [6]

    Neural machine translation by jointly learning to align and translate

    Bahdanau, D. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    arXiv 2014

  7. [7]

    F., Della Pietra, V

    Brown, P. F., Della Pietra, V. J., Desouza, P. V., Lai, J. C., and Mercer, R. L. Class-based n-gram models of natural language. Computational linguistics, 18 0 (4): 0 467--480, 1992

    work page 1992

  8. [8]

    T., Li, Y., Lundberg, S., et al

    Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023

    arXiv 2023

Show all 71 references
  1. [9]

    D., Rainforth, T., Deligiannidis, G., and Doucet, A

    Campbell, A., Benton, J., Bortoli, V. D., Rainforth, T., Deligiannidis, G., and Doucet, A. A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, 2022

    2022

  2. [10]

    Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design

    Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum...

    2024

  3. [11]

    Analog bits: Generating discrete data using diffusion models with self-conditioning

    Chen, T., Zhang, R., and Hinton, G. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202, 2022

    2022 arXiv

  4. [12]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [13]

    Transformer-xl: Attentive language models beyond a fixed-length context

    Dai, Z. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019

    1901 arXiv

  6. [14]

    I., Bronstein, M., and Bose, A

    Davis, O., Kessler, S., Petrache, M., Ceylan, I. I., Bronstein, M., and Bose, A. J. Fisher flow matching for generative modeling over discrete data. arXiv preprint arXiv:2405.14664, 2024

    2024 arXiv

  7. [15]

    M., Dias Tadeu, T., and Paraboni, I

    De Novais, E. M., Dias Tadeu, T., and Paraboni, I. Improved text generation using n-gram statistics. In Advances in Artificial Intelligence--IBERAMIA 2010: 12th Ibero-American Conference on AI, Bah \' a Blanca, Argentina, November 1-5, 2010. Proceedings 12 , pp.\ 316--325. Spr...

    2010

  8. [16]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technol...

    2019

  9. [17]

    H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., Hawthorne, C., Leblond, R., Grathwohl, W., and Adler, J

    Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., Hawthorne, C., Leblond, R., Grathwohl, W., and Adler, J. Continuous diffusion for categorical data. ArXiv, abs/2211.15089, 2022

    2022 arXiv

  10. [18]

    Eddy, S. R. Hidden markov models. Current opinion in structural biology, 6 0 (3): 0 361--365, 1996

    1996

  11. [19]

    and Chiriatti, M

    Floridi, L. and Chiriatti, M. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30: 0 681--694, 2020

    2020

  12. [20]

    T., Synnaeve, G., Adi, Y., and Lipman, Y

    Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T., Synnaeve, G., Adi, Y., and Lipman, Y. Discrete flow matching. arXiv preprint arXiv:2407.15595, 2024

    2024 arXiv

  13. [21]

    Scaling diffusion language models via adaptation from autoregressive models

    Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J., et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891, 2024

    2024 arXiv

  14. [22]

    and Hashimoto, T

    Gulrajani, I. and Hashimoto, T. Likelihood-based diffusion language models. In Advances in Neural Information Processing Systems, 2023

    2023

  15. [23]

    Diffusionbert: Improving generative masked language models with diffusion models

    He, Z., Sun, T., Wang, K., Huang, X., and Qiu, X. Diffusionbert: Improving generative masked language models with diffusion models. In Annual Meeting of the Association for Computational Linguistics, 2022

    2022

  16. [24]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  17. [25]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020

    2020

  18. [26]

    Can perplexity reflect large language model's ability in long text understanding?, 2024

    Hu, Y., Huang, Q., Tao, M., Zhang, C., and Feng, Y. Can perplexity reflect large language model's ability in long text understanding?, 2024. URL https://arxiv.org/abs/2405.06105

    2024 arXiv

  19. [27]

    On the learning of non-autoregressive transformers

    Huang, F., Tao, T., Zhou, H., Li, L., and Huang, M. On the learning of non-autoregressive transformers. ArXiv, abs/2206.05975, 2022. URL https://api.semanticscholar.org/CorpusID:249626415

    2022 arXiv

  20. [28]

    L., Bahl, L

    Jelinek, F., Mercer, R. L., Bahl, L. R., and Baker, J. K. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62 0 (S1): 0 S63--S63, 1977

    1977

  21. [29]

    Elucidating the design space of diffusion-based generative models

    Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022

    2022

  22. [30]

    Train for the worst, plan for the best: Understanding token ordering in masked diffusions, 2025

    Kim, J., Shah, K., Kontonis, V., Kakade, S., and Chen, S. Train for the worst, plan for the best: Understanding token ordering in masked diffusions, 2025. URL https://arxiv.org/abs/2502.06768

    2025 arXiv

  23. [31]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  24. [32]

    Promises and pitfalls of generative masked language modeling: Theoretical framework and practical guidelines, 2024

    Li, Y., Kirchmeyer, A., Mehta, A., Qin, Y., Dadachev, B., Papineni, K., Kumar, S., and Risteski, A. Promises and pitfalls of generative masked language modeling: Theoretical framework and practical guidelines, 2024. URL https://arxiv.org/abs/2407.21046

    2024 arXiv

  25. [33]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.\ 74--81, 2004

    2004

  26. [34]

    Infini-gram: Scaling unbounded n-gram language models to a trillion tokens

    Liu, J., Min, S., Zettlemoyer, L., Choi, Y., and Hajishirzi, H. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377, 2024

    2024 arXiv

  27. [35]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Lou, A., Meng, C., and Ermon, S. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 32819--32848. PMLR, 2024

    2024

  28. [36]

    Lovelace, J., Kishore, V., Chen, Y., and Weinberger, K. Q. Diffusion guided language modeling. arXiv preprint arXiv:2408.04220, 2024

    2024 arXiv

  29. [37]

    Beyond perplexity: Examining temporal generalization in large language models via definition generation

    Luden, I., Giulianelli, M., and Fern \'a ndez, R. Beyond perplexity: Examining temporal generalization in large language models via definition generation. Computational Linguistics in the Netherlands Journal, 13: 0 205--232, 2024

    2024

  30. [38]

    and Bunke, H

    Marti, U.-V. and Bunke, H. Using a statistical language model to improve the performance of an hmm-based cursive handwriting recognition system. International journal of Pattern Recognition and Artificial intelligence, 15 0 (01): 0 65--90, 2001

    2001

  31. [39]

    Concrete score matching: Generalized score matching for discrete data

    Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data. In Advances in Neural Information Processing Systems, 2022

    2022

  32. [40]

    Scaling up masked diffusion models on text

    Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M., and Li, C. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024

    2024 arXiv

  33. [41]

    Large language diffusion models

    Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025

    2025 arXiv

  34. [42]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data

    Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736, 2024

    2024 arXiv

  35. [43]

    M., Harman, M., and Wang, M

    Ouyang, S., Zhang, J. M., Harman, M., and Wang, M. Llm is like a box of chocolates: the non-determinism of chatgpt in code generation. arXiv preprint arXiv:2308.02828, 2023

    2023 arXiv

  36. [44]

    Bleu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

    2002

  37. [45]

    R., and Kuleshov, V

    Rastogi, R., Schiff, Y., Hacohen, A., Li, Z., Lee, I., Deng, Y., Sabuncu, M. R., and Kuleshov, V. Semi-parametric inducing point networks and neural processes. arXiv preprint arXiv:2205.11718, 2022

    2022 arXiv

  38. [46]

    Code llama: Open foundation models for code

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Ellen, X., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  39. [47]

    S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J

    Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models, 2024

    2024

  40. [48]

    Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024

    Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2024

    2024

  41. [49]

    Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406.04329, 2024

    2024 arXiv

  42. [50]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, 2015

    2015

  43. [51]

    Denoising diffusion implicit models

    Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021 a . URL https://openreview.net/forum?id=St1giarCHLP

    2021

  44. [52]

    P., Kumar, A., Ermon, S., and Poole, B

    Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021 b . URL https://openreview.net/forum?id=PxTIG12RRHS

    2021

  45. [53]

    and Yang, Y

    Sun, Z. and Yang, Y. Difusco: Graph-based diffusion solvers for combinatorial optimization. Advances in Neural Information Processing Systems, 36: 0 3706--3731, 2023

    2023

  46. [54]

    Sequence to sequence learning with neural networks

    Sutskever, I. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215, 2014

    2014 arXiv

  47. [55]

    Qwen2.5: A party of foundation models, September 2024

    Team, Q. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  48. [56]

    Digress: Discrete denoising diffusion for graph generation

    Vignac, C., Krawczuk, I., Siraudin, A., Wang, B., Cevher, V., and Frossard, P. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734, 2022

    2022 arXiv

  49. [57]

    S., and Kuleshov, V

    Wang, G., Schiff, Y., Sahoo, S. S., and Kuleshov, V. Remasking discrete diffusion models with inference-time scaling, 2025. URL https://arxiv.org/abs/2503.00307

    2025

  50. [58]

    Emergent abilities of large language models

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022 a

    2022

  51. [59]

    V., and Zhou, D

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Info...

    2022

  52. [60]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022 c

    2022

  53. [61]

    V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al

    Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016

    2016 arXiv

  54. [62]

    Energy-based diffusion language models for text generation

    Xu, M., Geffner, T., Kreis, K., Nie, W., Xu, Y., Leskovec, J., Ermon, S., and Vahdat, A. Energy-based diffusion language models for text generation. arXiv preprint arXiv:2410.21357, 2024

    2024 arXiv

  55. [63]

    Diffusion language models can perform many tasks with scaling and instruction-finetuning

    Ye, J., Zheng, Z., Bao, Y., Qian, L., and Gu, Q. Diffusion language models can perform many tasks with scaling and instruction-finetuning. arXiv preprint arXiv:2308.12219, 2023

    2023 arXiv

  56. [64]

    Dream 7b, 2025

    Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b, 2025. URL https://hkunlp.github.io/blog/2025/dream

    2025

  57. [65]

    Thought propagation: An analogical approach to complex reasoning with large language models, 2024

    Yu, J., He, R., and Ying, R. Thought propagation: An analogical approach to complex reasoning with large language models, 2024. URL https://arxiv.org/abs/2310.03965

    2024 arXiv

  58. [66]

    Tractable control for autoregressive language generation

    Zhang, H., Dang, M., Peng, N., and Van den Broeck, G. Tractable control for autoregressive language generation. In International Conference on Machine Learning, pp.\ 40932--40945. PMLR, 2023

    2023

  59. [67]

    Language rectified flow: Advancing diffusion language generation with probabilistic flows

    Zhang, S., Wu, L., Gong, C., and Liu, X. Language rectified flow: Advancing diffusion language generation with probabilistic flows. arXiv preprint arXiv:2403.16995, 2024

    2024 arXiv

  60. [68]

    Improving and unifying discrete&continuous-time discrete denoising diffusion

    Zhao, L., Ding, X., Yu, L., and Akoglu, L. Improving and unifying discrete&continuous-time discrete denoising diffusion. arXiv preprint arXiv:2402.03701, 2024

    2024 arXiv

  61. [69]

    Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling

    Zheng, K., Chen, Y., Mao, H., Liu, M.-Y., Zhu, J., and Zhang, Q. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908, 2024

    2024 arXiv

  62. [70]

    A reparameterized discrete diffusion model for text generation

    Zheng, L., Yuan, J., Yu, L., and Kong, L. A reparameterized discrete diffusion model for text generation. ArXiv, abs/2302.05737, 2023

    2023 arXiv

  63. [71]

    Deductive beam search: Decoding deducible rationale for chain-of-thought reasoning, 2024

    Zhu, T., Zhang, K., Xie, J., and Su, Y. Deductive beam search: Decoding deducible rationale for chain-of-thought reasoning, 2024. URL https://arxiv.org/abs/2401.17686

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.