Pith

open record

sign in

arxiv: 2607.05381 · v1 · pith:NJCUCNBD · submitted 2026-07-06 · cs.LG · cs.AI· cs.CL· cs.IT· math.IT· stat.ML

What Does a Discrete Diffusion Model Learn?

Rodrigo Casado Noguerales , Bernhard Sch\"olkopf , Thomas Hofmann , Aran Raoufi This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-07-07 13:36 UTCglm-5.2pith:NJCUCNBDrecord.jsonopen to challenge →

Figure 1
Figure 1. Figure 1: The model rates Qbθ t (zt , y) from a trained uniform-diffusion denoiser neural network, sampled across many (zt , y, t), average to the oracle rate Qbt(zt , y) = Eq(z0|zt) h Qbt(zt , y | z0) i . 8 [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] reproduced from arXiv: 2607.05381
classification cs.LG cs.AIcs.CLcs.ITmath.ITstat.ML
keywords elboprocessdenoiserdiffusionbridgedataexactlymodel
0
0 comments X

The pith

Discrete diffusion ELBO is an exact path distance, not a bound

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proves that the negative ELBO of a discrete diffusion model is not merely a loose likelihood bound but an exact identity: it equals the data entropy plus the KL divergence between the entire trajectory of the true reverse process (the oracle) and the learned one. The unique optimal model is the marginal reverse jump rate, the posterior average of the clean-conditioned rate given only the noisy state. The irreducible cost of training, shared by every noising process, is exactly the rate at which the forward process destroys information about the clean data, which integrates to the data entropy. For sequence models, the paper shows that the denoiser, cavity (bridge plug-in), and concrete score parameterizations are three exact coordinate representations of this single optimal reverse rate, with closed-form conversions, and explains why a denoiser parameterization diverges at initialization under uniform diffusion while the cavity stays finite.

Core claim

The central object is the Oracle Distance theorem: for any discrete diffusion noising process and any learned reverse rates, the data-averaged negative ELBO minus the data entropy equals a reconstruction KL at the lower endpoint plus the path-space KL from the oracle reverse process to the learned one. The path KL decomposes into a terminal-prior KL plus an integral of local rate divergences between the true marginal reverse rate and the model rate. This is an exact equality, not an inequality. Its unique optimizer is the marginal reverse rate, obtained by projecting the clean-conditioned bridge rate onto the information available at the noisy state, and its irreducible per-time cost is d/dt

What carries the argument

The local rate divergence Phi(a,b) = a log(a/b) - a + b, a Bregman divergence of the negative-entropy potential, serves as the per-jump KL between two CTMC intensities. A Bregman-Pythagoras identity splits any model's cost into the oracle cost plus model mismatch with no cross term. For product noising processes, the reverse rate factors into per-token jumps expressible in three coordinates: the denoiser (clean-token posterior given the full noisy state), the cavity law (clean-token posterior given the noisy context but not the local noisy token), and the concrete score (ratio of noisy conditional laws), with closed-form conversions via local Bayes inversion of the forward kernel.

If this is right

  • ELBO values reported across different noising processes (masked, uniform, GIDD) become directly comparable once boundary terms are retained, since they all share the same entropy floor and the excess measures path divergence to the oracle.
  • A uniform cavity head initialized to 1/V yields per-token NELBO of log V regardless of process or training window, providing an exact calibration tool for debugging ELBO implementations.
  • Using a denoiser parameterization for uniform or GIDD diffusion causes the ELBO to diverge at initialization, scaling as (V-1)/V * log(1/beta), while the cavity parameterization remains finite, explaining a known training instability.
  • The framework cleanly separates reverse-rate error (what the ELBO measures) from sampling factorization error (invisible to the ELBO), showing that even an oracle model with zero rate error degrades at few-step sampling due to the forced product approximation over positions.
  • The information-uniform clock, defined by the cumulative conditional entropy, drains information linearly and straightens process-dependent cost curves onto a single diagonal, offering a principled alternative to variance-optimal importance sampling for time discretization.

Load-bearing premise

The learned reverse process must assign positive jump rate to every transition the true reverse process uses with positive probability. If the model ever assigns zero rate to a jump the oracle needs, the path-KL identity breaks because the relevant density ratio becomes undefined. This holds for standard parameterizations of masked, uniform, and GIDD diffusion but is an active constraint on the model class: a neural network that collapses its output distribution too narrowly,

What would settle it

If the Oracle Distance identity holds, then for any noising process and any model rate, the data-averaged NELBO minus the data entropy must exactly equal the sum of the reconstruction KL, the terminal-prior KL, and the integrated local rate divergence evaluated at marginal (not clean-conditioned) rates. A single numerical counterexample on a finite-state model would refute it. Additionally, the Pythagorean split predicts that training a denoiser head and reading it as a cavity head without conversion yields a strictly suboptimal reverse process whose generative perplexity exceeds the oracle, a

Figures

Figures reproduced from arXiv: 2607.05381 by Aran Raoufi, Bernhard Sch\"olkopf, Rodrigo Casado Noguerales, Thomas Hofmann.

Figure 2
Figure 2. Figure 2: Every noising process destroys exactly H(q0) information, at its own rate, computed at the exact oracle on the GIDD family (§8) that interpolates masked and uniform diffusion through the parameter λ. (a) Information destroyed by time t, H(Z0 | Zt)/L, follows different paths to the common ceiling H(q0)/L (Theorem 4, §5.1). (b) Its rate J ⋆ t = d dtH(Z0 | Zt): uniform destroys information earlier than masked… view at source ↗
Figure 3
Figure 3. Figure 3: The NELBO is an exact distance to the oracle (Corollary [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The reverse rate in three coordinates: each vertex lists a coordinate of the reverse rate, [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Reading the exact reverse rate held in coordinate [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Sampling with the exact oracle denoiser and the factorized ancestral samplers of §7 zeroes the reverse-rate error, so the excess of the generative perplexity is pure sampling error, which even the oracle rates incur. It vanishes only as the sampling budget grows, masked being penalized more due to its inability to self-correct. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) A cavity head µ θ i ≡ 1/V has per-token NELBO log V once boundary terms are kept (Proposition 4). (b) The same head read as a denoiser makes the uniform NELBO diverge, linearly in log(1/βt1 ) with slope V −1 V (Proposition 6), while masked stays finite. (c) The oracle NELBO splits as reconstruction + path + terminal prior, summing to the same H(q0) for any noising process (Corollary 3; the window [0.1,… view at source ↗
read the original abstract

What does a discrete diffusion model learn: a denoiser, a score ratio, or a bridge plug-in predictor? At the level of jump rates, these are one object in different coordinates, and reading a neural network in the wrong coordinate changes the process being trained and sampled. Starting with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process, boundary terms included, we prove the \emph{Oracle Distance} theorem: the negative ELBO is exactly equal to the data entropy plus the path KL from the oracle reverse process to the learned one, not merely a bound. Its unique optimizer is therefore the conditional expectation of the true reverse jump rate given the current noisy state, and its irreducible cost is the rate at which the forward process $Z_t$ destroys information about the clean data $Z_0$, $-\tfrac{d}{dt}I(Z_0; Z_t)$, so every noising process shares the same best achievable negative ELBO: the data entropy. For sequences with token-factorizing noise, the oracle projection yields three exact coordinates for the optimizer: denoiser, cavity (bridge plug-in), and score, with closed-form conversions among them. This framework identifies which law each loss in the literature actually optimizes, recovering MDM, UDM, SEDD, and GIDD as special cases; explains why denoiser and cavity coincide for masked diffusion but not for uniform diffusion; proves that a denoiser parameterization makes the uniform ELBO diverge at initialization while the bridge plug-in stays finite; and calibrates ELBO implementations exactly at initialization. Every identity is verified numerically, without approximation, on an exactly solvable model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 9 minor

Summary. This paper provides a rigorous, self-contained theoretical account of continuous-time discrete diffusion models. The central result is the Oracle Distance theorem (Theorem 3), which establishes that the negative ELBO equals the data entropy plus an exact path-law KL divergence from the oracle reverse process to the learned one—not merely a bound. The paper identifies the population optimizer as the marginal reverse jump rate (the posterior average of the clean-conditioned rate), shows the irreducible oracle cost equals the information-destruction rate d/dt H(Z₀|Z_t), and proves a universal NELBO floor at the data entropy. For token-factorizable processes, the paper derives three exact coordinate representations of the optimal reverse rate (denoiser, cavity, score) with closed-form conversions, recovers MDM/UDM/SEDD/GIDD as special cases, and proves that a denoiser parameterization makes the uniform-diffusion ELBO diverge at initialization while the cavity parameterization stays finite. Two independent proof routes are provided for both the CTMC ELBO (infinitesimal KL and Girsanov) and the Oracle Distance theorem (Pythagorean assembly and direct path-law factorization). All identities are verified numerically on an exactly solvable toy model.

Significance. The paper makes several strong contributions. It ships parameter-free derivations from standard tools (Girsanov's formula, KL chain rule, Bregman divergence identities, Kolmogorov forward equations) with no fitted constants. The Oracle Distance identity is an exact result, not a bound, and is proved twice via independent routes. The information-loss identity J*_t = d/dt H(Z₀|Z_t) connects the ELBO to the I-MMSE relation in the CTMC setting. The three-coordinate dictionary (denoiser/cavity/score) with exact conversion formulas is a practically useful unification that cleanly separates what each literature loss optimizes. The divergence result for the uniform-diffusion denoiser parameterization (Proposition 6) gives a theoretical account of a known empirical failure mode. The numerical verification, while on a toy model, is exact and tests every identity without approximation. The concurrent work of Gourevitch et al. [GJS+26] independently discovered the denoiser/cavity distinction for uniform diffusion; this paper generalizes that insight to all token-factorizable processes from a single projection principle, which is a meaningful extension.

major comments (3)
  1. The paper is theoretically sound. I examined both proof routes for the Oracle Distance theorem (§5.1 assembly via Proposition 2 + Theorem 4, and §5.5 direct path-law factorization), the Girsanov formula (Theorem 2), the Bregman-Pythagorean identity (Lemma 2), the reverse-rate projection (Theorem 5), and the information-loss identity (Theorem 4). The key algebraic steps are correct: the Pythagorean decomposition follows from the standard Bregman conditional-mean identity, the information-loss proof correctly uses the Kolmogorov forward equation and the generator's zero-row-sum property to insert the vanishing term in Eq. (34), and the alternative proof in §5.5 correctly applies the KL chain rule using the Markov property factorization (38). Assumption 2 (rate-support) is load-bearing for the Girsanov formula but reduces to support inclusion on finite state spaces and is automatically满足ed由
  2. The paper is theoretically sound. I examined both proof routes for the Oracle Distance theorem (§5.1 assembly via Proposition 2 + Theorem 4, and §5.5 direct path-law factorization), the Girsanov formula (Theorem 2), the Bregman-Pythagorean identity (Lemma 2), the reverse-rate projection (Theorem 5), and the information-loss identity (Theorem 4). The key algebraic steps are correct: the Pythagorean decomposition follows from the standard Bregman conditional-mean identity, the information-loss proof correctly uses the Kolmogorov forward equation and the generator's zero-row-sum property, and the alternative proof in §5.5 correctly applies the KL chain rule. Assumption 2 (rate-support) is load-bearing for Girsanov but reduces to support inclusion on finite spaces and is automatically satisfied for standard softmax/cavity parameterizations. Assumption 3 (fixed support, piecewise continuity)
  3. The paper is theoretically sound. I examined both proof routes for the Oracle Distance theorem, the Girsanov formula, the Bregman-Pythagorean identity, the reverse-rate projection, and the information-loss identity. The key algebraic steps are correct. Assumption 2 (rate-support) is load-bearing for Girsanov but reduces to support inclusion on finite spaces and is automatically satisfied for standard parameterizations. Assumption 3 is only needed for Theorem 4, not for Theorem 3 itself (as the §5.5 alternative proof shows). I do not identify any load-bearing error that would undermine the central claims.
minor comments (9)
  1. §4.2, Definition 1: The footnote defining the Skorokhod space D([t₁,t₂],X) is helpful, but the σ-field is described as 'generated by the coordinate maps' without specifying whether this is the raw or predictable σ-field. For finite state spaces this distinction is immaterial, but a brief clarification would improve rigor.
  2. §6.2, Remark 3: The convention 0/0 := 0 for terms where both numerator and denominator vanish is introduced here but used implicitly earlier (e.g., in the denoiser rate formula (52)). Stating this convention at first use (or in §4.2) would prevent confusion.
  3. §7.1, Proposition 5: The Itakura–Saito divergence D_IS(p∥q) = p/q - log(p/q) - 1 is defined inline but its non-negativity is not explicitly stated. A one-line note that D_IS ≥ 0 (by log x ≤ x - 1) would help readers verify that the cavity integrand (62) is non-negative.
  4. Table 5: The '1/V denoiser NELBO' entry for GIDD references Remark 4 rather than a numbered proposition. Consider elevating Remark 4 to a corollary of Proposition 6 for formal parity, since the divergence result is load-bearing for the practical recommendation to avoid denoiser parameterizations.
  5. §8: The toy model uses V=8, L=8 with a bigram chain. While appropriate for exact verification, a brief note on why this model is sufficient to test all identities (e.g., it has non-trivial position correlations so denoiser ≠ cavity for uniform/GIDD) would help readers appreciate the design choice.
  6. Figure 5: The panel labels (a)–(f) are referenced in the text but the figure caption could more explicitly state which theoretical result each panel verifies (e.g., 'Panel (b): Theorem 7 convert-or-pay penalty').
  7. §3 (Related Work): The paper's relationship to Generator Matching [HHY+25] is discussed at the objective level but the distinction could be sharper. Specifically, a sentence noting that Generator Matching's Propositions 1–2 already contain the conditional-mean optimality and Bregman gradient-sharing at the matching-objective level, while this paper's contribution is the likelihood-side identity (ELBO = entropy + path KL), would make the novelty boundary clearer.
  8. Several references are dated 2026 (e.g., [GJS+26], [SLY+26], [RNB+26], [ST25]). If these are accepted/published papers, please verify the citation metadata; if they are preprints, consider noting 'preprint' or 'to appear' consistently.
  9. Notation: The paper uses both Q̂_t (with hat) and bQ_t (with backslash-b) for reverse rates. While defined in §4.2, the two notations appear in different sections and a unified choice would improve readability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

The referee report is a positive review recommending minor revision. The three 'major comments' appear to be repeated instances of the same assessment: the paper is theoretically sound, the proofs are correct, and no load-bearing errors undermine the central claims. The referee makes two specific observations about assumptions that we can address with minor clarifications.

read point-by-point responses

  1. Referee: The paper is theoretically sound. Both proof routes for the Oracle Distance theorem, the Girsanov formula, the Bregman-Pythagorean identity, the reverse-rate projection, and the information-loss identity were examined and found correct.

    Authors: We thank the referee for the careful and thorough verification of our proofs. We are gratified that the referee independently checked both routes for the Oracle Distance theorem (§5.1 assembly and §5.5 direct factorization), the Girsanov formula (Theorem 2), the Bregman-Pythagorean identity (Lemma 2), the projection result (Theorem 5), and the information-loss identity (Theorem 4), including the key algebraic steps such as the Kolmogorov forward equation insertion in Eq. (34) and the KL chain rule factorization in Eq. (38). revision: no

  2. Referee: Assumption 2 (rate-support) is load-bearing for the Girsanov formula but reduces to support inclusion on finite state spaces and is automatically satisfied for standard softmax/cavity parameterizations.

    Authors: We agree with the referee's characterization. Assumption 2 is indeed the absolute-continuity condition required for the Girsanov change-of-measure (Theorem 2) and hence for the CTMC ELBO (Theorem 1). On finite state spaces it is simply support inclusion: supp bQt(zt,·|z0) ⊆ supp bQθ_t(zt,·). For the standard parameterizations studied in §§6–7, this holds automatically: the cavity rate (55) never divides by per-token likelihood and is confined to the forward support Si by construction, while the score rate (57) inherits support from the score head. The denoiser rate (52) requires the additional convention that 0/0 := 0 and that supp πθ_i ⊆ Si(zi_t, t), which we discuss in Remark 3. We will add a brief sentence after Assumption 2 explicitly noting that it reduces to support inclusion on finite spaces and is automatically satisfied for the cavity and score parameterizations, and that the denoiser parameterization requires the support restriction of Remark 3. revision: yes

  3. Referee: Assumption 3 (fixed support, piecewise continuity) is only needed for Theorem 4, not for Theorem 3 itself, as the §5.5 alternative proof shows.

    Authors: The referee is exactly right. Assumption 3 (regularity and fixed support) is used only in the proof of Theorem 4 (the information-loss identity J*_t = d/dt H(Z0|Zt)), where it ensures that the conditional entropy H(Z0|Zt) is absolutely continuous and that the differentiation under the sum, the Kolmogorov forward equation substitution, and the zero-row-sum insertion in Eq. (34) are all valid. The Oracle Distance theorem (Theorem 3) itself does not require Assumption 3: the §5.5 alternative proof uses only the reverse-time Markov factorization (38), the KL chain rule, and Girsanov's formula (Theorem 2), none of which needs fixed support or piecewise continuity beyond boundedness and measurability. We already state this in §5.1 ('The regularity assumption is needed only for this second ingredient, not for Theorem 3 itself; §5.5 gives an alternative proof of the latter that avoids it'), but we will make the point more prominent by adding a remark after Theorem 3 explicitly cross-referencing the §5.5 proof's weaker hypotheses. revision: yes

Circularity Check

0 steps flagged

No significant circularity found. The Oracle Distance theorem and supporting lemmas are parameter-free derivations from standard tools (Girsanov, KL chain rule, Bregman identities, Kolmogorov equations).

full rationale

The paper's central results are derived from standard, externally verifiable mathematical tools, not from self-citation chains or fitted parameters. Theorem 2 (Girsanov formula for finite-state CTMCs) is proved from the trajectory likelihood (Eq. 12) and the jump compensator identity (Eq. 13), citing textbook sources [Nor97, Bre81, JS03]. Theorem 3 (Oracle Distance) is proved twice: once via the Pythagorean decomposition (Proposition 2, using the Bregman identity of Lemma 2, citing [BGW05]) and the information-loss identity (Theorem 4, proved by differentiating H(Z_0|Z_t) via the Kolmogorov forward equation), and once via the KL chain rule on path laws (§5.5). No constants are fitted to data. The optimizer (Theorem 5) is the conditional mean E[bQ_t(Z_t,y|Z_0)|Z_t=z_t], shown to equal the marginal reverse rate bQ_t(z_t,y) by a direct Bayes computation. The information-loss identity J*_t = d/dt H(Z_0|Z_t) is proved by matching the algebraic expansion of the oracle cost (Eq. 33) against the derivative of conditional entropy (Eq. 34), using the Kolmogorov forward equation. The coordinate dictionary (Theorem 7) follows from substituting the product-CTMC structure into the projection formula. Self-citations ([CBDB+22] for the CTMC ELBO framework, [vRFD+25] for GIDD) are contextual, not load-bearing for the proofs. The numerical verification (§8) uses an exactly solvable toy model with closed-form oracle quantities, not fitted parameters. The only minor concern is that the paper builds on [CBDB+22] for the CTMC ELBO formulation, but it re-derives this from scratch (Theorem 1) with two independent proofs, so the self-citation is not load-bearing. Score 1 reflects this minor, non-circular self-citation.

Axiom & Free-Parameter Ledger

0 free parameters · 7 axioms · 0 invented entities

No invented entities. The framework uses standard CTMC objects (generators, path laws, reverse rates) and standard information-theoretic quantities (entropy, mutual information, KL divergence).

axioms (7)
  • domain assumption Rate-support condition: bQ_t(z_t,·|z_0) ≪ bQ^θ_t(z_t,·) for dt dq(z_t,z_0)-a.e. (t,z_t,z_0)
    Assumption 2, §4.3. Required for the Girsanov formula (Theorem 2) and hence for the entire ELBO identity. Equivalent to support inclusion: the learned process can jump wherever the true reverse process can.
  • domain assumption Bounded, measurable reverse rates on compact windows [t1,t2] ⊂ (0,T)
    Theorem 1, §4.3. Required for well-posedness of the CTMC path laws and finiteness of the Girsanov integral.
  • domain assumption Terminal mixing: q_{t|0}(·|z_0) → q_T as t↑T, independent of z_0
    Assumption 1, §4.2. Needed for the practical interpretation of diffusion (sampling from q_T without data access) and for the universal floor corollary at t2=T.
  • domain assumption Regularity and fixed support: piecewise continuous bounded rates with piecewise supports of q_t and q_{t|0}(·|z_0) independent of t on [t1,t2]
    Assumption 3, §5.1. Needed for the information-loss identity (Theorem 4) in the assembly proof. The alternative proof in §5.5 avoids this assumption.
  • domain assumption Token-factorizable noising: forward kernel factors as product of per-position kernels
    Theorem 6, §6.1. Required for the three-coordinate dictionary (Theorem 7) and all sequence-modeling results. Standard in the literature but a structural restriction.
  • standard math Girsanov formula for finite-state CTMCs (path-law relative entropy = initial KL + integral of rate divergences)
    Theorem 2, §4.3. Standard result from jump process theory [Brémaud 1981, Jacod-Shiryaev 2003]. Proved in the paper for completeness.
  • standard math Bregman-Pythagoras identity: E[Φ(A,B)] = E[Φ(A,B*)] + E[Φ(B*,B)] with B* = E[A|G]
    Lemma 2, §5.2. Standard result from Banerjee-Guo-Wang 2005. Specialized to F(u) = u log u - u.

pith-pipeline@v1.1.0-glm · 60122 in / 4522 out tokens · 112234 ms · 2026-07-07T13:36:40.445373+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages · 3 internal anchors

  1. [1]

    Advances in Neural Information Processing Systems , volume =

    Simplified and Generalized Masked Diffusion for Discrete Data , author =. Advances in Neural Information Processing Systems , volume =. 2024 , url =

    work page 2024

  2. [2]

    Advances in Neural Information Processing Systems , volume =

    Variational Diffusion Models , author =. Advances in Neural Information Processing Systems , volume =. 2021 , url =

    work page 2021

  3. [3]

    and Welling, Max , booktitle =

    Kingma, Diederik P. and Welling, Max , booktitle =. Auto-Encoding Variational. 2014 , url =

    work page 2014

  4. [4]

    Proceedings of the 31st International Conference on Machine Learning (ICML) , series =

    Stochastic Backpropagation and Approximate Inference in Deep Generative Models , author =. Proceedings of the 31st International Conference on Machine Learning (ICML) , series =. 2014 , publisher =

    work page 2014

  5. [5]

    Advances in Neural Information Processing Systems , volume =

    A Continuous Time Framework for Discrete Denoising Models , author =. Advances in Neural Information Processing Systems , volume =. 2022 , url =

    work page 2022

  6. [6]

    Proceedings of the 42nd International Conference on Machine Learning (ICML) , series =

    Generalized Interpolating Discrete Diffusion , author =. Proceedings of the 42nd International Conference on Machine Learning (ICML) , series =. 2025 , publisher =

    work page 2025

  7. [7]

    2026 , eprint =

    Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation , author =. 2026 , eprint =

    work page 2026

  8. [8]

    International Conference on Learning Representations (ICLR) , year =

    Score-based Continuous-time Discrete Diffusion Models , author =. International Conference on Learning Representations (ICLR) , year =

    work page

  9. [9]

    Proceedings of the 41st International Conference on Machine Learning (ICML) , series =

    Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution , author =. Proceedings of the 41st International Conference on Machine Learning (ICML) , series =. 2024 , publisher =

    work page 2024

  10. [10]

    Advances in Neural Information Processing Systems , volume =

    Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions , author =. Advances in Neural Information Processing Systems , volume =. 2021 , url =

    work page 2021

  11. [11]

    Advances in Neural Information Processing Systems , volume =

    Structured Denoising Diffusion Models in Discrete State-Spaces , author =. Advances in Neural Information Processing Systems , volume =. 2021 , url =

    work page 2021

  12. [12]

    Advances in Neural Information Processing Systems , volume =

    Simple and Effective Masked Diffusion Language Models , author =. Advances in Neural Information Processing Systems , volume =. 2024 , url =

    work page 2024

  13. [13]

    Advances in Neural Information Processing Systems , volume =

    Concrete Score Matching: Generalized Score Matching for Discrete Data , author =. Advances in Neural Information Processing Systems , volume =. 2022 , url =

    work page 2022

  14. [14]

    Advances in Neural Information Processing Systems , volume =

    Discrete Flow Matching , author =. Advances in Neural Information Processing Systems , volume =. 2024 , url =

    work page 2024

  15. [15]

    International Conference on Machine Learning (ICML) , year =

    Scaling Beyond Masked Diffusion Language Models , author =. International Conference on Machine Learning (ICML) , year =. 2602.15014 , archiveprefix =

    work page arXiv

  16. [16]

    Proceedings of the 42nd International Conference on Machine Learning (ICML) , series =

    The Diffusion Duality , author =. Proceedings of the 42nd International Conference on Machine Learning (ICML) , series =. 2025 , publisher =

    work page 2025

  17. [17]

    Holderrieth, Peter and Havasi, Marton and Yim, Jason and Shaul, Neta and Gat, Itai and Jaakkola, Tommi and Karrer, Brian and Chen, Ricky T. Q. and Lipman, Yaron , booktitle =. Generator Matching: Generative Modeling with Arbitrary. 2025 , note =

    work page 2025

  18. [18]

    International Conference on Learning Representations (ICLR) , year =

    Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data , author =. International Conference on Learning Representations (ICLR) , year =

    work page

  19. [19]

    Proceedings of the 42nd International Conference on Machine Learning (ICML) , series =

    Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion , author =. Proceedings of the 42nd International Conference on Machine Learning (ICML) , series =. 2025 , publisher =

    work page 2025

  20. [20]

    On the Optimality of Conditional Expectation as a

    Banerjee, Arindam and Guo, Xin and Wang, Hui , journal =. On the Optimality of Conditional Expectation as a. 2005 , doi =

    work page 2005

  21. [21]

    2025 , eprint =

    Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds , author =. 2025 , eprint =

    work page 2025

  22. [22]

    Proceedings of the 32nd International Conference on Machine Learning (ICML) , series =

    Deep Unsupervised Learning using Nonequilibrium Thermodynamics , author =. Proceedings of the 32nd International Conference on Machine Learning (ICML) , series =. 2015 , publisher =

    work page 2015

  23. [23]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Denoising Diffusion Probabilistic Models , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2020 , url =

    work page 2020

  24. [24]

    Norris, J. R. , series =. 1997 , doi =

    work page 1997

  25. [25]

    2003 , doi =

    Limit Theorems for Stochastic Processes , author =. 2003 , doi =

    work page 2003

  26. [26]

    1981 , isbn =

    Point Processes and Queues: Martingale Dynamics , author =. 1981 , isbn =

    work page 1981

  27. [27]

    2011 , doi =

    Entropy and Information Theory , author =. 2011 , doi =

    work page 2011

  28. [28]

    Proceedings of the 38th International Conference on Machine Learning , series =

    Improved Denoising Diffusion Probabilistic Models , author =. Proceedings of the 38th International Conference on Machine Learning , series =. 2021 , publisher =

    work page 2021

  29. [29]

    Noise Scheduling as Information-Guided Allocation in Diffusion Training

    Noise Scheduling as Information-Guided Allocation in Diffusion Training , author =. 2026 , eprint =. doi:10.48550/arXiv.2602.18647 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2602.18647 2026

  30. [30]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Information-Theoretic Discrete Diffusion , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =. 2510.24088 , archiveprefix =

    work page arXiv

  31. [31]

    Mutual Information, Relative Entropy, and Estimation in the

    Atar, Rami and Weissman, Tsachy , journal =. Mutual Information, Relative Entropy, and Estimation in the. 2012 , doi =

    work page 2012

  32. [32]

    International Conference on Learning Representations (ICLR) , year =

    Information-Theoretic Diffusion , author =. International Conference on Learning Representations (ICLR) , year =

    work page

  33. [33]

    Mutual Information and Minimum Mean-Square Error in

    Guo, Dongning and Shamai, Shlomo and Verd. Mutual Information and Minimum Mean-Square Error in. IEEE Transactions on Information Theory , volume =. 2005 , doi =

    work page 2005

  34. [34]

    Proceedings of the 41st International Conference on Machine Learning (ICML) , series =

    Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design , author =. Proceedings of the 41st International Conference on Machine Learning (ICML) , series =. 2024 , publisher =

    work page 2024

  35. [35]

    A Reparameterized Discrete Diffusion Model for Text Generation

    A Reparameterized Discrete Diffusion Model for Text Generation , author =. Conference on Language Modeling (COLM) , year =. 2302.05737 , archiveprefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [36]

    Neural Computation , volume =

    A Connection Between Score Matching and Denoising Autoencoders , author =. Neural Computation , volume =. 2011 , doi =

    work page 2011

  37. [37]

    International Conference on Learning Representations (ICLR) , year =

    Simple Guidance Mechanisms for Discrete Diffusion Models , author =. International Conference on Learning Representations (ICLR) , year =

    work page

  38. [38]

    Large Language Diffusion Models

    Large Language Diffusion Models , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =. 2502.09992 , archiveprefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  39. [39]

    and Casella, George , edition =

    Robert, Christian P. and Casella, George , edition =. Monte. 2004 , doi =

    work page 2004

  40. [40]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Maximum Likelihood Training of Score-Based Diffusion Models , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

    work page

  41. [41]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    A Variational Perspective on Diffusion-Based Generative Models and Score Matching , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

    work page

  42. [42]

    From Denoising Diffusions to Denoising

    Benton, Joe and Shi, Yuyang and De Bortoli, Valentin and Deligiannidis, George and Doucet, Arnaud , journal =. From Denoising Diffusions to Denoising. 2024 , doi =

    work page 2024

  43. [43]

    Proceedings of the 29th International Conference on Machine Learning (ICML) , pages =

    On Causal and Anticausal Learning , author =. Proceedings of the 29th International Conference on Machine Learning (ICML) , pages =. 2012 , publisher =

    work page 2012

  44. [44]

    New Journal of Physics , volume =

    Algorithmic Independence of Initial Condition and Dynamical Law in Thermodynamics and Causal Inference , author =. New Journal of Physics , volume =. 2016 , doi =

    work page 2016

  45. [45]

    Proceedings of the 26th Annual International Conference on Machine Learning (ICML) , pages =

    Detecting the Direction of Causal Time Series , author =. Proceedings of the 26th Annual International Conference on Machine Learning (ICML) , pages =. 2009 , publisher =. doi:10.1145/1553374.1553477 , url =

    work page doi:10.1145/1553374.1553477 2009

  46. [46]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Causal Inference on Time Series using Restricted Structural Equation Models , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2013 , url =

    work page 2013