Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis

Florian Shkurti; Jierui Zhu; Qi Chen

REVIEW 3 major objections 4 minor 39 references

Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims a computable, training-data-only bound on generation error for VAEs and diffusion models, exposing an explicit trade-off between encoder and generator generalization as diffusion time grows.

desk verdict The central proof step fails, so the paper's main generalization bound and diffusion-time trade-off are not established; the writing and experiments are fine but the theory needs a real fix. read the letter →

arxiv 2506.00849 v1 pith:GKJGAZ4K submitted 2025-06-01 cs.LG cs.AI

Qi Chen , Jierui Zhu , Florian Shkurti This is my paper

classification cs.LGcs.AI

keywords generalizationboundsinformation-theoreticlearningtheoryvariationalautoencodersdiffusionmodelsconditionalmutualinformationscorematchingtimetrade-offgenerative

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to prove that generalization in any encoder–generator generative model is governed by two quantities: how far each training point's latent code strays from the prior, and how much of the training point still shows through in the generated output once the latent is fixed. It claims this split yields the first VAE generalization bound that accounts for the generator as well as the encoder, and a diffusion-model bound on the divergence between the real and generated distributions whose three terms move in opposite directions as the diffusion time $T$ grows. Because every term can be estimated from training data alone, the framework promises a practical way to choose $T$ and to regularize score matching, and it predicts that longer diffusion time is not automatically better generalization.

What carries the argument

The load-bearing object is a single information-theoretic identity used throughout the proofs: the divergence between the data-joint and model-joint distributions, $D_{\mathrm{KL}}(P_{\hat X, Z \mid X} \| Q_{\hat X \mid Z} \times \pi)$, is claimed to split exactly into the encoder-to-prior divergence $E_X[D_{\mathrm{KL}}(E(X)\|\pi)]$ plus the conditional mutual information $I(\hat X; X \mid Z)$, the term assigned to the generator's generalization. That split is pushed through a standard decoupling lemma, which converts a KL divergence between a coupled distribution and a product distribution into a bound on the difference of expectations under an $R$-sub-Gaussian loss, producing the gap bound of Theorem 4.1. For diffusion models, two further components carry the argument: Lemma 6.1, which uses Girsanov's theorem and the reverse-time SDE to equate the reconstruction gap between the ideal and learned backward processes with the integrated Fisher divergence (the score-matching loss), and Theorem 6.3, an entropy bound on the discretized Langevin update that makes $T_3$ grow linearly in $T$. The collision of these opposing limits is what the paper presents as the diffusion-time trade-off.

What would settle it

Verify the identity behind Theorem 4.1 directly: on a trained VAE with the generator frozen, the sampling chain is $X \to Z \to \hat X$, so the conditional mutual information $I(\hat X; X \mid Z)$ is zero by construction, while the proof's key decomposition requires it to equal the full generator contribution. A reader can estimate this mutual information with the paper's own variational estimator (Appendix D.2) on MNIST and compare the two sides of the equality in Appendix C.1; if they disagree, the generator-generalization term — and with it the diffusion-time trade-off of Theorem 6.2 — is not supported by that derivation.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is Theorem 6.2: after score-matching training, the KL divergence between the true data distribution and the distribution generated at diffusion time $T$ is bounded above by a training-data-only expression written as three labeled terms — $T_1$, a negative per-example KL divergence between each end-time posterior $E_T(X_i)$ and the aggregate end-time distribution $E_T\#\hat P_X$; $T_2$, the end-time posterior's KL divergence to the prior $\pi$; and $T_3$, the conditional mutual information $I(\hat X_0; X_i \mid \hat X_T)$ that the paper reads as the generator's generalization — together with the explicit score-matching loss $\hat L_{\mathrm{ESM}}$. The authors argue that as $T \to \infty$, $T_1$ and $T_2$ both vanish because the forward process reaches its steady-state noise prior, while $T_3$ stays non-zero and grows linearly in $T$ (Theorem 6.3), so the optimal diffusion time is finite, and longer diffusion does not necessarily generalize better. For VAEs, the parallel claim is Theorem 5.1: a Wasserstein-1 bound that includes the generator's generalization, which earlier PAC-Bayes analyses of a fixed generator omitted. The paper also claims the diffusion bound can be estimated from the training set alone, so $T$ can be selected by grid search, and reports synthetic and real-data experiments consistent with the trade-off.

Load-bearing premise

Everything downstream rests on a single decomposition: the divergence between real and generated data splits exactly into an encoder-to-prior term plus a term measuring how much a generated sample still reveals about its training point once the latent is known; when the generator builds a sample from the latent alone, that second term is zero, and the decomposition that carries every later bound stops being the thing the proofs claim it is.

Editorial extensions

If this is right

Diffusion time $T$ can be selected by minimizing the bound estimated on training data alone, an alternative to test-set divergence estimates, which the paper argues are unreliable for high-dimensional data.
Score-matching training can be regularized by the bound's generalization terms, for instance by tuning the noise-schedule coefficients $\beta_0, \beta_1$ or penalizing the score model's gradient, to push the estimated bound down.
Longer diffusion time does not necessarily improve generalization: the encoder terms vanish as $T \to \infty$ while the generator term grows linearly, so the best $T$ is finite and interior.
For VAEs, the generator's generalization can enter the objective as a regularizer estimated with an auxiliary randomly initialized generator, which the paper's experiments connect to a memorization score.
The generator-term sample complexity is $O(1/\sqrt{m})$, whereas a previously analyzed random-feature diffusion model gave $O(m^{-2/5})$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test it, but the same bound could run as a training-time certificate: refit at the $T$ that minimizes the estimated bound and monitor the bound during training, an early-stopping-style loop the framework naturally supports.
Because $T_3$ sums a per-example conditional mutual information over training points, it is a direct measure of per-datapoint leakage into generated samples; the paper draws the memorization link for VAEs only, and the same reading would make the diffusion bound a candidate privacy-auditing score.
Theorem 6.2 leaves the noise schedule $\lambda(t)$ inside the estimated bound, so the framework implies the schedule itself is a second dial on the same encoder–generator balance; comparing schedules through the bound is a natural extension the authors do not undertake.
Latent diffusion models compose a VAE pair with a diffusion pair; the paper motivates them but analyzes only the single stage, and the unified framework suggests the trade-off would appear twice, once per stage, which a direct extension could make precise.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's main result is not proved. In Appendix C.1, the step that introduces I(\hat X_i; X_i | Z_i) relies on an equality that is false under the paper's own definitions. Once E and G are fixed, \hat X_i is just G(Z_i), so it is conditionally independent of X_i given Z_i, and the conditional mutual information is zero. That means the generator-generalization term in Theorem 4.1, the claimed improvement over Mbacke et al., the T3 term in the diffusion bound, and the linear-in-T bound in Theorem 6.3 are all vacuous. This is not a technicality; it is the load-bearing step of the paper.

What is good: the paper is clearly written, the unified VAE/DM perspective is a legitimate angle, and the authors make an honest attempt to compare with prior PAC-Bayes and information-theoretic bounds. The experimental setup is reasonable and code is provided. The idea of a computable, training-data-only bound for selecting diffusion time is useful if it worked.

The soft spots beyond the central proof error: the sub-Gaussian constant R is never specified in experiments, the bounded score norm L is assumed and estimated heuristically, and the empirical validation essentially checks that the estimated bound behaves like the test KL, which is at risk of selection bias since the same bound is used to choose T. The limitations appendix acknowledges the deterministic-setting issue but does not flag the fixed-generator CMI problem.

Overall, the paper should be rejected in its current form. The theory is the point, and the theory does not hold. That said, with a corrected proof that actually captures generator generalization (perhaps via a data-dependent reference measure or a true mutual information between parameters and sample), the underlying idea could be salvageable. A referee would have to re-derive the chain of inequalities from scratch.

Who this is for: readers interested in info-theoretic generalization bounds for generative models. It is not a paper to build on until the proof is fixed. I would still send it to peer review if it came in fresh, because the flaw is non-obvious and the question is worth refereeing, but I would not cite it.

Referee Report

3 major / 4 minor

Summary. The paper proposes a unified information-theoretic framework for bounding the generalization gap of encoder-generator generative models, covering VAEs and score-based diffusion models. The main result, Theorem 4.1, bounds the gap by a sum of an encoder term E[D_KL(E(X_i)||π)] and a generator term I(hat X_i; X_i | Z_i). Corollaries specialize this to Wasserstein and KL divergences, leading to Theorem 5.1 for VAEs and Theorems 6.2–6.3 for diffusion models, where a trade-off in diffusion time T is claimed and the bound is said to be computable from training data alone. Experiments on Swiss Roll, MNIST, and CIFAR10 are presented to support the bound and the trade-off.

Significance. If the proofs were correct, the paper would offer a valuable unified perspective on VAE and diffusion generalization and a practical criterion for tuning T. The paper is ambitious, clearly written, and ships code; Lemma 6.1's connection between reconstruction loss and Fisher divergence is a useful observation. However, the central theorem's proof contains a load-bearing error: the generator term I(hat X_i; X_i | Z_i) is identically zero under the paper's own fixed-generator definitions, so the claimed generator-generalization contribution and the derived diffusion trade-off are not established. This limits the significance of the framework as presented.

major comments (3)

[Appendix C.1, Theorem 4.1] The proof claims the equality D_KL(P_{hat X_i,Z_i|X_i} || Q_{hat X|Z} × π) = E_{X_i}[D_KL(E(X_i)||π)] + I(hat X_i; X_i | Z_i). Under the paper's definitions, for a fixed generator G the reconstruction satisfies hat X_i ~ G(Z_i) regardless of X_i, so P_{hat X_i | Z_i, X_i} = Q_{hat X_i | Z_i} = G(Z_i) and I(hat X_i; X_i | Z_i) = 0. The cross term E log(P_{hat X_i|Z_i,X_i}/Q_{hat X_i|Z_i}) is identically zero. Therefore the generator-generalization term is vacuous, and the bound does not capture any generalization of G. This flaw propagates to Corollaries 4.2 and 4.3, Theorem 5.1, and the T3 term in Theorems 6.2 and 6.3, which are the paper's main contributions. If the authors instead intend G to be data-dependent, the proof needs to define a joint distribution over (S, E, G) and the decoupling argument would involve the mutual information between the learned parameters and the sample, not the per-sample conditional MI as written.
[Theorem 6.3] The first inequality, (1/m) Σ_{i=1}^m I(hat X0; X_i | hat XT) ≤ (1/m) I(hat X0; X_1:m | hat XT), is not justified by the chain rule of mutual information. Conditioning on additional variables can increase mutual information, so I(hat X0; X_i | hat XT, X_1:i-1) is not necessarily bounded below by I(hat X0; X_i | hat XT). Thus the upper bound on T3, and consequently the linear-in-T growth claim, is unsupported even aside from the flaw in Theorem 4.1.
[Sec. 4, Appendix C.1, Sec. G.2.1] The bound requires the loss -log q_G(X|Z) to be R-sub-Gaussian, but for the Gaussian decoder used in Sec. 5 and Sec. G.1 this loss is quadratic in X and is generally not sub-Gaussian unless X is bounded or has very strong tail conditions; the paper does not verify this condition for the experiments. Furthermore, the 'computable' bound depends on the variance proxy R and the score-norm bound L, which are chosen heuristically in Sec. G.2.1, so the claim that the bound is estimated 'based solely on the training data' is overstated. These choices should be made explicit and justified, or the practical-guidance claims should be softened.

minor comments (4)

[Sec. 3.2] The definition of L^π_PX(E,G) does not actually depend on E, since Z is sampled from π rather than from E(X); the role of E in the generation error should be clarified in the notation or text.
[Theorem 4.1 statement] The phrase '∀X_i ∈ S' is unconventional; it should be 'for each i ∈ [m]' or 'for all i' to avoid treating X_i as a realized value in the probabilistic statement.
[Proof of Theorem 6.2, Appendix E.2] The step E_S[h(hat P_X)] - h(P_X) ≤ 0 uses Jensen's inequality on the entropy, but h(hat P_X) is the entropy of the empirical measure (equal to log m for continuous data) while h(P_X) is a differential entropy; the two are not directly comparable in this form, and this step needs a careful justification or a corrected argument for the T1 term.
[Sec. 7.1 and Fig. 2] The Monte Carlo estimation of the bound uses only 5 seeds for the Swiss Roll experiments, and the few-shot MNIST/CIFAR10 results in Fig. 3 do not report standard deviations; error bars or multiple seeds would help support the trade-off and non-vacuousness claims.

Circularity Check

1 steps flagged · score 6.0 of 10

The generator-generalization term I(\hat X_i; X_i | Z_i) is identically zero under the paper's own sampling definition, so the claimed generator term and the T3 diffusion trade-off reduce by construction.

self definitional [Appendix C.1 (proof of Theorem 4.1); Theorem 4.1 statement; Section 4 Discussion]
"∀Xi ∈S, Zi ∼E(X i), ˆXi ∼G(Z i) ... the last equality holds because the reconstruction process and the generation process use the same generator G. ... = EXi [DKL (E(Xi)∥π)] +I( ˆXi;X i|Zi). The condition mutual information in the second term I( ˆXi;X i|Zi) measures the generalization of the generator G."

Under the theorem's own sampling, \hat X_i is drawn from G(Z_i) and the reference conditional is Q_{\hat X|Z}=G(Z); hence P_{\hat X_i|Z_i,X_i}=G(Z_i)=Q_{\hat X|Z}. The conditional KL D_KL(P_{\hat X_i|Z_i,X_i}||Q_{\hat X|Z}) is therefore zero, and I(\hat X_i;X_i|Z_i)=0 identically. The claimed 'generator generalization' term is thus fixed to zero by the paper's definition of the generative process, not measured or predicted. All downstream uses of this term (Theorem 5.1's generator term, Theorem 6.2's T3, Theorem 6.3's linear-in-T bound) inherit this definitional collapse: they are bounds on a quantity that the paper's own model sets to zero.

full rationale

The encoder-side derivation is self-contained: Theorem 4.1 uses Donsker-Varadhan plus sub-Gaussianity, Corollaries 4.2/4.3 follow by standard couplings, Lemma 6.1 uses Girsanov, and Theorem 6.3 is an information-chain-rule bound. No parameter is fitted to the target KL and no load-bearing conclusion rests on a self-citation. However, the paper's advertised generator-generalization contribution is definitionally empty: because \hat X_i is sampled from G(Z_i) and the reference generator is the same G, the conditional mutual information I(\hat X_i;X_i|Z_i) vanishes by construction. The paper nonetheless interprets this zero term as 'the generalization of the generator' and propagates it into the VAE bound and the diffusion trade-off (T3). That is a self-definitional reduction of one of the paper's central predictions, though the encoder-side terms retain independent content. The empirical sections also estimate the proposed bound with heuristically chosen R and L and then cite the bound's own U-shape as validation; this is weaker than an external falsification, but it is not a fitted-parameter circularity because the test KL is held out.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. The theoretical result rests on standard information-theoretic tools, plus several domain assumptions (sub-Gaussianity, bounded score, SDE regularity) and two ad hoc structural assumptions used in proofs. The sub-Gaussian constant R and the score norm L are not pinned down, which leaves the bound estimation under-specified.

free parameters (3)

Sub-Gaussian proxy R = unspecified
Scales the bound and the position of the minimum over T; the paper never states how R is chosen in the experiments, making the computed bound not reproducible and the trade-off potentially adjustable.
Score norm bound L = estimated heuristically
Used in Theorem 6.3 to bound T3; estimated via 'step-wise estimation' similar to gradient norms, with no error analysis.
Step size tau / number of steps N = N=1000 or tau=0.001
Affects the discretized bound and therefore the value of T3; the choice is experimental and not derived from the theory.

assumptions (5)

domain assumption The loss function Delta is R-sub-Gaussian under the product measure (Def. A.4)
Used in Theorem 4.1 and Corollaries; may fail for -log q_G(X|Z) which is typically not sub-Gaussian (chi-squared-type tails).
domain assumption Existence and regularity of the reverse-time SDE as in Song et al. (2021)
Invoked in Lemma 6.1 and Theorem 6.2 to relate the empirical reconstruction error to the Fisher divergence.
domain assumption Bounded score norm ||grad log p_t(x)|| <= L
Used in Theorem 6.3 to bound the CMI term; strong on data manifolds where the score can blow up.
ad hoc to paper The learning algorithm is symmetric with respect to training samples (Appendix D.2)
Used to claim I(X_hat_i; S | Z_i) >= m I(X_hat_i; X_i | Z_i); not stated among the main assumptions in Section 3.
ad hoc to paper Markov chain conditional independences in the proof of Theorem 6.3
The proof claims X_j -> Z_i, X_{1:j-1} -> Z_i and Z_i -> X_hat_i, X_j -> X_hat_i; these are not justified and are questionable for a data-dependent generator.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis." pith.science (2026). https://pith.science/paper/GKJGAZ4K

@misc{pith2026250600849,
  author       = {Pith},
  title        = {Pith review of: Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GKJGAZ4K}},
  note         = {Machine review of arXiv:2506.00849}
}

abstract

Despite the empirical success of Diffusion Models (DMs) and Variational Autoencoders (VAEs), their generalization performance remains theoretically underexplored, especially lacking a full consideration of the shared encoder-generator structure. Leveraging recent information-theoretic tools, we propose a unified theoretical framework that provides guarantees for the generalization of both the encoder and generator by treating them as randomized mappings. This framework further enables (1) a refined analysis for VAEs, accounting for the generator's generalization, which was previously overlooked; (2) illustrating an explicit trade-off in generalization terms for DMs that depends on the diffusion time $T$; and (3) providing computable bounds for DMs based solely on the training data, allowing the selection of the optimal $T$ and the integration of such bounds into the optimization process to improve model performance. Empirical results on both synthetic and real datasets illustrate the validity of the proposed theory.

Figures

Figures reproduced from arXiv: 2506.00849 by the authors.

**Figure 2.** Evolution of bound and test-data KL-divergence estimation w.r.t (a) train dataset size [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** (a) For DM trained on few-shot MNIST and CIFAR10 data ( [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: (a)The test VAE loss, previous PAC Bayes bound (converted to expectation bound) and our [PITH_FULL_IMAGE:figures/full_fig_p031_4.png]

**Figure 5.** Figure 5: Sampling results w.r.t. different train data size [PITH_FULL_IMAGE:figures/full_fig_p033_5.png]

**Figure 6.** Figure 6: Sampling results w.r.t. different diffusion time [PITH_FULL_IMAGE:figures/full_fig_p034_6.png]

**Figure 7.** Figure 7: The evolution of the estimated bounds, test data KL divergences, and test data log densities [PITH_FULL_IMAGE:figures/full_fig_p035_7.png]

**Figure 8.** Figure 8: The generated images for different diffusion times [PITH_FULL_IMAGE:figures/full_fig_p036_8.png]

**Figure 9.** Figure 9: The generated images for different diffusion times [PITH_FULL_IMAGE:figures/full_fig_p036_9.png]

**Figure 10.** Figure 10: The generated images for different diffusion times [PITH_FULL_IMAGE:figures/full_fig_p037_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 22 canonical work pages

[1]

Catoni-style

log ( x+1 2 )) yields the Jensen-Shannon (JS) divergence DJS (P∥Q). Definition A.2(Fisher Divergence).Let P and Q be two probability measures defined on X , then, we have the fisher divergence: DF isher(P∥Q) def =E X∼P ∥∇X logp(X)− ∇X logq(X)∥ 2 2 , wherep(x)andq(x)are the probability density functions. Definition A.3(Lipschitz function).Let (W,∥ · ∥)be a...

work page 2019
[3]

the trade-off on diffusion time T , showed in Fig. 2 (b). To make such a comparison, we need a quantitative estimation of the two sides. Recall that DKL (PX ∥Qπ Gθ T ) measures the proximity of the original data distribution to the generated data distribution. A similar metric used to evaluate the performance of generative models is the Fréchet inception ...

work page 2018
[8]

beta-vae: Learning basic visual concepts with a constrained variational framework.ICLR (Poster), 3,

12 Published as a conference paper at ICLR 2025 Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework.ICLR (Poster), 3,

work page 2025
[10]

Dataset size dependence of rate-distortion curve and threshold of posterior collapse in linear vae.arXiv preprint arXiv:2309.07663,

Yuma Ichikawa and Koji Hukushima. Dataset size dependence of rate-distortion curve and threshold of posterior collapse in linear vae.arXiv preprint arXiv:2309.07663,

arXiv
[11]

Learning dynamics in linear vae: Posterior collapse threshold, superfluous latent space pitfalls, and speedup with kl annealing

Yuma Ichikawa and Koji Hukushima. Learning dynamics in linear vae: Posterior collapse threshold, superfluous latent space pitfalls, and speedup with kl annealing. InInternational Conference on Artificial Intelligence and Statistics, pp. 1936–1944. PMLR,

work page 1936
[12]

Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,

Diederik P Kingma and Max Welling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,

arXiv
[13]

On generalization error bounds of noisy gradient methods for non-convex learning.arXiv preprint arXiv:1902.00621,

Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization error bounds of noisy gradient methods for non-convex learning.arXiv preprint arXiv:1902.00621,

arXiv 1902
[14]

Adversarial autoencoders.arXiv preprint arXiv:1511.05644,

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders.arXiv preprint arXiv:1511.05644,

arXiv

Show all 39 references

[17]

Dispersed exponential family mixture vaes for interpretable text generation

14 Published as a conference paper at ICLR 2025 Wenxian Shi, Hao Zhou, Ning Miao, and Lei Li. Dispersed exponential family mixture vaes for interpretable text generation. InInternational Conference on Machine Learning, pp. 8840–8851. PMLR,

2025
[18]

Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020a

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020a. Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32,

2010 arXiv
[19]

Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020b

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020b. Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum...

2011 arXiv
[20]

A note on the evaluation of generative models.arXiv preprint arXiv:1511.01844,

Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models.arXiv preprint arXiv:1511.01844,

arXiv
[21]

Wasserstein auto-encoders

Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558,

arXiv
[22]

Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit.arXiv preprint arXiv:1905.09883,

Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit.arXiv preprint arXiv:1905.09883,

1905 arXiv
[24]

An information-theoretic analysis for transfer learning: Error bounds and applications.arXiv preprint arXiv:2207.05377,

Xuetong Wu, Jonathan H Manton, Uwe Aickelin, and Jingge Zhu. An information-theoretic analysis for transfer learning: Error bounds and applications.arXiv preprint arXiv:2207.05377,

arXiv
[25]

Information-theoretic analysis of generalization capability of learning algorithms

15 Published as a conference paper at ICLR 2025 Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. InAdvances in Neural Information Processing Systems, pp. 2524–2533,

2025
[26]

Intersectional unfairness discovery.arXiv preprint arXiv:2405.20790,

Gezheng Xu, Qi Chen, Charles Ling, Boyu Wang, and Changjian Shui. Intersectional unfairness discovery.arXiv preprint arXiv:2405.20790,

arXiv
[27]

, m} Upper case letter (e.g

16 Published as a conference paper at ICLR 2025 A PRELIMINARIES A.1 NOTATIONTABLE Table 1: Summary of major notations Symbol Meaning [m] {1, . . . , m} Upper case letter (e.g. Y) Random variable Calligraphic letters (e.g.Y) Support sets of random variables P(Y) The set of all ...

2025
[31]

36 Published as a conference paper at ICLR 2025 (a) train data (b)T= 0.2 (c)T= 0.4 (d)T= 0.6 (e)T= 0.8 (f)T= 1.0 (g) train data (h)T= 1.2 (i)T= 1.4 (j)T= 1.6 (k)T= 1.8 (l)T= 2.0 Figure 10: The generated images for different diffusion timesT on the CIFAR10 dataset (we randomly ...

2025
[32]

By improving the generalization, we can decrease replicated generation to help address the privacy and copyright issues in generative models

H BROADERIMPACTS ANDLIMITATIONS Broader Impacts • Potential positive impactsWe study the theoretical aspects of generative models. By improving the generalization, we can decrease replicated generation to help address the privacy and copyright issues in generative models. • Po...

2020
[34]

and location-scale families (Park et al., 2019), balancing the rate-distortion trade-off (Higgins et al., 2017; Rybkin et al., 2021), replacing the regularization term with adversarial objectives (Makhzani et al., 2015), or using other divergences like the Wasserstein distance...

2019
[35]

37 Published as a conference paper at ICLR 2025 Score-based diffusion modelsSong et al

(W AE). 37 Published as a conference paper at ICLR 2025 Score-based diffusion modelsSong et al. (2020b) unifies the previous two main diffusion ap- proaches: Score matching with Langevin dynamics (SMLD) (Song & Ermon,

2020
[36]

and Diffusion probabilistic modeling (DDPM) (Sohl-Dickstein et al., 2015; Ho et al.,

2015
[37]

Later on, the variational perspective of these models was studied in (Huang et al., 2021; Kingma et al., 2021; Franzese et al., 2023)

as score-based diffusion models, where their forward processes are considered as different families of Stochastic Differential Equations (SDEs). Later on, the variational perspective of these models was studied in (Huang et al., 2021; Kingma et al., 2021; Franzese et al., 2023...

2021
[38]

Convergence theory for diffusion modelsDe Bortoli et al

have gained great success in generating high-resolution images, extended to further applications like text-to-image editing (Han et al., 2024; Huberman-Spiegelglas et al., 2024). Convergence theory for diffusion modelsDe Bortoli et al. (2021) are the first to give quantitative...

2021
[39]

have provided a rigorous framework for understanding the generalization capabilities of deep learning models, and have further been extended to complex learning scenarios, such as meta-learning (Chen et al., 2021; 2023b) and domain adaptation (Wu et al., 2022; Chen & Marchand,...

2021
[1000]

5 and Fig

Additional ResultsIn Fig. 5 and Fig. 6, we plot the generated data with the score model obtained at the last iteration for each specific setting,e.g., different train sample sizemand diffusion timeT. m = 10 m = 100 m = 200 m = 600 m = 800 m = 2000 Figure 5: Sampling results w....

2020
[1997]

Vae approximation error: Elbo and exponential families.arXiv preprint arXiv:2102.09310,

Alexander Shekhovtsov, Dmitrij Schlesinger, and Boris Flach. Vae approximation error: Elbo and exponential families.arXiv preprint arXiv:2102.09310,

arXiv
[2002]

Rate-regularization and generalization in vaes.arXiv preprint arXiv:1911.04594,

Alican Bozkurt, Babak Esmaeili, Jean-Baptiste Tristan, Dana H Brooks, Jennifer G Dy, and Jan-Willem van de Meent. Rate-regularization and generalization in vaes.arXiv preprint arXiv:1911.04594,

1911 arXiv
[2011]

Tighter information-theoretic generalization bounds from supersam- ples.arXiv preprint arXiv:2302.02432,

Ziqiao Wang and Yongyi Mao. Tighter information-theoretic generalization bounds from supersam- ples.arXiv preprint arXiv:2302.02432,

arXiv
[2013]

has been widely applied and improved algorithmically through numerous extensions that include changing the posterior distribution to exponential families (Shi et al., 2020; Shekhovtsov et al.,

2020
[2015]

Pac-bayesian generalization bounds for adversarial generative models

13 Published as a conference paper at ICLR 2025 Sokhna Diarra Mbacke, Florence Clerc, and Pascal Germain. Pac-bayesian generalization bounds for adversarial generative models. InInternational Conference on Machine Learning, pp. 24271– 24290. PMLR,

2025
[2017]

Multi-rate vae: Train once, get the full rate-distortion curve.arXiv preprint arXiv:2212.03905,

Juhan Bae, Michael R Zhang, Michael Ruan, Eric Wang, So Hasegawa, Jimmy Ba, and Roger Grosse. Multi-rate vae: Train once, get the full rate-distortion curve.arXiv preprint arXiv:2212.03905,

arXiv
[2019]

Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

1901
[2020]

Quantifying memorization across neural language models.arXiv preprint arXiv:2202.07646,

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models.arXiv preprint arXiv:2202.07646,

arXiv
[2021]

On the stability-plasticity dilemma in continual meta-learning: Theory and algorithm.Advances in Neural Information Processing Systems, 36:27414–27468, 2023b

Qi Chen, Changjian Shui, Ligong Han, and Mario Marchand. On the stability-plasticity dilemma in continual meta-learning: Theory and algorithm.Advances in Neural Information Processing Systems, 36:27414–27468, 2023b. Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, an...

arXiv
[2022]

Convergence of denoising diffusion models under the manifold hypothesis

Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.05314,

arXiv
[2023]

Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions

11 Published as a conference paper at ICLR 2025 Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. InInternational Conference on Machine Learning, pp. 4735–4763. PMLR, 2023...

2025
[2024]

Adversarial networks and autoencoders: The primal-dual relationship and generalization bounds.arXiv preprint arXiv:1902.00985,

Hisham Husain, Richard Nock, and Robert C Williamson. Adversarial networks and autoencoders: The primal-dual relationship and generalization bounds.arXiv preprint arXiv:1902.00985,

1902 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Catoni-style

log ( x+1 2 )) yields the Jensen-Shannon (JS) divergence DJS (P∥Q). Definition A.2(Fisher Divergence).Let P and Q be two probability measures defined on X , then, we have the fisher divergence: DF isher(P∥Q) def =E X∼P ∥∇X logp(X)− ∇X logq(X)∥ 2 2 , wherep(x)andq(x)are the probability density functions. Definition A.3(Lipschitz function).Let (W,∥ · ∥)be a...

work page 2019

[2] [3]

the trade-off on diffusion time T , showed in Fig. 2 (b). To make such a comparison, we need a quantitative estimation of the two sides. Recall that DKL (PX ∥Qπ Gθ T ) measures the proximity of the original data distribution to the generated data distribution. A similar metric used to evaluate the performance of generative models is the Fréchet inception ...

work page 2018

[3] [8]

beta-vae: Learning basic visual concepts with a constrained variational framework.ICLR (Poster), 3,

12 Published as a conference paper at ICLR 2025 Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework.ICLR (Poster), 3,

work page 2025

[4] [10]

Dataset size dependence of rate-distortion curve and threshold of posterior collapse in linear vae.arXiv preprint arXiv:2309.07663,

Yuma Ichikawa and Koji Hukushima. Dataset size dependence of rate-distortion curve and threshold of posterior collapse in linear vae.arXiv preprint arXiv:2309.07663,

arXiv

[5] [11]

Learning dynamics in linear vae: Posterior collapse threshold, superfluous latent space pitfalls, and speedup with kl annealing

Yuma Ichikawa and Koji Hukushima. Learning dynamics in linear vae: Posterior collapse threshold, superfluous latent space pitfalls, and speedup with kl annealing. InInternational Conference on Artificial Intelligence and Statistics, pp. 1936–1944. PMLR,

work page 1936

[6] [12]

Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,

Diederik P Kingma and Max Welling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,

arXiv

[7] [13]

On generalization error bounds of noisy gradient methods for non-convex learning.arXiv preprint arXiv:1902.00621,

Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization error bounds of noisy gradient methods for non-convex learning.arXiv preprint arXiv:1902.00621,

arXiv 1902

[8] [14]

Adversarial autoencoders.arXiv preprint arXiv:1511.05644,

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders.arXiv preprint arXiv:1511.05644,

arXiv