REVIEW 3 major objections 4 minor 19 references
Generative Models with ELBOs Converging to Entropy Sums
T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read For sigmoid belief nets, Gaussian-observable models, and exponential-family mixtures, the ELBO at every stationary point equals a compact sum of entropies.
desk verdict Useful model-specific entropy-sum expressions, but the 'any well-behaved variational family' claim has a concrete counterexample and needs restriction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing identity is Eq. (3), the entropy-sum form of the ELBO at stationary points. To get it, each generative model is written as an exponential-family generative model, and its natural-parameter mappings $\zeta(\Psi)$ and $\eta(z;\Theta)$ are checked against the parameterization criterion (Definition 1): the natural parameters must be recoverable from their own Jacobian via a parameter-dependent vector $\alpha(\Psi)$ or $\beta(\Theta)$. Verifying these two functional equations (parts A and B) is what unlocks Theorem 1 or Theorem 2 of the companion paper; the Gaussian cases work because the variance parameter scales the natural parameters, the sigmoid belief net works because the Bernoulli natural parameter is the logit of $\pi$, and mixtures work because the Jacobian of the standard-to-natural mapping is invertible.
What would settle it
Pick a concrete instance, e.g., a sigmoid belief net with one latent and one binary observable and a finite data set; run ELBO optimization to a stationary point that is a saddle point (or any stationary point) and numerically evaluate both sides of Eq. (10). Any mismatch between the ELBO value and the entropy-sum value would disprove Proposition 1 and the general claim. The same check can be repeated for probabilistic PCA using Eq. (29).
Extended reading notes
Core claim
The central discovery is that for each considered generative model, the ELBO $F(\Phi,\Theta)$ is exactly equal to $\tfrac{1}{N}\sum_n H[q_\Phi^{(n)}(z)] - H[p_\Theta(z)] - \mathbb{E}_{q_\Phi}\{H[p_\Theta(x|z)]\}$ at all stationary points of learning, where $q_\Phi$ is the aggregate posterior. This is established model by model (Propositions 1–5) by expressing the prior and observable distributions in exponential-family form and checking a parameterization criterion (Definition 1) that lets the companion Theorem 1 or Theorem 2 apply. As a consequence, previously known but separate entropy-sum results for variational autoencoders are unified and extended: the list includes sigmoid belief networks, Gaussian observables with scalar or diagonal variance (hence probabilistic PCA and factor analysis), and mixtures of exponential-family components with constant base measure (e.g., gamma mixtures) or non-constant base measure (Poisson mixtures, using pseudo-entropies).
Load-bearing premise
The whole argument leans on the two convergence theorems of the companion paper (arXiv:2209.03077): if those theorems carry hidden regularity conditions that some "well-behaved" variational distributions or non-linear mean functions fail to meet, every model-specific entropy-sum result here inherits the gap.
Editorial extensions
If this is right
- For Gaussian-observable models, the entropy-sum expression simplifies further because the observation entropy is constant, giving closed-form stationary-point ELBOs such as Eq. (29) for probabilistic PCA.
- For exponential-family mixtures, the stationary-point ELBO becomes a closed-form function once the variational posteriors $q^{(n)}(c)$ are known, and for Poisson mixtures the pseudo-entropy sidesteps infinite sums.
- Because the equality holds at saddle points as well as maxima, convergence analyses of ELBO optimization for these models can work with the simpler entropy-sum objective rather than the full data-sum objective.
- The same verification strategy extends to deeper sigmoid belief nets: adding another latent layer adds one more expected-entropy term (Eq. 21).
Reading between the lines
- If the entropy-sum form is exact at stationary points, then the ELBO landscape for these models is governed by a trade-off among three global entropies; one could use the formula to design model-selection criteria that do not require re-estimating the bound after convergence.
- The parameterization criterion may hold for other exponential-family generative models beyond the five treated here, so the same proof template could apply to, e.g., Poisson factor analysis or beta-Bernoulli mixtures, yielding analogous closed forms.
- Because the result holds for any well-behaved variational family, the stationary-point value depends on the variational choice only through the average posterior entropy; this suggests that the tightness of the bound at stationary points is largely controlled by that entropy term.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proves that for several generative models—sigmoid belief networks (Prop. 1), Gaussian-observable models including probabilistic PCA (Props. 2–3), and exponential-family mixtures with constant or non-constant base measures (Props. 4–5)—the ELBO equals an entropy sum at every stationary point. The proofs verify a parameterization criterion (Def. 1) and invoke Theorem 1 or 2 of Luecke and Warnken (2024). The paper also sketches a three-layer SBN extension and discusses gamma and Poisson mixture examples.
Significance. If the entropy-sum identity holds as stated, the paper gives a useful and surprisingly simple characterization of ELBO fixed points for prominent models, with potential applications to model selection and VAE analysis. The model-side verifications are explicit and the gamma/Poisson examples are concrete. However, the claimed generality over variational families is not established and appears false for common fixed-variance Gaussian encoders; the p-PCA example also contains a sign error. The contribution is therefore conditional on repairing these points and on the companion theorems' assumptions being satisfied by the intended variational families.
major comments (3)
- [Abstract, §1, Prop. 2 (Eq. 24)] The identity is claimed to hold for "any well-behaved family of variational distributions," but no condition on the variational family is stated or verified, and the claim is false for a natural reading. For the scalar case of Definition 3 with p(z)=N(0,1), p(x|z)=N(z,v), N=1, x=2, and q(z)=N(m,1), the ELBO stationary equations are m=2/(1+v) and v=(2-m)^2+1, with solution v≈3.383, m≈0.456. The ELBO at this point is ≈−2.132, whereas the right-hand side of Eq. (24) is H[q]−H[p(z)]−H[p(x|z)]≈−2.028, a gap of ≈−0.104. This directly contradicts the universality claim unless the companion theorem contains an unstated assumption excluding fixed-variance Gaussian encoders; because such encoders are common in amortized VAEs, the manuscript must either state and verify the variational-family condition or restrict the claim.
- [§3, Eqs. (28)–(29)] The sign in front of the determinant term is wrong. For p-PCA with full posteriors, H[p(z|x)] = (H/2)log(2πe) − (1/2)log det(σ^{-2}W^TW + I), not plus. Substituting into Eq. (27) gives a final expression with −(1/2)log det(σ^{-2}W^TW + I) − (D/2)log(2πeσ^2), rather than Eq. (29). The printed expression is not equal to the ELBO at stationary points and is inconsistent with the p-PCA marginal log-likelihood.
- [§1 and Props. 2–5] The manuscript relies entirely on Theorem 1 or Theorem 2 of Luecke and Warnken (2024) but never states the full assumptions of those theorems, especially any conditions on the variational family q(n)_Φ. Since the paper's stated goal is to verify those conditions for each model, the missing statement of the q-side assumptions makes the transfer opaque. The authors should either reproduce the relevant theorem assumptions or explicitly point to the exact statements, and they should verify those assumptions for the variational families used in the examples.
minor comments (4)
- [§2, Eq. (21)] The three-layer SBN result is stated "without giving an explicit proof"; since the paper's contribution is proofs, this claim should be either proven or clearly marked as conjectural.
- [Introduction, §3] There are several typographical artifacts in the text, including "avarage" for "average", "/BX" for the expectation symbol, and "fullfilled" in the proof of Prop. 3; these should be corrected.
- [Definition 3] The phrase "well-behaved function" for μ(z;w) is undefined; the theorem application may require regularity conditions, so the term should be made precise.
- [§4.1, Eq. (55)] The notation L(Θ) ≥ F(Φ,Θ) is confusing because L and F are not otherwise distinguished; please define both quantities explicitly before this display.
Circularity Check
No circular reduction by construction; the model-specific results are applications of a self-cited companion theorem, with an unstated variational-family condition that needs independent verification.
full rationale
The derivation chain is not circular in the construction sense: the paper does not fit parameters and call them predictions, nor does it define the entropy-sum equality into the model. Propositions 1-5 verify Definition 1 (parameterization criterion) for each generative model and then invoke Theorem 1 or 2 of Luecke and Warnken (2024); the equality in Eq. 3 is the theorem's conclusion, not an assumed input. The self-citation is load-bearing, but the cited theorem is a separate prior result with its own stated assumptions (EF generative model, constant base measure, parameterization criterion), so this is normal theorem application rather than equivalence-by-definition. Two caveats prevent a score of 0: (i) the paper never states the q-side 'well-behaved' condition, and the fixed-variance Gaussian q counterexample suggests the claimed universality is either false or inherited from an unstated assumption in the companion paper; (ii) the three-layer SBN entropy sum is stated without proof. These are verification and correctness gaps, not circular steps.
Assumptions & free parameters
assumptions (4)
- domain assumption Theorem 1 and Theorem 2 of Luecke and Warnken (2024) are correct and applicable to the listed models.
- domain assumption For Propositions 2 and 3, the prior is an exponential family distribution with constant base measure and satisfies part A of the parameterization criterion.
- domain assumption For Propositions 4 and 5, the Jacobian of the natural-parameter mapping eta(.) is everywhere invertible.
- domain assumption The variational distributions and the mean function mu(z;w) are sufficiently well-behaved for the entropy-sum theorem's regularity conditions.
Cite this review
Pith. "Pith review of Generative Models with ELBOs Converging to Entropy Sums." pith.science (2026). https://pith.science/paper/ZGIDH2A3
@misc{pith2026250109022,
author = {Pith},
title = {Pith review of: Generative Models with ELBOs Converging to Entropy Sums},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZGIDH2A3}},
note = {Machine review of arXiv:2501.09022}
}
read the original abstract
The evidence lower bound (ELBO) is one of the most central objectives for probabilistic unsupervised learning. For the ELBOs of several generative models and model classes, we here prove convergence to entropy sums. As one result, we provide a list of generative models for which entropy convergence has been shown, so far, along with the corresponding expressions for entropy sums. Our considerations include very prominent generative models such as probabilistic PCA, sigmoid belief nets or Gaussian mixture models. However, we treat more models and entire model classes such as general mixtures of exponential family distributions. Our main contributions are the proofs for the individual models. For each given model we show that the conditions stated in Theorem 1 or Theorem 2 of [arXiv:2209.03077] are fulfilled such that by virtue of the theorems the given model's ELBO is equal to an entropy sum at all stationary points. The equality of the ELBO at stationary points applies under realistic conditions: for finite numbers of data points, for model/data mismatches, at any stationary point including saddle points etc, and it applies for any well behaved family of variational distributions.
Reference graph
Works this paper leans on
- [1]
-
[2]
A. Banerjee, S. Merugu, I. S. Dhillon, J. Ghosh, and J. Lafferty. Clustering with bregman divergences. Journal of Machine Learning Research, 6 0 (58): 0 1705--1749, 2005
work page 2005
-
[3]
D. J. Bartholomew, M. Knott, and I. Moustaki. Latent Variable Models and Factor Analysis: A Unified Approach. John Wiley & Sons, 3rd edition, 2011
work page 2011
- [4]
-
[5]
S. Damm, D. Forster, D. Velychko, Z. Dai, A. Fischer, and J. Lücke. The ELBO of variational autoencoders converges to a sum of entropies. In International Conference on Artificial Intelligence and Statistics, volume 206, pages 3931--3960. PMLR, 2023
work page 2023
- [6]
-
[7]
B. Everitt. An Introduction to Latent Variable Models. Chapman and Hall, 1984
work page 1984
- [8]
Show all 19 references
-
[9]
Jordan, Z
M. Jordan, Z. Ghahramani, T. Jaakkola, and L. Saul. An introduction to variational methods for graphical models. Machine Learning, 37: 0 183--233, 1999
1999
-
[10]
D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014
2014
-
[11]
L\"ucke and J
J. L\"ucke and J. Warnken. On the convergence of the ELBO to entropy sums. arXiv preprint arXiv:2209.03077, 2024
2024 arXiv
-
[12]
Makhzani, J
A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644, 2016
2016 arXiv
-
[13]
McLachlan and D
G. McLachlan and D. Peel. Finite mixture models. John Wiley & Sons, 2004
2004
-
[14]
Neal and G
R. Neal and G. Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. In M. I. Jordan, editor, Learning in Graphical Models, pages 355--368. Kluwer, 1998
1998
-
[15]
R. M. Neal. Connectionist learning of belief networks. Artificial intelligence, 56 0 (1): 0 71--113, 1992
1992
-
[16]
S. T. Roweis. EM algorithms for PCA and SPCA . In Advances in Neural Information Processing Systems, volume 10, pages 626--632, 1998
1998
-
[17]
Tipping and C
M. Tipping and C. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society. Series B, 61, 1999
1999
-
[18]
Tomczak and M
J. Tomczak and M. Welling. VAE with a vampprior. In International Conference on Artificial Intelligence and Statistics, volume 84, pages 1214--1223. PMLR, 2018
2018
-
[19]
Velychko, S
D. Velychko, S. Damm, A. Fischer, and J. L \"u cke. Learning sparse codes with entropy-based ELBO s. In International Conference on Artificial Intelligence and Statistics, volume 238, pages 2089--2097. PMLR, 2024
2024
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.