Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Exponential Convergence of CAVI for Bayesian PCA

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CAVI for Bayesian PCA converges exponentially, and in the single-component case it is exactly the power iteration algorithm.

desk verdict Genuinely nice k=1 power-iteration result; general-k exponential convergence is conditional on an unverified Hessian assumption that fails for isotropic priors. read the letter →

arxiv 2505.16145 v1 pith:MG7IWS3G submitted 2025-05-22 stat.ML cs.LG

Arghya Datta , Philippe Gagnon , Florian Maire This is my paper · ORCID
classification stat.MLcs.LG MSC 62H2562F1565F15
keywords coordinateascentvariationalinferenceBayesianPCAprobabilisticpoweriterationexponentialconvergenceKullback-Leiblerdivergencemean-field
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Coordinate ascent variational inference (CAVI) for Bayesian probabilistic PCA is the standard way to approximate the posterior, but its convergence rate was uncharacterized. This paper establishes that CAVI converges exponentially: for a model with one principal component, the normalized mean iterates are exactly the power iteration algorithm on the data covariance, so they contract to the leading principal direction at a rate governed by the eigenvalue gap. For any number of components, the paper proves a local exponential contraction of the Kullback–Leibler divergence to the variational optimum, provided a stationary point satisfies two regularity conditions. A by-product is a new lower bound on the symmetric KL divergence between multivariate normals. If correct, these results give the first quantitative convergence guarantees for a widely used variational algorithm and explain why Bayesian PCA reproduces classical PCA point estimates in the single-component case.

What carries the argument

The central objects are the sequential CAVI update maps. For $k=1$ the composed map $G\circ F$ on the mean parameters, after normalization, collapses to $\mu \mapsto XX'\mu/\|XX'\mu\|$, which is precisely the power iteration algorithm for the leading singular vector of $XX'$; this identity carries the sharp rate in Theorem 1. For general $k$, the machinery is the generalized-correlation bound of Bhattacharya et al., which measures how strongly the two variational blocks $q_W$ and $q_Z$ interact through the log-posterior, together with the paper's Lemma 3, a lower bound on the symmetric KL divergence between two multivariate normals in terms of the parameter differences $\|\mu-\mu^*\|$ and $\|\Sigma-\Sigma^*\|$. The lower bound is what turns the abstract contraction condition into the explicit membership condition (16) for the stationary point.

What would settle it

Run CAVI on a $k=2$ BPCA model with non-isotropic prior precision $\Lambda$ (as in the paper's numerical experiment), locate a stationary point $q^*$ by running CAVI then Newton refinement, and compute the eigenvalues of the Hessian $\nabla^2\Psi(q^*)$. If the smallest absolute eigenvalue is at machine precision rather than $O(10^{-2})$, Assumption 2 fails and the local contraction claim collapses. Alternatively, compute the quantity in (16) for the found $q^*$; if it exceeds 1, the generalized-correlation condition is violated.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is an exact dynamical correspondence: when the BPCA model has a single principal component, the CAVI update for the latent mean, normalized, equals the power iteration map $\mu \mapsto XX'\mu / \|XX'\mu\|$ (and analogously for the loadings with $X'X$). This yields Theorem 1, which bounds the distance of the normalized iterates to the leading singular vector by $C(1 - \rho_i/\lambda_i)^t$ from any initialization not orthogonal to that direction. For general $k$, the paper adapts the contraction framework based on the generalized correlation between the two variational blocks, and proves in Theorem 2 that initialization in a KL ball around a suitable stationary point $q^*$ gives $KL(q_Z^{(t+1)}\|q^*_Z) \le C_Z \kappa^t$ and $KL(q_W^{(t+1)}\|q^*_W) \le C_W \kappa^t$ with universal $\kappa\in(0,1)$. The proof rests on a sharp lower bound for the symmetric KL divergence between matrix-normal distributions, Lemma 3, which converts the generalized-correlation condition into an explicit inequality on the stationary-point parameters, displayed in (16).

Load-bearing premise

Assumption 2 — that the variational loss has a non-singular Hessian (no flat directions) at every stationary point — is the load-bearing premise; the authors could not prove it, and it is outright false when the prior precision matrix is a multiple of the identity for $k\ge 2$.

Editorial extensions

If this is right

  • For single-component BPCA, users can now predict the number of iterations needed to reach a given directional accuracy, because the rate is $(1-\rho_1/\lambda_1) = \lambda_2/\lambda_1$ for a generic initialization.
  • The equivalence with power iteration shows that, under a Gaussian prior on the loadings, the variational point estimates of the principal direction coincide with the classical PCA solution for $k=1$.
  • For $k\ge 2$, the local contraction guarantee means that CAVI, when started close enough to a suitable fixed point, cannot oscillate or drift: it converges at a geometric rate, and the parameters (means and covariances) converge at half that rate.
  • The new symmetric-KL lower bound can be reused in any contraction analysis where the two variational families are multivariate normal, providing a tighter alternative to Pinsker-type inequalities.
  • The rotational ambiguity result shows that the non-identifiability of BPCA becomes an algorithmic obstacle for $k\ge2$ when the prior precision is isotropic, and suggests anisotropic priors are needed for local contraction guarantees.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the single-component correspondence is exact, the same normalized-mean dynamics should survive when the noise precision $\tau$ is also estimated by CAVI rather than fixed, so the exponential rate may extend to an empirical-Bayes flavour of BPCA.
  • The local nature of Theorem 2 suggests a practical diagnostic: run CAVI from several random starts; if all land on the same point and the smallest Hessian eigenvalue is bounded away from zero, the contraction assumptions are likely met, giving a certificate of geometric convergence.
  • Lemma 3 may yield convergence rates for other coordinate-ascent schemes on Gaussian latent-variable models, such as factor analysis or Gaussian process latent variable models, where the same block structure appears.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies the convergence rate of coordinate ascent variational inference (CAVI) for Bayesian probabilistic PCA (BPCA). For the single-component case k=1, Theorem 1 shows that the normalized mean iterates of CAVI coincide with the power iteration algorithm applied to X'X or XX', yielding an exponential convergence rate (1-ρ_i/λ_i)^t for the directional components. Section 3.2 then analyzes the scaling components (a(t), b(t)) through a two-dimensional dynamical system, but stops short of a theoretical guarantee. For general k, Theorem 2, under Assumptions 1 and 2 and using a novel lower bound on the symmetric KL divergence between matrix normal distributions (Lemma 3), establishes local exponential contraction of KL(q_Z^{(t)}∥q_Z^*) and KL(q_W^{(t)}∥q_W^*) when the algorithm is initialized in a KL ball around a stationary point q* satisfying condition (16). The paper provides explicit derivations of the CAVI updates, a coercivity-based existence proof, and numerical experiments illustrating the rates.

Significance. The k=1 result is an elegant and precise characterization: it shows that the normalized mean updates are exactly power iteration, yielding a sharp and interpretable rate and substantiating the connection between BPCA point estimates and classical PCA. The general-k result is among the first applications of the Bhattacharya et al. contraction framework to a non-log-concave matrix factorization problem, and Lemma 3 may be of independent use in information theory. However, the general-k theorem is local and conditional on Assumptions 1-2; since Assumption 2 fails for isotropic priors (Proposition 4) and is unverified otherwise, the abstract's claim of proven exponential convergence 'for the model with any number of PCs' is stronger than what is actually established. The paper is candid about these limitations, which is to its credit, but the central claim of the title remains narrower than the results justify.

major comments (4)
  1. [Section 4 and Appendix K (Proposition 4)] Assumption 2 (non-singular Hessian of Ψ at every stationary point) is load-bearing for Theorem 2, as it is used to apply the Bhattacharya et al. (2025) contraction framework and to ensure that q* is isolated. The authors state in Section 4 that proving this assumption is infeasible with standard tools, and Proposition 4 shows that it fails whenever Λ ∝ I_k for k ≥ 2, because the rotational symmetry produces a continuum of stationary points. Since isotropic priors are a natural default in Bayesian PCA, the general-k exponential convergence claim is not established in a standard setting; for non-proportional Λ, the only evidence is the single numerical experiment in Appendix K.1 with n=4, d=3, k=2, τ0=100. This gap directly affects the title-level claim and should be addressed, either by proving Assumption 2 under explicit conditions on Λ and X, or by sharply re-scoping the abstract and title to state the conditional nature of the result.
  2. [Section 3.2] Theorem 1 and the surrounding discussion establish exponential convergence only for the normalized mean directions µ^{(t)}/∥µ^{(t)}∥. The text in Section 3.2 explicitly says that 'it is not possible using such tools to provide a theoretical guarantee of exponential convergence of (a(t), b(t)) and thus of CA VI iterates.' Since the abstract and title claim exponential convergence of CAVI for k=1, the full CAVI iterate sequence (including the scaling components and covariance parameters) is not covered. The claim should be qualified to 'exponential convergence of the estimated principal directions', or the missing analysis of the scaling component must be supplied.
  3. [Section 3.1, proof of Theorem 1 (Appendix B)] The statement of Theorem 1 allows an arbitrary initial index i = min{j : c_j ≠ 0} and gives the rate (1-ρ_i/λ_i)^t, but the proof in Appendix B begins by assuming c_1 ≠ 0 and derives the bound using λ_2/λ_1 = 1-ρ_1/λ_1. The case i>1 is never treated in the proof. While this gap is likely repairable by repeating the same expansion from the index i, as written the proof does not substantiate the full statement of the theorem. Please either generalize the proof or restrict the theorem statement to the case c_1 ≠ 0.
  4. [Appendix C, Remark 1 and Theorem 2 statement] The conversion from the symmetric KL bound that follows from the Bhattacharya et al. framework to the one-sided KL divergence stated in Theorem 2 is not fully established. Remark 1 appeals to the tensorization of KL for matrix normal distributions with row independence, but tensorization alone does not yield a bound on KL(q_Z^{(t)}∥q_Z^*) in terms of D_KL,1/2 without additional control on the eigenvalues of Σ_Z^{(t)} and Σ_Z^*. Please provide a rigorous derivation with explicit conditions, or cite a specific theorem (with assumptions) from Zhang et al. (2023) that justifies this step.
minor comments (6)
  1. [Throughout] The notation is inconsistent: the title and abstract use 'CAVI' while the body mostly uses 'CA VI' with a space; please standardize.
  2. [Equation (16)] The norm notation in condition (16) is inconsistent with the definitions earlier in the paper: (16) uses ∥Σ_W∥ and ∥Σ_Z∥ without specifying the norm, while Lemma 4 and Lemma 13 use the operator norm; please clarify which norm is intended in the condition.
  3. [Section 3.2, Proposition 2] The algebra in the derivation of the quadratic polynomial P is not fully displayed; in particular, the linear coefficient has a term of the form (λ1τ0-n)^2 whose sign should be double-checked, as an error here would change the stated condition for a unique positive root.
  4. [Figures 1 and 2] The y-axes in these figures show only tick marks without axis titles; please add labels such as 'log absolute error' so that the log scale and the plotted quantities are unambiguous.
  5. [Algorithm 1] The stopping criterion is a relative ELBO increase, while the theoretical results are stated for fixed iteration counts; please add a sentence clarifying the relationship between the algorithmic stopping rule and the theoretical iteration-index guarantees.
  6. [References] In Remark 1, the reference to Zhang et al. (2023) is made without a theorem number or page; please cite the specific result that justifies the KL conversion, as the current citation is too vague for a reader to verify the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 1 reduces to a standard power-iteration identity, and Theorem 2 is a conditional external-framework result with independent KL lower bounds.

full rationale

The derivation chain is self-contained against the claimed results. Theorem 1 is an algebraic identification: the paper proves that for k=1 the normalized CAVI mean updates satisfy mu_Z^(t)/||mu_Z^(t)|| = (XX')^t mu_Z^(0)/||(XX')^t mu_Z^(0)||, which is exactly the power iteration algorithm on XX'; convergence to the leading eigenspace then follows from the standard, parameter-free convergence of power iteration. This connection is derived from the CAVI update equations, not assumed in their definition. Theorem 2 is obtained by applying the external contraction framework of Bhattacharya et al. (2025), which is not a self-citation, and by supplying a genuinely new lower bound, Lemma 3, for the symmetric KL divergence of matrix normal distributions. The hypotheses of Theorem 2, namely Assumptions 1 and 2 and condition (16), are sufficiency conditions on the stationary point q*, not restatements of exponential convergence; the constants C_Z, C_W and the rate kappa are existential and are not calibrated to data or fitted values. The paper's own admission that Assumption 2 is infeasible to prove and Proposition 4's demonstration that it fails for Lambda proportional to I_k are honest limitations and correctness risks, but they do not make the theorem circular: the theorem is explicitly conditional on that assumption, and the proof does not invoke the conclusion to establish it. The only self-citation, Datta and Chakrabarty (2023), is background on MLE consistency and is not load-bearing for the CAVI convergence results. No predicted quantity is constructed from a fitted parameter, and no known result is merely renamed. Overall, I find no circular step.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

The theoretical results add no fitted parameters; the only numeric quantities are data and hyperparameters. The load-bearing premises are the spectral assumptions for k=1 and the two unverified assumptions (Assumption 1 and Assumption 2) for general k. The paper also relies on external theorems from Bhattacharya et al. (2025) and Zhang et al. (2023) as black boxes.

assumptions (8)
  • domain assumption X has full rank with distinct positive singular values lambda1 > lambda2 > ... > lambda_d.
    Stated in Section 3.1 before Theorem 1; needed for the power iteration rate and the spectral gap rho_i = lambda_i - lambda_{i+1}.
  • domain assumption CAVI is initialized with mu_Z^(0) not equal to 0 and q_Z^(0) a matrix normal with row-independent covariance; the trivial zero fixed point is excluded.
    Section 2, after the update equations; required for the directional convergence analysis.
  • domain assumption Noise precision tau is fixed and known, equal to tau_0.
    Section 1.1 explicitly assumes this to simplify the theoretical analysis.
  • ad hoc to paper Assumption 1: the set M of stationary points intersects I, the set defined by condition (16).
    Section 4; the condition is expressed in terms of the unknown stationary point q* and is not proven to hold.
  • ad hoc to paper Assumption 2: the loss function Psi has a non-singular Hessian at every stationary point.
    Section 4; Proposition 4 shows it fails when Lambda is proportional to the identity for k >= 2, and only numerical evidence is provided otherwise.
  • standard math Bhattacharya et al. (2025) Lemma 2 and Theorem 4, the generalized correlation framework for CAVI contraction.
    Used as a black box in the proof of Theorem 2; no formal verification is provided in this paper.
  • standard math Zhang et al. (2023) result that KL between matrix normals with row independence tensorizes, allowing reverse KL to be controlled.
    Used in Remark 1 after the proof of Theorem 2 to justify the initialization condition.
  • standard math Standard matrix normal moment formulas from Smidl and Quinn (2006) and Gupta and Nagar (1999).
    Used in Appendices E, F, H to derive the CAVI updates and the ELBO expression.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exponential Convergence of CAVI for Bayesian PCA." pith.science (2026). https://pith.science/paper/MG7IWS3G

@misc{pith2026250516145,
  author       = {Pith},
  title        = {Pith review of: Exponential Convergence of CAVI for Bayesian PCA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MG7IWS3G}},
  note         = {Machine review of arXiv:2505.16145}
}
abstract

Probabilistic principal component analysis (PCA) and its Bayesian variant (BPCA) are widely used for dimension reduction in machine learning and statistics. The main advantage of probabilistic PCA over the traditional formulation is allowing uncertainty quantification. The parameters of BPCA are typically learned using mean-field variational inference, and in particular, the coordinate ascent variational inference (CAVI) algorithm. So far, the convergence speed of CAVI for BPCA has not been characterized. In our paper, we fill this gap in the literature. Firstly, we prove a precise exponential convergence result in the case where the model uses a single principal component (PC). Interestingly, this result is established through a connection with the classical $\textit{power iteration algorithm}$ and it indicates that traditional PCA is retrieved as points estimates of the BPCA parameters. Secondly, we leverage recent tools to prove exponential convergence of CAVI for the model with any number of PCs, thus leading to a more general result, but one that is of a slightly different flavor. To prove the latter result, we additionally needed to introduce a novel lower bound for the symmetric Kullback--Leibler divergence between two multivariate normal distributions, which, we believe, is of independent interest in information theory.

Figures

Figures reproduced from arXiv: 2505.16145 by the authors.

Figure 1
Figure 1. ∥µ (t) Z /∥µ (t) Z ∥ − sgn(c1)µ1∥ (left panel) and ∥µ (t) W /∥µ (t) W ∥ − sgn(c1)X ′µ1/∥X ′µ1∥∥ (right panel) as the algorithm progresses, and upper bounds provided in Theorem 1, on the log scale. where (a (t) , b(t) ) ∈ R 2 + with a (t) representing the norm of µ (t) Z . We now investigate whether the sequence (a (t) , b(t) ) admits a limit. With the initialization (µ (0) Z , Σ (0) Z ) = (a (0)µ1, b(0)), we are in … view at source ↗
Figure 2
Figure 2. |a (t) − a ∗ | (left panel) and |b (t) − b ∗ | (right panel) as the algorithm progresses, on log scale. 4 Theoretical analysis of CAVI in the general case In this section, we analyze CAVI for BPCA when k is an arbitrary integer. We view KL(q∥π) as a function of the variational parameters of q. Thus, we identify a tuple q = (N (µW, Id ⊗ ΣW), N (µZ, In⊗ΣZ)) by its corresponding parameters (µW, ΣW, µZ, ΣZ) ∈ R d×k×S k … view at source ↗
Figure 3
Figure 3. Visualization of Ψ near q ∗ 28 [PITH_FULL_IMAGE:figures/full_fig_p028_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 22 canonical work pages

  1. [1]

    and Lacker, D

    Arnese, M. and Lacker, D. (2024). Convergence of coordinate ascent variational inference for log-concave measures via optimal transport. arXiv preprint arXiv:2404.08792

    arXiv 2024

  2. [2]

    Bhattacharya, A., Pati, D., and Yang, Y. (2025). On the convergence of coordinate ascent variational inference. arXiv preprint arXiv:2306.01122, To appear in the Annals of Statistics

    work page Pith review arXiv 2025

  3. [3]

    Bishop, C. (1998). Bayesian PCA . In Kearns, M., Solla, S., and Cohn, D., editors, Advances in Neural Information Processing Systems , volume 11. MIT Press

    work page 1998

  4. [4]

    Bishop, C. (1999). Variational principal components. In 1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470) , volume 1, pages 509--514 vol.1

    work page 1999

  5. [5]

    Bishop, C. M. and Nasrabadi, N. M. (2006). Pattern recognition and machine learning , volume 4. Springer

    2006

  6. [6]

    Ch \'e rief-Abdellatif, B.-E. (2019). Consistency of ELBO maximization for model selection. In Symposium on advances in approximate Bayesian inference , pages 11--31. PMLR

    work page 2019

  7. [7]

    and Chakrabarty, S

    Datta, A. and Chakrabarty, S. (2023). On the consistency of maximum likelihood estimation of probabilistic principal component analysis. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems , volume 36, pages 28648--28662. Curran Associates, Inc

    work page 2023

  8. [8]

    Ghodhbani, E., Kaaniche, M., and Benazza-Benyahia, A. (2019). Close approximation of K ullback-- L eibler divergence for sparse source retrieval. IEEE Signal Processing Letters , 26(5):745--749

    work page 2019

Show all 26 references
  1. [9]

    Goplerud, M., Papaspiliopoulos, O., and Zanella, G. (2024). Partially factorized variational inference for high-dimensional mixed models. Biometrika , page asae067

    2024

  2. [10]

    Gupta, A. K. and Nagar, D. K. (1999). Matrix variate distributions , volume 104. CRC Press

    1999

  3. [11]

    Hastings, W. K. (1970). Monte carlo sampling methods using markov chains and their applications

    1970

  4. [12]

    and Raiko, T

    Ilin, A. and Raiko, T. (2010). Practical approaches to principal component analysis in the presence of missing values. The Journal of Machine Learning Research , 11:1957--2000

    2010

  5. [13]

    and Zanella, G

    Lavenant, H. and Zanella, G. (2024). Convergence rate of random scan coordinate ascent variational inference under log-concavity. SIAM Journal on Optimization , 34(4):3750--3761

    2024

  6. [14]

    Milnor, J. W. (1963). Morse theory . Number 51. Princeton university press

    1963

  7. [15]

    Neal, R. M. et al. (2011). MCMC using H amiltonian dynamics. Handbook of M arkov chain M onte C arlo , 2(11):2

    2011

  8. [16]

    Plummer, S., Pati, D., and Bhattacharya, A. (2020). Dynamics of coordinate ascent variational inference: A case study in 2d ising models. Entropy , 22(11):1263

    2020

  9. [17]

    Rousseau, H., Ertel, S., Defr \`e re, D., Faramaz, V., and Wagner, K. (2024). Improving mid-infrared thermal background subtraction with principal component analysis. Astronomy & Astrophysics , 687:A147

    2024

  10. [18]

    and Quinn, A

    S m \' dl, V. and Quinn, A. (2006). The variational Bayes method in signal processing . Springer Science & Business Media

    2006

  11. [19]

    R., Xiong, W., and Yang, X

    Swanson, N. R., Xiong, W., and Yang, X. (2020). Predicting interest rates using shrinkage methods, real-time diffusion indexes, and model combinations. Journal of Applied Econometrics , 35(5):587--613

    2020

  12. [20]

    Teschl, G. (2012). Ordinary differential equations and dynamical systems , volume 140. American Mathematical Soc

    2012

  13. [21]

    Tipping, M. E. and Bishop, C. M. (1999). Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 61(3):611--622

    1999

  14. [22]

    Trefethen, L. N. and Bau, D. (2022). Numerical linear algebra . SIAM

    2022

  15. [23]

    Veronesi, P. (2016). Handbook of fixed-income securities . John Wiley & Sons

    2016

  16. [24]

    Wu, S., Hannig, J., and Lee, T. C. (2021). Uncertainty quantification for principal component regression

    2021

  17. [25]

    Zhang, A. Y. and Zhou, H. H. (2020). Theoretical and computational guarantees of mean field variational inference for community detection. The Annals of Statistics , 48(5):2575--2598

    2020

  18. [26]

    K., Liu, W., Chen, Z., Liu, X., and Wang, J

    Zhang, Y., Pan, J., Li, L. K., Liu, W., Chen, Z., Liu, X., and Wang, J. (2023). On the properties of K ullback-- L eibler divergence between multivariate gaussian distributions. Advances in Neural Information Processing Systems , 36:58152--58165

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.