REVIEW 4 major objections 8 minor 27 references
Geodesic Variational Bayes for Multiway Covariances
T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A joint Kronecker-structured variational family, optimized on a pullback metric made positive definite by determinant normalization, converges faster and tracks the unstructured Inverse-Wishart posterior better than mean-field variational…
desk verdict Real geometry result, unsupported empirical headline; deserves review but not acceptance as is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the pullback of the affine-invariant metric under the Kronecker product map, together with a determinant projection. The affine-invariant metric $g_{\Sigma}(U,V)=\mathrm{tr}(\Sigma^{-1}U\Sigma^{-1}V)$ is the Hessian of the Boltzmann entropy and connects to Fisher information. Its pullback to the product manifold is degenerate; Proposition 2.3 repairs this by requiring $|\Sigma_i|=1$ for $i>1$, which zeroes out cross terms and yields the direct-sum metric $G(\Sigma_i|\Sigma_{-i})=(1/d_{-i})\Sigma_i^{-1}\otimes\Sigma_i^{-1}$. Optimization then uses geodesic updates $\Sigma_i(t)=\Sigma_i^{1/2}\exp(t\Sigma_i^{-1/2}V_i\Sigma_i^{-1/2})\Sigma_i^{1/2}$, with tangent vectors projected by $P_{\Sigma_i}(V)=V-\mathrm{tr}(V\Sigma_i^{-1})/d_i\,\Sigma_i$. The trace identity of Proposition 1.3 supplies the efficient gradient terms $T^{(k)}(S,\Sigma_{-k})$ that avoid sample-size-scaled tensor contractions.
What would settle it
Take a small array, say $D=3$ with dimensions $(5,6,4)$, choose two feasible points in the orthogonalized product manifold, evaluate the ELBO along the affine-invariant geodesic connecting them, and test the convexity inequality $f(\gamma(t)) \leq (1-t)f(p)+t f(q)$. A single violation would refute the geodesic-convexity premise; equivalently, running Algorithm 2 from many random starts and seeing divergent optima would indicate the reported convergence is not to a global optimum.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that the degeneracy of the pullback metric has a simple fix, and the fix is what makes joint variational inference practical. The map $\phi: \times_{i=1}^D P(d_i) \to P(\otimes_{i=1}^D d_i)$ sending $(\Sigma_1,\ldots,\Sigma_D)$ to $\otimes_{i=1}^D \Sigma_i$ induces a metric whose off-diagonal blocks wipe out the determinant: under the affine-invariant metric, the pullback $\Phi^*G$ is not positive definite. Proposition 2.3 gives the resolution: impose $|\Sigma_i|=1$ for $i>1$, so the tangent space condition $\mathrm{tr}(d\Sigma_i \Sigma_i^{-1})=0$ kills all cross terms and leaves $\Phi^*G = \oplus_{i=1}^D (1/d_{-i}) \Sigma_i^{-1}\otimes\Sigma_i^{-1}$, which is positive definite. On this orthogonalized manifold, the Riemannian gradients of the joint ELBO reduce to simple expressions, the geodesic updates factor mode by mode, and the joint variational family $\mathrm{IW}(\nu, \otimes_{i=1}^D A_i)$ has a single variational degree of freedom $\nu$. The empirical claim, demonstrated in simulations, is that this joint approximation is both faster and more faithful to the unstructured posterior than the mean-field approximation of the independently-prior model, and that it reproduces the mode-wise eigenstructure of a full sampler on commodity-trade data.
Load-bearing premise
The load-bearing premise is that the full ELBO, including the variational entropy and trace terms, is geodesically convex on the orthogonalized product manifold; the paper asserts this from a cited convexity result for the Gaussian likelihood but does not prove it for the whole objective.
Editorial extensions
If this is right
- In the truly separable case, a single joint Inverse-Wishart family with Kronecker scale replaces $D$ independent families, so variational inference has one degrees-of-freedom parameter instead of $D$ and converges in fewer iterations.
- The orthogonalized pullback metric gives positive-definite Riemannian gradients and geodesic updates that factor mode by mode, making the separable covariance variational problem a well-defined geometric optimization.
- The trace identity lets the ELBO and its gradients be computed in a way that does not scale with sample size, removing a computational bottleneck of Gibbs samplers for tensor data.
- On commodity-trade data, the joint approximation recovers the same exporter, importer, and commodity eigenstructure as a full sampler, so the approximate posterior remains interpretable.
- Under misspecification with low-rank noise added to a separable covariance, the joint approximation stays stable while the mean-field approximation needs dramatically more iterations or fails to converge.
Reading between the lines
- If the ELBO's geodesic convexity is confirmed, Algorithm 2 would come with a global-optimality guarantee, making the comparison with mean-field a fair test rather than a race to different local optima.
- The determinant normalization singles out mode 1 as the carrier of overall scale; an equivariant counterpart that spreads the normalization across modes could make the geometry invariant to how the tensor is indexed.
- Because mean-field samples are always perfectly Kronecker-separable while joint Inverse-Wishart samples are not, the joint approximation should also be preferred when the goal is uncertainty about nonseparable structure, not just the separable mean.
- A natural extension is to test the two approximations by posterior predictive coverage on held-out tensor entries under misspecification, since average Mahalanobis distance on separable truth is only one scoring rule.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a Riemannian optimization framework for variational Bayes on Kronecker-structured covariance matrices arising from tensor-normal (multiway) data. It constructs the pullback of the affine-invariant metric under the Kronecker product map, proves that the naive pullback metric is degenerate, and proposes an orthogonalization condition (unit determinants on all but one factor) that renders the pullback metric positive definite and block-diagonal. On this basis, the paper derives Euclidean and Riemannian gradients for ELBOs of a joint Inverse-Wishart variational family with Kronecker scale and of a mean-field family, and it reports simulation experiments comparing the two families in terms of optimization iterations and posterior predictive Mahalanobis distance, plus an application to UN commodity trade data.
Significance. If the geometric and variational claims survive scrutiny, the paper would make a useful contribution to Bayesian inference for multiway covariance models: the pullback-metric construction with its explicit degeneracy resolution is a natural and nontrivial geometric result, and the efficient computation of the trace term in Proposition 1.3 is practically valuable. The paper also ships a concrete algorithm with gradient derivations, which strengthens its reproducibility potential. However, the current manuscript contains load-bearing weaknesses in the convexity justification, the mean-field ELBO derivation, and the experimental comparisons, so the significance can only be assessed after these are repaired.
major comments (4)
- [Section 2.1] The assertion that 'all ELBO computations considered later in this paper are geodesically convex' is not supported. Wiesel [26] establishes geodesic convexity for maximum likelihood estimation of Kronecker-structured covariance matrices, not for a variational ELBO that includes an entropy term, an Inverse-Wishart prior term, and a trace term, all under a pullback metric on an orthogonalized product manifold. No theorem or proof is given for the joint or mean-field ELBOs. This is load-bearing because Algorithm 2's convergence and the speed comparisons in Section 4 rely on the optimization being well-behaved; without a convexity proof, iterates could be converging to local optima, and the reported gap between joint and mean-field would not measure what the paper claims. Please either prove geodesic convexity for the specific ELBOs or substantially weaken the claim and discuss the risk of local optima.
- [Supplement, Section 6.2] The supplement's derivation of the mean-field ELBO is internally inconsistent with Section 3.2. The final displayed equation in Supplement Section 6.2 is exactly the joint ELBO from Supplement Section 6.1, with a single degree-of-freedom parameter ν_v and the same trace term tr([⊗_i A_i^{-1}](S+Λ)). In contrast, the main-text mean-field bound in Section 3.2 uses separate ν_{vi} and a product ∏_j (ν_{vj}/2) in the trace term. Because the mean-field gradients in Section 3.2 and all mean-field experiments derive from the mean-field ELBO, this duplication means the mean-field results lack a correct derivation. Please fix the supplement to derive the factored mean-field bound consistently with Section 3.2.
- [Section 4.1, Figures 1-3 and Table 1] The optimization-efficiency claim is not demonstrated because the comparison is not controlled. The joint approximation uses a global step size ε_joint = 10^{-4.4} while the mean-field uses ε_MF = 10^{-6}, and the text states that the mean-field was 'not necessarily run until convergence.' A single step-size choice can dominate iteration counts in Riemannian gradient descent, so the reported gap may be an optimization artifact rather than a property of the variational family. Please report wall-clock times, run both methods to a common convergence criterion, and perform a step-size sensitivity analysis across a grid for both methods, rather than selecting the 'most successful' step per method post hoc.
- [Section 4.1, Figure 2 and Abstract] The fidelity comparison is not apples-to-apples. The joint approximation targets the unstructured Inverse-Wishart posterior, while the mean-field baseline approximates a different Bayesian model with independent priors (Section 3.2, based on Hoff [12]). Therefore the abstract's claim that the joint approximation provides 'a superior approximation to an unstructured Inverse-Wishart posterior' is not established: the mean-field family is not attempting to approximate that posterior. The caption itself concedes the mean-field penalty may be 'unsurprising.' A fair comparison would use a mean-field family targeting the same unstructured posterior (for example, a factorized approximation over the modes or entries of the covariance), or the claim should be restated as a comparison of two different model/posterior choices rather than a statement about approximation fidelity to a fixed target.
minor comments (8)
- [Section 2.2, Proposition 2.2] The proof's Schur-complement computation appears incorrect in the diagonal blocks: g_ii - g_{i1} g_{11}^{-1} g_{1i} is not generally zero because the outer product v(Σ_i^{-1})v(Σ_i^{-1})^T differs from Σ_i^{-1}⊗Σ_i^{-1}. The degeneracy nevertheless holds via the scaling-invariance direction (V_i = c_i Σ_i with ∑ c_i = 0), so the proposition's conclusion is correct, but the proof should be revised.
- [Section 3.1] The Euclidean gradient formula for ∇_{A_i} LB includes a term '-(n+ν)/2 d_{-1} A_1^{-1}' for a general index i. Under the orthogonalization condition, the prior log-determinant term contributes only to A_1; for i>1 the term should be absent (or, in a non-orthogonalized formulation, it would be d_{-i} A_i^{-1}). As written, the formula is dimensionally inconsistent for i>1.
- [Section 3.1] In the sentence 'we would instead have ∑_{j=1}^D d_{-j} log|A_j| = d_{-1} A_1', the right-hand side should be 'd_{-1} log|A_1|'; the log is missing.
- [Section 3.2] The gradient for ν_{vi} contains the undefined symbol 'C_p(S+Λ)'; please define this quantity or correct the notation.
- [Supplement, Sections 6.1 and 6.2] The paper consistently misspells 'ELBO' as 'EBLO' in the supplement (e.g., 'non-orthogonalized EBLO'); please correct throughout.
- [Section 1.3, Proposition 1.3] The definition of T^{(k)}(S, Σ_{-i}) is difficult to parse; the formula for C_k(S, Σ_{-i}) has a missing closing parenthesis and the indexing of S_Γ is not immediately transparent. A small worked example would improve readability.
- [Section 4.1] The data-generation details are incomplete: no random seeds, prior hyperparameter values for Λ, or exact model specification are given for the simulations, which hinders reproducibility. Please provide these details and, preferably, the code used for the experiments.
- [Section 4.2] The trade-data analysis claims an 'identical eigenvalue structure' and 'similar clustering pattern' compared to Hoff [12], but no quantitative comparison (e.g., Procrustes alignment or eigenvector correlation) is provided; the claim is currently based on visual inspection.
Circularity Check
The average-Mahalanobis-distance claim is vacuous because the Mahalanobis statistic is pivotal, forcing identical distributions for every fitted covariance; the geometric and ELBO derivations are otherwise self-contained.
-
self definitional
[Section 4.1, simulated data examples (definition of M^(t)); also Abstract's 'superior approximation ... under the average Mahalanobis distance']
"In the joint model, we generate Σ(1),...,Σ(K) ∼ IW (ˆνv, ⊗D i=1 ˆAi) y(t) 1 , . . . , y(t) m ∼ N (0, Σ(t)) M (t) = 1 m Pm i=1 y(t) i Σ−1 y(t) i."
Conditional on the drawn Σ(t), y_i ∼ N(0,Σ(t)), so every quadratic form y_i^T Σ(t)^{-1} y_i is χ²_d with d=∏d_i, independent of Σ(t). Thus M(t) ∼ χ²_d/m exactly, for the unstructured IW, joint, and mean-field distributions alike. The paper's abstract claim of 'superior approximation ... under the average Mahalanobis distance of the data' is therefore not a derived comparison: the metric cancels the fitted covariance by construction. Figure 2's left panel compares the same pivotal statistic to itself; any observed separation is not a function of the variational approximation.
full rationale
The geometric core (Propositions 2.1-2.4) and the ELBO/gradient derivations in Section 3 and the Supplement are derived from first principles, not from the quantities they are claimed to predict; there is no self-citation chain or imported uniqueness theorem. The unsupported geodesic-convexity assertion in Section 2.1 is a proof gap (Wiesel [26] covers ML convexity, not the full ELBO), but it is a correctness risk rather than a circular reduction. The uncontrolled step-size choices in the efficiency comparison are an empirical-control concern, not circularity. The one genuine circular reduction is the Mahalanobis-distance evaluation: because the data are simulated from N(0,Σ(t)) and standardized by the same Σ(t), the statistic is pivotal and identical for every candidate. The paper's self-contained contributions therefore stand, but a central advertised comparison is forced by construction, giving a partial circularity score of 6.
Assumptions & free parameters
free parameters (3)
- Global step size epsilon grid =
epsilon_joint=10^-4.4, epsilon_MF=10^-6 (Figure 1); epsilon_PB=10^-3.5, epsilon_PM in {10^-4.9,...,10^-6} (Figure 3)…
- Prior scale matrix Lambda for the unstructured IW posterior in simulations =
unspecified
- Trade-data prior hyperparameters =
gamma=5, nu=prod(d_i)+2=5402, Psi=C (sample covariance)
assumptions (4)
- domain assumption The full ELBO (including entropy and trace terms) is geodesically convex under the orthogonalized pullback metric.
- standard math Inverse-Wishart expectation identities for log-determinant and inverse.
- standard math Bartlett decomposition for efficient Wishart variate generation.
- domain assumption The affine-invariant metric corresponds to the Fisher information for a multivariate normal.
Cite this review
Pith. "Pith review of Geodesic Variational Bayes for Multiway Covariances." pith.science (2026). https://pith.science/paper/NA6NFRYT
@misc{pith2026250104935,
author = {Pith},
title = {Pith review of: Geodesic Variational Bayes for Multiway Covariances},
year = {2026},
howpublished = {\url{https://pith.science/paper/NA6NFRYT}},
note = {Machine review of arXiv:2501.04935}
}
read the original abstract
This article explores the optimization of variational approximations for posterior covariances of Gaussian multiway arrays. To achieve this, we establish a natural differential geometric optimization framework on the space using the pullback of the affine-invariant metric. In the case of a truly separable covariance, we demonstrate a joint approximation in the multiway space outperforms a mean-field approximation in optimization efficiency and provides a superior approximation to an unstructured Inverse-Wishart posterior under the average Mahalanobis distance of the data while maintaining a multiway interpretation. We moreover establish efficient expressions for the Euclidean and Riemannian gradients in both cases of the joint and mean-field approximation. We end with an analysis of commodity trade data.
Figures
Reference graph
Works this paper leans on
-
[26]
Geodesic convexity and covariance estimation
Ami Wiesel. Geodesic convexity and covariance estimation. IEEE Transactions on Signal Processing, 60(12):6182–6189, 2012. 14
work page 2012
-
[12]
Peter D Hoff. Separable covariance arrays via the Tucker product, with applications to mul- tivariate relational data. Bayesian Analysis, 6(2):179–196, 2011. 2, 3, 5, 22, 25, 29, 30, 36
work page 2011
-
[1]
Florent Bouchard, Arnaud Breloy, Ammar Mian, and Guillaume Ginolhac. On-line Kronecker product structured covariance estimation with Riemannian geometry for t-distributed data. In 2021 29th European Signal Processing Conference (EUSIPCO), pages 856–859. IEEE, 2021. 3
work page 2021
-
[2]
Variational inference for large-scale models of discrete choice
Michael Braun and Jon McAuliffe. Variational inference for large-scale models of discrete choice. Journal of the American Statistical Association , 105(489):324–335, 2010. 34
work page 2010
-
[3]
Morris H DeGroot. Optimal Statistical Decisions . John Wiley & Sons, 2005. 29
work page 2005
-
[4]
Training VAEs Under Structured Residuals
Garoe Dorta, Sara Vicente, Lourdes Agapito, Neill DF Campbell, and Ivor Simpson. Training vaes under structured residuals. arXiv preprint arXiv:1804.01050 , 2018. 32
work page Pith review arXiv 2018
-
[5]
The mle algorithm for the matrix normal distribution
Pierre Dutilleul. The mle algorithm for the matrix normal distribution. Journal of Statistical Computation and Simulation , 64(2):105–123, 1999. 5
work page 1999
-
[6]
Separable factor analysis with applications to mortality data
Bailey K Fosdick and Peter D Hoff. Separable factor analysis with applications to mortality data. The Annals of Applied Statistics , 8(1):120, 2014. 2
work page 2014
Show all 27 references
-
[7]
A tutorial on variational Bayesian inference
Charles W Fox and Stephen J Roberts. A tutorial on variational Bayesian inference. Artificial Intelligence Review, 38:85–95, 2012. 2
2012
-
[8]
Kronecker Products and Matrix Calculus with Applications
Alexander Graham. Kronecker Products and Matrix Calculus with Applications. Courier Dover Publications, 2018. 20
2018
-
[9]
Bayesian tensor regression
Rajarshi Guhaniyogi, Shaan Qamar, and David B Dunson. Bayesian tensor regression. Journal of Machine Learning Research, 18(79):1–31, 2017. 2
2017
-
[10]
On Riemannian optimization over positive definite matrices with the Bures-Wasserstein geometry
Andi Han, Bamdev Mishra, Pratik Kumar Jawanpuria, and Junbin Gao. On Riemannian optimization over positive definite matrices with the Bures-Wasserstein geometry. Advances in Neural Information Processing Systems , 34:8940–8953, 2021. 13 38
2021
-
[11]
Hierarchical multilinear models for multiway data
Peter D Hoff. Hierarchical multilinear models for multiway data. Computational Statistics & Data Analysis, 55(1):530–543, 2011. 2
2011
-
[13]
Log-euclidean metric learning on symmetric positive definite manifold with application to image set clas- sification
Zhiwu Huang, Ruiping Wang, Shiguang Shan, Xianqiu Li, and Xilin Chen. Log-euclidean metric learning on symmetric positive definite manifold with application to image set clas- sification. In International Conference on Machine Learning , pages 720–729. PMLR, 2015. 13
2015
-
[14]
Tensor decompositions and applications
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM Review, 51(3):455–500, 2009. 2, 6, 7
2009
-
[15]
Bartlett decomposition and Wishart distribution
Anant M Kshirsagar. Bartlett decomposition and Wishart distribution. The Annals of Math- ematical Statistics, 30(1):239–241, 1959. 11
1959
-
[16]
Riemannian geometry of symmetric positive definite matrices via cholesky de- composition
Zhenhua Lin. Riemannian geometry of symmetric positive definite matrices via cholesky de- composition. SIAM Journal on Matrix Analysis and Applications , 40(4):1353–1370, 2019. 13, 30
2019
-
[17]
Information geometry and asymptotics for Kronecker covariances
Andrew McCormack and Peter Hoff. Information geometry and asymptotics for Kronecker covariances. arXiv preprint arXiv:2308.02260 , 2023. 2
2023 arXiv
-
[18]
Symmetric positive-definite matrices: From geome- try to applications and visualization
Maher Moakher and Philipp G Batchelor. Symmetric positive-definite matrices: From geome- try to applications and visualization. In Visualization and Processing of Tensor Fields , pages 285–298. Springer, 2006. 14
2006
-
[19]
The Riemannian geometry of the space of positive-definite matrices and its application to the regularization of positive-definite matrix-valued data
Maher Moakher and Mourad Z´ era ¨ ı. The Riemannian geometry of the space of positive-definite matrices and its application to the regularization of positive-definite matrix-valued data. Jour- nal of Mathematical Imaging and Vision , 40(2):171–187, 2011. 14, 15
2011
-
[20]
Tensor-train decomposition
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing , 33(5):2295–2317, 2011. 2 39
2011
-
[21]
Structured variational Bayesian infer- ence for gaussian state-space models with regime switching
Yohan Petetin, Yazid Janati, and Fran¸ cois Desbouvries. Structured variational Bayesian infer- ence for gaussian state-space models with regime switching. IEEE Signal Processing Letters , 28:1953–1957, 2021. 2
1953
-
[22]
N. P. Pitsanis. The Kronecker product in approximation and fast transform generation . PhD thesis, Cornell University, Ithaca, NY, 1997. 32
1997
-
[23]
Scalable Bayesian low-rank decomposition of incomplete multiway tensors
Piyush Rai, Yingjian Wang, Shengbo Guo, Gary Chen, David Dunson, and Lawrence Carin. Scalable Bayesian low-rank decomposition of incomplete multiway tensors. In International Conference on Machine Learning , pages 1800–1808. PMLR, 2014. 2
2014
-
[24]
Variational Bayes on manifolds.Statistics and Computing , 31:1–17, 2021
Minh-Ngoc Tran, Dang H Nguyen, and Duy Nguyen. Variational Bayes on manifolds.Statistics and Computing , 31:1–17, 2021. 2
2021
-
[25]
Structured variational inference in Bayesian state-space models
Honggang Wang, Anirban Bhattacharya, Debdeep Pati, and Yun Yang. Structured variational inference in Bayesian state-space models. In International Conference on Artificial Intelligence and Statistics , pages 8884–8905. PMLR, 2022. 2
2022
-
[27]
Bayesian nonparametric models for multiway data analysis
Zenglin Xu, Feng Yan, and Yuan Qi. Bayesian nonparametric models for multiway data analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence , 37(2):475–487,
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.