REVIEW 4 major objections 7 minor 39 references
Fisher Information based Stochastic Gradient Ascent for Online Learning of Dirichlet Process Mixture and Theory
T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A Fisher-information step size makes stochastic Bayesian clustering as accurate as closed-form inference.
desk verdict A useful adaptive-gradient heuristic for DPM variational inference, but the Fisher-information justification does not hold up; deserves revision, not desk rejection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is an empirical Fisher-information scaling of the stochastic gradient. For each variational parameter $\theta$ (cluster mean $\mu_k$ or stick-breaking weight $v_k$), the update replaces the plain gradient step by one divided by $F'_\theta = \frac{1}{M}\sum_{n=1}^M (\nabla_\theta \ln q(\theta))^2$, the mean square of the minibatch gradient; with the diagonal-covariance assumption the Fisher term is a scalar per dimension, so the cost is just a per-coordinate division. This scaling does the work of adapting the stepsize to the local curvature: the derivation from Eq. (12)–(13) treats $F_\theta$ as the metric in a KL-divergence constraint, which is what makes the scaled gradient an approximate natural-gradient step rather than an arbitrary damping factor.
What would settle it
Run SGA+F on a synthetic Dirichlet process Gaussian mixture with a known posterior, record the effective stepsize $F_\theta^{-1}\eta$ for each coordinate, and compare it with the inverse diagonal of the true posterior covariance. If the ratio between the two varies strongly across coordinates, or diverges when the likelihood is flat, the empirical-Fisher metric is not the natural-gradient metric and the adaptive-step claim is refuted.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that Fisher information can be injected into stochastic gradient ascent for variational inference in Dirichlet process mixtures, turning a slow constant-stepsize learner into a fast adaptive one. Concretely, the variational expectations for the cluster means and stick-breaking weights are updated by $E[\theta] = E[\theta]' + F_\theta^{-1} \eta \nabla_\theta \ln q(\theta)$, where $F_\theta = E[(\nabla_\theta \ln q(\theta))^2]$ is the Fisher information of the variational log-posterior, computed empirically over each minibatch with negligible overhead since the gradient is already available. The authors derive this scaling from the KL-divergence constraint that makes $F_\theta$ the Riemannian metric for steepest ascent, and they position it as the first use of Fisher information in stochastic-gradient-ascent variational inference for a Dirichlet process mixture. Their experiments compare against the closed-form maximization–maximization learner and report on-par or better NMI and accuracy on all six datasets, better model selection on several, and lower CPU time, including on Caltech256 and SUN397 with VGG16 features.
Load-bearing premise
The load-bearing premise is that the empirical second moment of the gradient, $F_\theta = E[(\nabla_\theta \ln q(\theta))^2]$, is the right curvature metric for choosing the stepsize; if this scalar scaling does not capture the true curvature of the variational posterior, the adaptive-stepsize claim collapses and the method becomes a heuristic preconditioned gradient.
Editorial extensions
If this is right
- The closed-form requirement for scalable variational inference in Dirichlet process mixtures is removed: any model where $\nabla_\theta \ln q(\theta)$ can be computed, even without an analytic expectation, becomes learnable by SGA+F.
- Adaptive stepsize via Fisher information should converge faster and with less fluctuation than constant-stepsize stochastic gradient ascent on the same objective, as the convergence plots in the paper indicate.
- The method inherits the scalability of minibatch training, so a Dirichlet process mixture can be applied to datasets at the scale of SUN397 (108K images, 397 classes) that batch closed-form learners cannot fully see.
- For the six datasets tested, SGA+F should deliver NMI and accuracy at least as good as the closed-form MM baseline while using less CPU time.
- High-dimensional deep features (4096-dimensional VGG16) are usable because the diagonal-covariance assumption keeps per-iteration cost linear in the feature dimension.
Reading between the lines
- An extension the authors do not pursue: the same empirical-Fisher scaling applies to any exponential-family variational posterior with computable gradients, so topic models and other Bayesian nonparametrics are natural next targets.
- A test the paper does not run: on a synthetic Gaussian DPM with known posterior covariance, SGA+F's effective per-coordinate stepsize should track the inverse posterior variance; if it does not, the natural-gradient interpretation fails.
- Because $F_\theta$ is recomputed from each random minibatch, the adaptive stepsize inherits minibatch noise; the class-balanced minibatches used in all experiments are probably a silent condition for the reported stability.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes stochastic gradient ascent (SGA) updates for variational inference in Dirichlet process mixture models, with two variants: SGA+M (momentum) and SGA+F (Fisher-information-scaled adaptive stepsize). The authors claim that SGA learning is faster than closed-form coordinate ascent (the MM baseline) without sacrificing NMI or accuracy, that the Fisher information automatically provides an optimal stepsize, and that the approach scales to large datasets (Caltech256, SUN397) with deep ConvNet features. The paper reports experiments on six object/scene datasets, including model selection, CPU time, NMI, and accuracy, and compares with several Bayesian nonparametric baselines from the literature.
Significance. If the central claims were established, the paper would offer a practical closed-form-free DPM inference algorithm with adaptive stepsize, applicable to high-dimensional features and datasets with up to 397 classes. The authors provide pseudocode and a convergence diagnostic based on the variational lower bound. However, the theoretical justification for the Fisher-information stepsize is not valid as written, and the experimental protocol contains confounds that prevent the reported improvements from being attributed to the proposed method. With the derivation corrected and the method reframed as a heuristic adaptive-gradient preconditioner, an empirical contribution might remain, but the current manuscript does not support the 'automatic optimal stepsize' claim or the comparative performance claims.
major comments (4)
- [Section 3.3, Eqs. (10)-(13)] The natural-gradient derivation is misapplied. The Fisher information matrix in natural gradient is the expected outer product of the gradient of log q(theta; phi) with respect to the variational parameters phi. Here q is the variational posterior over hidden variables theta = (mu, v), and Eq. (10) defines F_theta as the expectation of squared gradients with respect to theta itself. For a Gaussian q(mu) with fixed variance, the Fisher information with respect to the mean is a constant, whereas Eq. (11) is data-dependent; if q is a point-mass MAP estimate, the Fisher metric is not even well-defined. Consequently, Eq. (13) does not follow from the KL-constrained optimization in Eq. (12), and the abstract's 'automatic optimal stepsize' is not established. At best, Eq. (9) is a diagonal adaptive-gradient preconditioner reminiscent of RMSprop, not a natural-gradient update.
- [Section 5.2, Table 2] The claim that 'all SGAs outperforms MM on all 5 evaluation criteria' is contradicted by Table 2. For Caltech10, the SGA+M model estimate is 24 versus MM's 13.3 (ground truth 10); for Caltech101, SGA+M is 139.7 versus MM's 126.3; for Scene15, SGA+F is 18 versus MM's 17.7. Thus not even the SGA+F variant is uniformly closer to ground truth across all datasets, and the SGA+M variant is often worse than MM. The summary in the text overstates the empirical evidence.
- [Sections 5.2 and 7.2, Table 3] The comparison against the MM baseline is not a fair test of closed-form coordinate ascent. For the larger datasets, MM is restricted to a fixed random subset of about 5K images ('there is a large portion of dataset that MM will never see'), while the SGA methods see the entire dataset through random minibatches. The reported CPU-time and accuracy gains therefore conflate algorithmic advantage with access to more data. In addition, the feature extractor is inconsistent: Section 5.1.4 states VGG16 pretrained on ImageNet, while Section 7.2 states that the proposed method uses VGG16 pretrained on Place205. Table 3 also compares methods with different feature types (SIFT, Fisher vector, AlexNet FC7, VGG16), so the NMI/ACC differences cannot be attributed to the DPM learner.
- [Sections 3.1 and 4.3] The paper's title promises theory, but no convergence theorem is provided for Algorithm 1. Eq. (7) cites a Barzilai-Borwein stepsize for stochastic gradient descent, but that rule is never used in the proposed algorithm. Section 4.3 only checks pointwise concavity of coordinate-wise log-posteriors, which does not establish convergence of the alternating stochastic updates. The paper should either provide a formal convergence analysis or remove the theoretical claim from the title and abstract.
minor comments (7)
- [Abstract and Section 1] There are numerous typos and grammatical errors, including 'As as intermediate approach', 'do not sacrifice performance for speed', 'motiviation', 'datset', 'trunction level', and 'reflex'. The manuscript needs thorough proofreading.
- [Section 5.2 and Algorithm 1] Section 5.2 states that all SGA learners use a fixed learning rate eta = 0.1, while Algorithm 1 line 3(i) gives the initialization range 1e-1 <= eta <= 1e-3. Clarify how eta is chosen and whether any sensitivity analysis was performed.
- [Section 7.5] The text references 'Table 6' when discussing LDPO-A-FC, but the manuscript contains only Table 3; correct the cross-reference.
- [Section 5.1.2] The equations for Accuracy and NMI are typeset incorrectly ('Uinf o', 'Hrefers'), making the definitions hard to read; rewrite them cleanly.
- [Section 3.3] There is a typo 'eqn (121)' in the discussion of Eq. (12), and the statement that F_theta is a scalar under diagonal covariance should be clarified: the diagonal entries of the Fisher matrix are used, not a scalar Fisher information.
- [Tables 2 and 3] No error bars or variance estimates are reported for NMI/ACC despite the statement that experiments were averaged over at least 5 runs; adding standard deviations would improve interpretability.
- [Section 4.5] The pruning threshold T_HR and the re-ordering of E[v_k] are described heuristically; please specify how T_HR was set for each dataset and whether the re-ordering affects model selection.
Circularity Check
No significant circularity: the adaptive stepsize is an input normalization, the comparison baseline is re-run, and the central performance claims are empirical.
full rationale
The paper's load-bearing claims are empirical comparisons against the MM baseline and literature results, not predictions derived from fitted constants. SGA+F's Fisher information F_theta is defined in Eq. (10) as the empirical second moment of the gradient, and Eq. (13) is a standard natural-gradient/KL-constraint result cited from Martens [32]; whether this is the correct curvature metric is a modeling/correctness issue, not circularity, because the scaling is part of the update rule rather than a parameter fitted to the evaluation metric. The MM baseline [28] is a self-citation (shared first author), but Section 5.2 states 'We rerun the experiments for at least 5 times and take their average results for each dataset,' so the comparison does not depend on numbers imported from [28]; the convergence-of-MM quote from [28] supports the baseline, not the proposed SGA+F contribution. No fitted input is renamed as a prediction: minibatch sizes are empirically adjusted, but this is hyperparameter tuning rather than a prediction of NMI/accuracy. The Table 3 feature mismatch (Place205 in Section 7.2 versus ImageNet in Section 5.1.4) is an experimental confound and a correctness concern, not a circular derivation. The SGA+F update resembles RMSprop/Adagrad, which the paper acknowledges, so treating it as a novel Fisher-information derivation is a novelty/correctness concern, not circularity. Overall the derivation chain is self-contained: the gradients in Eqs. (16) and (18) are computed from the model, the Fisher scaling is an input preconditioner, and the empirical claims are tested against external and re-run baselines.
Assumptions & free parameters
free parameters (6)
- Learning rate η =
0.1
- Momentum coefficient α =
0.9
- Minibatch size =
100-300 for Caltech10/Scene15; 1005-2560 for larger sets
- Pruning threshold T_HR =
not specified
- Truncation level K =
50, 100, 200, 500, 1000 (per dataset)
- Prior hyperparameters λ0, m0, σ, a0 =
Not reported; a0 set to N in Algorithm 1
assumptions (5)
- standard math Mean-field factorization q(θ) = ∏_i q(θ_i)
- domain assumption Gaussian mixture with diagonal covariance Σ_k = σ²_k I and constant σ_k = σ
- domain assumption Truncated stick-breaking representation of the Dirichlet process
- standard math Concavity of variational log-posteriors
- domain assumption Ground-truth labels of benchmark datasets are correct
Cite this review
Pith. "Pith review of Fisher Information based Stochastic Gradient Ascent for Online Learning of Dirichlet Process Mixture and Theory." pith.science (2026). https://pith.science/paper/7AGXC4UQ
@misc{pith2026241208951,
author = {Pith},
title = {Pith review of: Fisher Information based Stochastic Gradient Ascent for Online Learning of Dirichlet Process Mixture and Theory},
year = {2026},
howpublished = {\url{https://pith.science/paper/7AGXC4UQ}},
note = {Machine review of arXiv:2412.08951}
}
read the original abstract
Scalable algorithms of posterior approximation allow Bayesian nonparametrics such as Dirichlet process mixture to scale up to larger dataset at fractional cost. Recent algorithms, notably the stochastic variational inference performs local learning from minibatch. The main problem with stochastic variational inference is that it relies on closed form solution. Stochastic gradient ascent is a modern approach to machine learning and is widely deployed in the training of deep neural networks. In this work, we explore using stochastic gradient ascent as a fast algorithm for the posterior approximation of Dirichlet process mixture. However, stochastic gradient ascent alone is not optimal for learning. In order to achieve both speed and performance, we turn our focus to stepsize optimization in stochastic gradient ascent. As as intermediate approach, we first optimize stepsize using the momentum method. Finally, we introduce Fisher information to allow adaptive stepsize in our posterior approximation. In the experiments, we justify that our approach using stochastic gradient ascent do not sacrifice performance for speed when compared to closed form coordinate ascent learning on these datasets. Lastly, our approach is also compatible with deep ConvNet features as well as scalable to large class datasets such as Caltech256 and SUN397.
Figures
Reference graph
Works this paper leans on
- [1]
-
[2]
H. Wu, S. Prasad, Semi-supervised dimensionality reduction of hyperspec- tral imagery using pseudo-labels, Pattern Recognition 74 (2018) 212–224
work page 2018
-
[3]
X. Ye, J. Zhao, Multi-manifold clustering: A graph-constrained deep non- parametric method, Pattern Recognition 93 (2019) 215–227
work page 2019
-
[4]
Z. Ma, P. K. Rana, J. Taghia, M. Flierl, A. Leijon, Bayesian estimation of dirichlet mixture model with variational inference, Pattern Recognition 47 (9) (2014) 3143–3157
work page 2014
-
[5]
W. Fan, N. Bouguila, Variational learning of a dirichlet process of general- ized dirichlet distributions for simultaneous clustering and feature selection, Pattern Recognition 46 (10) (2013) 2754–2769
work page 2013
- [6]
-
[7]
D. M. Blei, M. I. Jordan, et al., Variational inference for dirichlet process mixtures, Bayesian analysis 1 (1) (2006) 121–144. 31
work page 2006
-
[8]
M. D. Hoffman, D. M. Blei, C. Wang, J. W. Paisley, Stochastic variational inference., Journal of Machine Learning Research 14 (1) (2013) 1303–1347
work page 2013
Show all 39 references
-
[9]
Mandt, M
S. Mandt, M. D. Hoffman, D. M. Blei, Stochastic gradient descent as ap- proximate bayesian inference, The Journal of Machine Learning Research 18 (1) (2017) 4873–4907
2017
-
[10]
Ranganath, S
R. Ranganath, S. Gerrish, D. Blei, Black box variational inference, in: Artificial Intelligence and Statistics, 2014, pp. 814–822
2014
-
[11]
Paisley, D
J. Paisley, D. M. Blei, M. I. Jordan, Variational bayesian inference with stochastic search, in: Proceedings of the 29th International Coference on International Conference on Machine Learning, Omnipress, 2012, pp. 1363– 1370
2012
-
[12]
D. P. Kingma, M. Welling, Stochastic gradient vb and the variational auto- encoder, in: Second International Conference on Learning Representations, ICLR, 2014
2014
-
[13]
Welling, Y
M. Welling, Y. W. Teh, Bayesian learning via stochastic gradient langevin dynamics, in: Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 681–688
2011
-
[14]
D. J. Rezende, S. Mohamed, Variational inference with normalizing flows, in: Proceedings of the 32nd International Conference on International Con- ference on Machine Learning-Volume 37, JMLR. org, 2015, pp. 1530–1538
2015
-
[15]
Robbins, S
H. Robbins, S. Monro, A stochastic approximation method, in: Herbert Robbins Selected Papers, Springer, 1985, pp. 102–109
1985
-
[16]
C. Tan, S. Ma, Y.-H. Dai, Y. Qian, Barzilai-borwein step size for stochastic gradient descent, in: Advances in Neural Information Processing Systems, 2016, pp. 685–693. 32
2016
-
[17]
Honkela, T
A. Honkela, T. Raiko, M. Kuusela, M. Tornio, J. Karhunen, Approximate riemannian conjugate gradient learning for fixed-form variational bayes, Journal of Machine Learning Research 11 (Nov) (2010) 3235–3268
2010
-
[18]
Duchi, E
J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization, Journal of Machine Learning Re- search 12 (Jul) (2011) 2121–2159
2011
-
[19]
D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014)
2014 arXiv
-
[20]
Kulis, M
B. Kulis, M. I. Jordan, Revisiting k-means: New algorithms via bayesian nonparametrics, in: Proceedings of the 29th International Conference on Machine Learning (ICML-12), 2012, pp. 513–520
2012
-
[21]
Broderick, N
T. Broderick, N. Boyd, A. Wibisono, A. C. Wilson, M. I. Jordan, Streaming variational bayes, in: Advances in neural information processing systems, 2013, pp. 1727–1735
2013
-
[22]
Kurihara, M
K. Kurihara, M. Welling, Bayesian k-means as a maximization-expectation algorithm, Neural computation 21 (4) (2009) 1145–1172
2009
-
[23]
Sato, Online model selection based on the variational bayes, Neural computation 13 (7) (2001) 1649–1681
M.-A. Sato, Online model selection based on the variational bayes, Neural computation 13 (7) (2001) 1649–1681
2001
-
[24]
W. Fan, H. Sallay, N. Bouguila, Online learning of hierarchical pitman–yor process mixture of generalized dirichlet distributions with feature selection, IEEE transactions on neural networks and learning systems 28 (9) (2016) 2048–2061
2016
-
[25]
Patacchiola, A
M. Patacchiola, A. Cangelosi, Head pose estimation in the wild using con- volutional neural networks and adaptive gradient methods, Pattern Recog- nition 71 (2017) 132–143. 33
2017
-
[26]
F. Li, H. Qiao, B. Zhang, Discriminatively boosted image clustering with fully convolutional auto-encoders, Pattern Recognition 83 (2018) 161–173
2018
-
[27]
C. M. Bishop, Pattern recognition and machine learning, springer, 2006
2006
-
[28]
K.-L. Lim, H. Wang, Fast approximation of variational bayes dirichlet process mixture using the maximization–maximization algorithm, Inter- national Journal of Approximate Reasoning 93 (2018) 153–177
2018
-
[29]
R. M. Neal, G. E. Hinton, A view of the em algorithm that justifies in- cremental, sparse, and other variants, in: Learning in graphical models, Springer, 1998, pp. 355–368
1998
-
[30]
D. M. Titterington, The em algorithm, variational approximations and ex- pectationpropagationformixtures, Mixtures: EstimationandApplications 896 (2011)
2011
-
[31]
Honkela, M
A. Honkela, M. Tornio, T. Raiko, J. Karhunen, Natural conjugate gradient in variational inference, in: International Conference on Neural Information Processing, Springer, 2007, pp. 305–314
2007
-
[32]
Martens, New insights and perspectives on the natural gradient method, arXiv preprint arXiv:1412.1193 (2014)
J. Martens, New insights and perspectives on the natural gradient method, arXiv preprint arXiv:1412.1193 (2014)
2014 arXiv
-
[33]
Ç. Arı, S. Aksoy, O. Arıkan, Maximum likelihood estimation of gaussian mixture models using stochastic search, Pattern Recognition 45 (7) (2012) 2804–2816
2012
-
[34]
Kurihara, M
K. Kurihara, M. Welling, Y. W. Teh, Collapsed variational dirichlet process mixture models., in: IJCAI, Vol. 7, 2007, pp. 2796–2801
2007
-
[35]
D. Cai, X. He, J. Han, Document clustering using locality preserving in- dexing, IEEE Transactions on Knowledge and Data Engineering 17 (12) (2005) 1624–1637. 34
2005
-
[36]
C. Chen, J. Zhu, X. Zhang, Robust bayesian max-margin clustering, in: Advances in Neural Information Processing Systems, 2014, pp. 532–540
2014
-
[37]
Nguyen, D
V. Nguyen, D. Phung, T. Le, H. Bui, Discriminative bayesian nonparamet- ric clustering, in: IJCAI 2017: Proceedings of the 26th International Joint Conference on Artificial Intelligence, AAAI Press, 2017, pp. 2550–2556
2017
-
[38]
X. Wang, L. Lu, H.-C. Shin, L. Kim, M. Bagheri, I. Nogues, J. Yao, R. M. Summers, Unsupervised joint mining of deep features and image labels for large-scale radiology image categorization and scene recognition, in: Applications of Computer Vision (WACV), 2017 IEEE Winter Conf...
2017
-
[39]
W. Fan, H. Sallay, N. Bouguila, S. Bourouis, Variational learning of hier- archical infinite generalized dirichlet mixture models and applications, Soft Computing 20 (3) (2016) 979–990. 35
2016
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.