Pith. sign in

REVIEW 2 major objections 2 minor 116 references

Large-scale empirical tuning and comparison of default optimizers for variational inference

T0 review · 2 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No single optimizer dominates black-box variational inference, but five algorithms reliably approach the best observed performance.

desk verdict Solid large-scale benchmark of BBVI optimizers shows five defaults get close to best observed performance, but the selection of those five looks data-driven without clear validation. read the letter →

arxiv 2606.07841 v1 pith:ORJIH4VR submitted 2026-06-05 stat.CO cs.LGstat.ML

Trevor Campbell , Jonathan H. Huggins , Kyurae Kim , Charles C. Margossian This is my paper · ORCID
classification stat.COcs.LGstat.ML
keywords black-boxvariationalinferencestochasticoptimizationadaptiveoptimizersempiricalcomparisonBayesianbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper runs a large empirical comparison of 56 stochastic gradient optimizers on 1092 Bayesian inference problems to test which adaptive methods can serve as tuning-free defaults for black-box variational inference. The study spans problems with dimensions from 1 to 10,000 and condition numbers up to 100 million, using over 550,000 optimization runs. Results show no method wins on every problem, yet a fixed selection of five algorithms consistently reaches performance close to the best observed across the collection. This finding supports treating black-box variational inference as a practical default tool when expert tuning is unavailable and supplies a concrete baseline for developing new optimizers.

What carries the argument

The large-scale empirical benchmark of 56 stochastic gradient-based optimization algorithms evaluated on 1092 Bayesian inference optimization problems with varying dimensions, condition numbers, and variational families.

What would settle it

A new optimizer that consistently outperforms the best of the five selected algorithms on a broad subset of the 1092 problems or on additional problems drawn from the same distribution of dimensions and condition numbers.

Watch

Extended reading notes

Core claim

A benchmark of 56 stochastic optimization algorithms applied to 1092 Bayesian problems demonstrates that no single method dominates, but running a selection of five algorithms suffices to reliably get close to the best-possible observed performance.

Load-bearing premise

The 1092 chosen Bayesian inference problems and the performance metric used are representative of typical real-world black-box variational inference tasks and success criteria.

Editorial extensions

If this is right

  • A practitioner can obtain reliable black-box variational inference results by running five default optimizers in parallel without any problem-specific tuning.
  • New stochastic optimization algorithms for variational inference can be evaluated against the performance distribution established by the five-algorithm selection.
  • Black-box variational inference becomes more usable in applications where expert tuning is not feasible.
  • The study supplies a standardized set of problems for future comparisons of optimization methods in this domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Portfolio or switching strategies that try several of the five algorithms may further improve reliability without increasing the total number of runs.
  • The same five-algorithm approach could be tested on other stochastic optimization tasks outside variational inference to check transferability.
  • Problem characteristics such as dimension or condition number could be used to choose among the five rather than always running all of them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper reports a large-scale empirical benchmark of 56 stochastic gradient optimizers for black-box variational inference (BBVI) across 1092 problems (posterior dimensions 1–10^4, condition numbers 1–10^8), totaling >550k runs and 15 core-years. No single optimizer dominates; the central claim is that a fixed selection of 5 algorithms suffices to reach near the best-observed performance on most problems, providing a practical default baseline when expert tuning is unavailable.

Significance. If the central claim holds after addressing selection methodology, the work supplies a valuable, reproducible empirical baseline for untuned BBVI and a reference point for new optimizer development. The scale and breadth of the problem set are genuine strengths for an empirical study in this area.

major comments (2)
  1. [Results (5-algorithm selection)] Results section (around the identification of the 5-algorithm subset): the headline claim that 'running a selection of 5 algorithms suffices to reliably get close to the best-possible observed performance' requires an explicit statement of the selection procedure. If the 5 were chosen by inspecting performance patterns across the entire 1092-problem collection without cross-validation, held-out problems, or a pre-specified rule, the reported reliability is at risk of being an in-sample observation rather than a generalizable recommendation.
  2. [Methods / benchmark construction] Methods or benchmark-construction section: the paper must detail the criteria used to select the 1092 problems and the performance metric (e.g., how 'best-possible' is computed, whether multiple-comparison corrections were applied, and any statistical testing of differences). Without these, it is difficult to assess whether the problems and metric are representative of typical real-world BBVI tasks, which directly affects the strength of the 'no single method dominates' and '5 suffice' conclusions.
minor comments (2)
  1. [Abstract / Introduction] Clarify in the abstract and introduction whether the 56 algorithms include all recent adaptive methods or a curated subset, and provide a brief justification for any omissions.
  2. [Figures / Tables] Figure captions and tables reporting per-problem or aggregate performance should include error bars or confidence intervals derived from the multiple runs per problem.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments and for recognizing the scale and potential utility of the benchmark. We address each major comment below.

read point-by-point responses

  1. Referee: [Results (5-algorithm selection)] Results section (around the identification of the 5-algorithm subset): the headline claim that 'running a selection of 5 algorithms suffices to reliably get close to the best-possible observed performance' requires an explicit statement of the selection procedure. If the 5 were chosen by inspecting performance patterns across the entire 1092-problem collection without cross-validation, held-out problems, or a pre-specified rule, the reported reliability is at risk of being an in-sample observation rather than a generalizable recommendation.

    Authors: We agree that the selection procedure requires an explicit statement. In the revision we will add a dedicated paragraph in the Results section describing the exact rule used to identify the 5-algorithm subset (top performers across binned regimes of dimension and condition number that together reach within 5% of the best observed ELBO on at least 85% of problems). While the initial identification used the full collection, we will also report performance of the same fixed 5 on a randomly selected held-out collection of 150 problems to provide evidence of out-of-sample behavior. revision: yes

  2. Referee: [Methods / benchmark construction] Methods or benchmark-construction section: the paper must detail the criteria used to select the 1092 problems and the performance metric (e.g., how 'best-possible' is computed, whether multiple-comparison corrections were applied, and any statistical testing of differences). Without these, it is difficult to assess whether the problems and metric are representative of typical real-world BBVI tasks, which directly affects the strength of the 'no single method dominates' and '5 suffice' conclusions.

    Authors: We will expand the Methods section with a new subsection on benchmark construction. This will specify the sampling procedure used to obtain the 1092 problems (stratified coverage of dimension and condition-number ranges), define the performance metric (negative evidence lower bound), state that the best-possible value for each problem is the minimum ELBO attained by any of the 56 optimizers across all random seeds, and report the statistical tests (paired Wilcoxon tests with Bonferroni correction) used to support claims of no single dominant method. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical benchmarking study with no derivation chain or self-referential claims

full rationale

This is a purely empirical paper reporting results from running 56 optimizers on 1092 problems. No mathematical derivations, equations, fitted parameters presented as predictions, ansatzes, or uniqueness theorems appear in the abstract or described content. The statement that a selection of 5 algorithms suffices is an in-sample empirical observation from the benchmark runs, not a derivation that reduces to its own inputs by construction. No self-citations are invoked as load-bearing premises. The paper is self-contained against external benchmarks and receives the default non-circularity finding.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Empirical benchmarking paper; contains no mathematical derivations, free parameters, axioms, or invented entities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large-scale empirical tuning and comparison of default optimizers for variational inference." pith.science (2026). https://pith.science/paper/ORJIH4VR

@misc{pith2026260607841,
  author       = {Pith},
  title        = {Pith review of: Large-scale empirical tuning and comparison of default optimizers for variational inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ORJIH4VR}},
  note         = {Machine review of arXiv:2606.07841}
}
read the original abstract

Black-box variational inference (BBVI) is a methodology for posterior approximation that relies on stochastic optimization. In practice, the stochastic optimizers underpinning BBVI generally require extensive problem-specific tuning, which undermines its promise as a truly "black box" inference algorithm. However, over the past decade, many new adaptive stochastic optimization algorithms have been developed that reduce or remove entirely the need for tuning. In this work, we investigate this new collection of adaptive methods in the context of BBVI, with the goal of establishing the current state of the art in tuning-free optimization-based inference. In particular, we present a large-scale empirical evaluation of 56 stochastic gradient-based optimization algorithms applied to 1092 Bayesian inference optimization problems, involving over 550,000 individual optimization runs and 15 core-years of compute. The optimization algorithms we evaluate are chosen to represent a wide spectrum of recent approaches and the benchmark problems are chosen to span a range of difficulty, with posterior target dimension 1-10^4, condition number 1-10^8, and a range of variational families. Our results show that no single method dominates, but running a selection of 5 algorithms suffices to reliably get close to the best-possible observed performance. We thus provide a strong baseline for applications where expert tuning is not possible and for comparison when developing new stochastic optimization algorithms.

Figures

Figures reproduced from arXiv: 2606.07841 by the authors.

Figure 1
Figure 1. Qualitative characteristics of the problems in the benchmarking suite. Fig. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Scatter plots of the number of completed iterations versus target posterior [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Heatmap of the average rank (in terms of final ELBO objective) of each step [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Heatmap of the probability that no soft failure occurs (including out-of-memory, [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Box plots of squared gradient norm at the 5-minute mark relative to the [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Example trace plots of performance on individual problems. For MAP problems [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Box plots of the 5-minute squared gradient norm of each tuned algorithm, [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Failure probability for each tuned algorithm grouped by objective function [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Box plots of the 5-minute negative ELBO rank for each tuned algorithm, split [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Cumulative distribution functions of the 5-minute negative ELBO rank for all [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Heatmap showing the probability that each tuned algorithm in row [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Trace plots of performance on the held-out challenge problems. For MAP [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Stan code for the original earn_height model [70], followed by Stan code for the subsampled earn_height_subsampled model. Note the additional parameter SUBIDX that is appended to the state, treated as an extra dimension in the BridgeStan input argument, and then used …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

116 extracted references · 6 canonical work pages

  1. [1]

    PosteriorDB.jl: A Julia package to work with posteriordb

    Axen, S. (2026). “PosteriorDB.jl: A Julia package to work with posteriordb.” GitHub Repository: https://github.com/sethaxen/PosteriorDB.jl. Version 0.6.0

    2026

  2. [2]

    Layer Normalization

    Ba, J., Kiros, J., and Hinton, G. (2016). “Layer normalization.”arXiv:1607.06450

    work page Pith review arXiv 2016

  3. [3]

    Online learning rate adaptation with hypergradient descent

    Baydin, A., Cornish, R., Rubio, D., Schmidt, M., and Wood, F. (2018). “Online learning rate adaptation with hypergradient descent.” InProceedings of the International Conference on Learning Representations

    2018

  4. [4]

    Training neural networks for and by interpolation

    Berrada, L., Zisserman, A., and Kumar, M. (2020). “Training neural networks for and by interpolation.” InProceedings of the International Conference on Machine Learning, volume 119 ofPMLR, 799–809

    2020

  5. [5]

    Julia: A fresh approach to numerical computing

    Bezanson, J., Edelman, A., Karpinski, S., and Shah, V. B. (2017). “Julia: A fresh approach to numerical computing.”SIAM review, 59(1): 65–98

    2017

  6. [6]

    Variational inference: A review for statisticians

    Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. (2017). “Variational inference: A review for statisticians.”Journal of the American Statistical Association, 112(518): 859–877

    2017

  7. [7]

    On-line learning and stochastic approximations

    Bottou, L. (1999). “On-line learning and stochastic approximations.” InOn-Line Learning in Neural Networks, 9–42. Cambridge University Press, 1 edition

    1999

  8. [8]

    Optimization methods for large-scale machine learning

    Bottou, L., Curtis, F. E., and Nocedal, J. (2018). “Optimization methods for large-scale machine learning.”SIAM Review, 60(2): 223–311

    2018

Show all 116 references
  1. [9]

    Quasi-Monte Carlo variational inference

    Buchholz, A., Wenzel, F., and Mandt, S. (2018). “Quasi-Monte Carlo variational inference.” InProceedings of the International Conference on Machine Learning, volume 80 ofPMLR, 668–677. JMLR

    2018

  2. [10]

    Importance weighted autoencoders

    Burda, Y., Grosse, R., and Salakhutdinov, R. (2015). “Importance weighted autoencoders.” InProceedings of the International Conference on Learning Repre- sentations

    2015

  3. [11]

    Sample average approximation for black-box variational inference

    Burroni, J., Domke, J., and Sheldon, D. (2024). “Sample average approximation for black-box variational inference.” InProceedings of the Conference on Uncertainty in Artificial Intelligence, volume 244 ofPMLR, 471–498. JMLR

    2024

  4. [12]

    EigenVI: Score-based variational inference with orthogonal function expansions

    Cai, D., Modi, C., Margossian, C., Gower, R., Blei, D., and Saul, L. (2024). “EigenVI: Score-based variational inference with orthogonal function expansions.” InAdvances in Neural Information Processing Systems, 132691–132721. Curran Associates, Inc

    2024

  5. [13]

    Batch and Match: Black-box variational inference with a score-based T. Campbell, J. H. Huggins, K. Kim, C. C. Margossian29 divergence

    Cai, D., Modi, C., Pillaud-Vivien, L., Margossian, C., Gower, R., Blei, D., and Saul, L. (2024). “Batch and Match: Black-box variational inference with a score-based T. Campbell, J. H. Huggins, K. Kim, C. C. Margossian29 divergence.” InProceedings International Conference on M...

    2024

  6. [14]

    Making SGD parameter-free

    Carmon, Y. and Hinder, O. (2022). “Making SGD parameter-free.” InProceedings of the Conference on Learning Theory, volume 178 ofPMLR, 2360–2389

    2022

  7. [15]

    Algorithms for computing the sample variance: Analysis and recommendations

    Chan, T. F., Golub, G. H., and LeVeque, R. J. (1983). “Algorithms for computing the sample variance: Analysis and recommendations.”The American Statistician, 37(3): 242–247

    1983

  8. [16]

    Symbolic discovery of optimization algorithms

    Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., and Le, Q. (2023). “Symbolic discovery of optimization algorithms.” InAdvances in Neural Information Processing Systems, volume 36, 49205–49233. Curran Associates, Inc

    2023

  9. [17]

    Mechanic: A learning rate tuner

    Cutkosky, A., Defazio, A., and Mehta, H. (2023). “Mechanic: A learning rate tuner.” InAdvances in Neural Information Processing Systems, volume 36, 47828–47848. Curran Associates, Inc

    2023

  10. [18]

    The Helmholtz Machine

    Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995). “The Helmholtz Machine.”Neural Computation, 7(5): 889–904

    1995

  11. [19]

    Big Batch SGD: Auto- mated inference using adaptive batch sizes

    De, S., Yadav, A., Jacobs, D., and Goldstein, T. (2017). “Big Batch SGD: Auto- mated inference using adaptive batch sizes.” InProceedings of the International Conference on Artificial Intelligence and Statistics, volume 52 ofPMLR, 1504–1513. JMLR

    2017

  12. [20]

    SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives

    Defazio, A., Bach, F., and Lacoste-Julien, S. (2014). “SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives.” In Advances in Neural Information Processing Systems, volume 27, 1646–1654. Curran Associates, Inc

    2014

  13. [21]

    Learning-rate-free learning by D- adaptation

    Defazio, A. and Mishchenko, K. (2023). “Learning-rate-free learning by D- adaptation.” InProceedings of the International Conference on Machine Learning, volume 202 ofPMLR, 7449–7479. JMLR

    2023

  14. [22]

    Robust, accurate stochastic optimization for variational inference

    Dhaka, A. K., Catalina, A., Andersen, M. R., ns Magnusson, M., Huggins, J., and Vehtari, A. (2020). “Robust, accurate stochastic optimization for variational inference.” InAdvances in Neural Information Processing Systems, volume 33, 10961–10973. Curran Associates, Inc

    2020

  15. [23]

    Challenges and opportunities in high-dimensional variational inference

    Dhaka, A. K., Catalina, A., Welandawe, M., Andersen, M. R., Huggins, J., and Vehtari, A. (2021). “Challenges and opportunities in high-dimensional variational inference.” InAdvances in Neural Information Processing Systems, volume 34, 7787–7798. Curran Associates, Inc

    2021

  16. [24]

    Forward- backward Gaussian variational inference via JKO in the Bures-Wasserstein space

    Diao, M. Z., Balasubramanian, K., Chewi, S., and Salim, A. (2023). “Forward- backward Gaussian variational inference via JKO in the Bures-Wasserstein space.” InProceedings of the International Conference on Machine Learning, volume 202 ofPMLR, 7960–7991. JMLR

    2023

  17. [25]

    Variational 30Default optimizers for variational inference inference viaχ-upper bound minimization

    Dieng, A. B., Tran, D., Ranganath, R., Paisley, J., and Blei, D. (2017). “Variational 30Default optimizers for variational inference inference viaχ-upper bound minimization.” InAdvances in Neural Information Processing Systems, volume 30, 2729–2738. Curran Associates, Inc

    2017

  18. [26]

    bridging the gap between constant step size stochastic gradient descent and Markov chains

    Dieuleveut, A., Durmus, A., and Bach, F. (2020). “bridging the gap between constant step size stochastic gradient descent and Markov chains.”The Annals of Statistics, 48(3): 1348 – 1382

    2020

  19. [27]

    Provable gradient variance guarantees for black-box variational inference

    Domke, J. (2019). “Provable gradient variance guarantees for black-box variational inference.” InAdvances in Neural Information Processing Systems, volume 32, 329–338. Curran Associates, Inc

    2019

  20. [28]

    Provable smoothness guarantees for black-box variational inference

    — (2020). “Provable smoothness guarantees for black-box variational inference.” InProceedings of the International Conference on Machine Learning, volume 119 ofPMLR, 2587–2596. JMLR

    2020

  21. [29]

    Provable convergence guarantees for black-box variational inference

    Domke, J., Gower, R., and Garrigos, G. (2023). “Provable convergence guarantees for black-box variational inference.” InAdvances in Neural Information Processing Systems, volume 36, 66289–66327. Curran Associates, Inc

    2023

  22. [30]

    Importance weighting and variational inference

    Domke, J. and Sheldon, D. R. (2018). “Importance weighting and variational inference.” InAdvances in Neural Information Processing Systems, volume 31, 4470–4479. Curran Associates, Inc

    2018

  23. [31]

    Divide and Couple: using Monte Carlo variational objectives for posterior approximation

    — (2019). “Divide and Couple: using Monte Carlo variational objectives for posterior approximation.” InAdvances in Neural Information Processing Systems, volume 32, 339–349. Curran Associates, Inc

    2019

  24. [32]

    Adaptive subgradient methods for online learning and stochastic optimization

    Duchi, J., Hazan, E., and Singer, Y. (2011). “Adaptive subgradient methods for online learning and stochastic optimization.”Journal of Machine Learning Research, 12: 2121–2159

    2011

  25. [33]

    Batch means and spectral variance estimators in Markov chain Monte Carlo

    Flegal, J. M. and Jones, G. L. (2010). “Batch means and spectral variance estimators in Markov chain Monte Carlo.”The Annals of Statistics, 38(2): 1034– 1070

    2010

  26. [34]

    Multilevel Monte Carlo variational inference

    Fujisawa, M. and Sato, I. (2021). “Multilevel Monte Carlo variational inference.” Journal of Machine Learning Research, 22(278): 1–44

    2021

  27. [35]

    Don’t be so monotone: Relax- ing stochastic line search in over-parametrized models

    Galli, L., Rauhut, H., and Schmidt, M. (2023). “Don’t be so monotone: Relax- ing stochastic line search in over-parametrized models.” InAdvances in Neural Information Processing Systems, volume 36, 34752–34764. Curran Associates, Inc

    2023

  28. [36]

    Empirical evaluation of biased methods for alpha divergence minimization

    Geffner, T. and Domke, J. (2021). “Empirical evaluation of biased methods for alpha divergence minimization.” InProceedings of the Symposium on Advances in Approximate Bayesian Inference

    2021

  29. [37]

    MCMC variational inference via uncorrected Hamiltonian annealing

    — (2021). “MCMC variational inference via uncorrected Hamiltonian annealing.” InAdvances in Neural Information Processing Systems, volume 34, 639–651. Curran Associates, Inc

    2021

  30. [38]

    On the difficulty of unbiased alpha divergence minimization

    — (2021). “On the difficulty of unbiased alpha divergence minimization.” In Proceedings of the International Conference on Machine Learning, volume 139 of PMLR, 3650–3659. JMLR. T. Campbell, J. H. Huggins, K. Kim, C. C. Margossian31

    2021

  31. [39]

    Inference from iterative simulation using multiple sequences

    Gelman, A. and Rubin, D. (1992). “Inference from iterative simulation using multiple sequences.”Statistical Science, 7(4): 457–511

    1992

  32. [40]

    Black box variational inference with a deterministic objective: Faster, more accurate, and even more black box

    Giordano, R., Ingram, M., and Broderick, T. (2024). “Black box variational inference with a deterministic objective: Faster, more accurate, and even more black box.”Journal of Machine Learning Research, 25: 1–39

    2024

  33. [41]

    Practical variational inference for neural networks

    Graves, A. (2011). “Practical variational inference for neural networks.” In Advances in Neural Information Processing Systems, volume 24, 2348–2356. Curran Associates, Inc

    2011

  34. [42]

    Shampoo: Preconditioned stochastic tensor optimization

    Gupta, V., Koren, T., and Singer, Y. (2018). “Shampoo: Preconditioned stochastic tensor optimization.” InProceedings of the International Conference on Machine Learning, volume 80 ofPMLR, 1842–1850. JMLR

    2018

  35. [43]

    Revisiting the Polyak step size

    Hazan, E. and Kakade, S. (2019). “Revisiting the Polyak step size.” arXiv:1905.00313

    2019

  36. [44]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. (2016). “Deep residual learning for image recognition.” InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778

    2016

  37. [45]

    Black-Box alpha divergence minimization

    Hernandez-Lobato, J., Li, Y., Rowland, M., Bui, T., Hernandez-Lobato, D., and Turner, R. (2016). “Black-Box alpha divergence minimization.” InProceedings of the International Conference on Machine Learning, volume 48 ofPMLR, 1511–1520. JMLR

    2016

  38. [46]

    Keeping the neural networks simple by minimizing the description length of the weights

    Hinton, G. E. and van Camp, D. (1993). “Keeping the neural networks simple by minimizing the description length of the weights.” InProceedings of the Annual Conference on Computational Learning Theory, 5–13. ACM Press

    1993

  39. [47]

    Perturbation analysis and optimization of queueing networks

    Ho, Y. C. and Cao, X. (1983). “Perturbation analysis and optimization of queueing networks.”Journal of Optimization Theory and Applications, 40(4): 559–582

    1983

  40. [48]

    Validated variational inference via practical posterior error bounds

    Huggins, J., Kasprzak, M., Campbell, T., and Broderick, T. (2020). “Validated variational inference via practical posterior error bounds.” InProceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 ofPMLR, 1792–1802. JMLR

    2020

  41. [49]

    Variational inference using implicit distributions

    Huszár, F. (2017). “Variational inference using implicit distributions.” arXiv:1702.08235

    2017 arXiv

  42. [50]

    DoG is SGD’s best friend: A parameter-free dynamic step size schedule

    Ivgi, M., Hinder, O., and Carmon, Y. (2023). “DoG is SGD’s best friend: A parameter-free dynamic step size schedule.” InProceedings of the International Conference on Machine Learning, volume 202 ofPMLR, 14465–14499. JMLR

    2023

  43. [51]

    Accelerating stochastic gradient descent using predictive variance reduction

    Johnson, R. and Zhang, T. (2013). “Accelerating stochastic gradient descent using predictive variance reduction.” InAdvances in Neural Information Processing Systems, volume 26, 315–323. Curran Associates, Inc

    2013

  44. [52]

    Muon: An optimizer for hidden layers in neural networks

    Jordan,K.,Jin,Y.,Boza,V., Jiacheng,Y.,Cesista,F.,Newhouse,L.,and Bernstein, J. (2024). “Muon: An optimizer for hidden layers in neural networks.” URLhttps://kellerjordan.github.io/posts/muon/ 32Default optimizers for variational inference

    2024

  45. [53]

    An introduction to variational methods for graphical models

    Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. (1999). “An introduction to variational methods for graphical models.”Machine Learning, 37(2): 183–233

    1999

  46. [54]

    DoWG unleashed: An effi- cient universal parameter-free gradient descent method

    Khaled, A., Mishchenko, K., and Jin, C. (2023). “DoWG unleashed: An effi- cient universal parameter-free gradient descent method.” InAdvances in Neural Information Processing Systems, volume 36, 6748–6769

    2023

  47. [55]

    The Bayesian learning rule

    Khan, M. E. and Rue, H. (2023). “The Bayesian learning rule.”Journal of Machine Learning Research, 24(281): 1–46

    2023

  48. [56]

    Linear convergence of black- box variational inference: Should we stick the landing?

    Kim, K., Ma, Y., and Gardner, J. R. (2024). “Linear convergence of black- box variational inference: Should we stick the landing?” InProceedings of the International Conference on Artificial Intelligence and Statistics, volume 238 of PMLR, 235–243. JMLR

    2024

  49. [57]

    On the convergence of black-box variational inference

    Kim, K., Oh, J., Wu, K., Ma, Y., and Gardner, J. R. (2023). “On the convergence of black-box variational inference.” InAdvances in Neural Information Processing Systems, volume 36, 44615–44657. Curran Associates Inc

    2023

  50. [58]

    A guide to sample average approximation

    Kim, S., Pasupathy, R., and Henderson, S. (2015). “A guide to sample average approximation.” InHandbook of Simulation Optimization, 207–243. Springer

    2015

  51. [59]

    Adam: A method for stochastic optimization

    Kingma, D. and Ba, J. (2015). “Adam: A method for stochastic optimization.” In Proceedings of the International Conference on Learning Representations

    2015

  52. [60]

    Auto-encoding variational Bayes

    Kingma, D. P. and Welling, M. (2014). “Auto-encoding variational Bayes.” In Proceedings of the International Conference on Learning Representations

    2014

  53. [61]

    Automatic differentiation variational inference

    Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., and Blei, D. M. (2017). “Automatic differentiation variational inference.”Journal of Machine Learning Research, 18(14): 1–45

    2017

  54. [62]

    Optimization guarantees for square-root natural-gradient variational inference

    Kumar, N., Möllenhoff, T., Khan, M. E., and Lucchi, A. (2025). “Optimization guarantees for square-root natural-gradient variational inference.”Transactions on Machine Learning Research

    2025

  55. [63]

    Varia- tional inference via Wasserstein gradient flows

    Lambert, M., Chewi, S., Bach, F., Bonnabel, S., and Rigollet, P. (2022). “Varia- tional inference via Wasserstein gradient flows.” InAdvances in Neural Information Processing Systems, volume 35, 14434–14447. Curran Associates, Inc

    2022

  56. [64]

    A stochastic gradient method with an exponential convergence rate for finite training sets

    Le Roux, N., Schmidt, M., and Bach, F. (2012). “A stochastic gradient method with an exponential convergence rate for finite training sets.” InAdvances in Neural Information Processing Systems, 2663–2671. Curran Associates, Inc

    2012

  57. [65]

    Rényi divergence variational inference

    Li, Y. and Turner, R. E. (2016). “Rényi divergence variational inference.” In Advances in Neural Information Processing Systems, volume 29, 1073–1081. Curran Associates, Inc

    2016

  58. [66]

    Fast and simple natural-gradient variational inference with mixture of exponential-family approximations

    Lin, W., Khan, M. E., and Schmidt, M. (2019). “Fast and simple natural-gradient variational inference with mixture of exponential-family approximations.” In Proceedings of the International Conference on Machine Learning, volume 97 of PMLR, 3992–4002. JMLR. T. Campbell, J. H. ...

    2019

  59. [67]

    Batch size selection for variance estimators in MCMC

    Liu, Y., Vats, D., and Flegal, J. M. (2022). “Batch size selection for variance estimators in MCMC.”Methodology and Computing in Applied Probability, 24(1): 65–93

    2022

  60. [68]

    Stochastic Polyak step-size for SGD: An adaptive learning rate for fast convergence

    Loizou, N., Vaswani, S., Laradji, I. H., and Lacoste-Julien, S. (2021). “Stochastic Polyak step-size for SGD: An adaptive learning rate for fast convergence.” In Proceedings of The International Conference on Artificial Intelligence and Statistics, PMLR, 1306–1314. JMLR

    2021

  61. [69]

    The power of interpolation: Un- derstanding the effectiveness of SGD in modern over-parametrized learning

    Ma, S., Bassily, R., and Belkin, M. (2018). “The power of interpolation: Un- derstanding the effectiveness of SGD in modern over-parametrized learning.” In Proceedings of the International Conference on Machine Learning, volume 80 of PMLR, 3325–3334. JMLR

    2018

  62. [70]

    posteriordb: Testing, benchmarking and developing Bayesian inference algorithms

    Magnusson, M., Torgander, J., Bürkner, P.-C., Zhang, L., Carpenter, B., and Vehtari, A. (2025). “posteriordb: Testing, benchmarking and developing Bayesian inference algorithms.” InProceedings of The International Conference on Artificial Intelligence and Statistics, volume 25...

    2025

  63. [71]

    Torsten: A platform for Bayesian inference of pharmacometric models

    Margossian, C. C., Zhang, Y., Gillespie, B., Bales, B., Volfovsky, A., Pavlovic, V., and Gelman, A. (2022). “Torsten: A platform for Bayesian inference of pharmacometric models.”Statistics and Computing, 32(6): 1–15

    2022

  64. [72]

    Expectation propagation for approximate Bayesian inference

    Minka, T. P. (2001). “Expectation propagation for approximate Bayesian inference.” InProceedings of the Conference on Uncertainty in Artificial Intelligence, 362–369. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc

    2001

  65. [73]

    Variational Inference with Gaussian Score Matching

    Modi, C., Gower, R., Margossian, C., Yao, Y., Blei, D., and Saul, L. (2023). “Variational Inference with Gaussian Score Matching.” InAdvances in Neural Information Processing Systems, volume 36, 29935–29950. Curran Associates, Inc

    2023

  66. [74]

    Monte Carlo Gradient Estimation in Machine Learning

    Mohamed, S., Rosca, M., Figurnov, M., and Mnih, A. (2020). “Monte Carlo Gradient Estimation in Machine Learning.”Journal of Machine Learning Research, 21(132): 1–62

    2020

  67. [75]

    Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients

    Moses, W. and Churavy, V. (2020). “Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients.” InAdvances in Neural Information Processing Systems, volume 33, 12472–12485. Curran Associates, Inc

    2020

  68. [76]

    Rectified linear units improve restricted Boltz- mann machines

    Nair, V. and Hinton, G. E. (2010). “Rectified linear units improve restricted Boltz- mann machines.” InProceedings of the International Conference on International Conference on Machine Learning, ICML, 807–814. Madison, WI, USA: Omnipress

    2010

  69. [77]

    Gaussian Variational Approximation with a Factor Covariance Structure

    Ong, V. M.-H., Nott, D. J., and Smith, M. S. (2018). “Gaussian Variational Approximation with a Factor Covariance Structure.”Journal of Computational and Graphical Statistics, 27(3): 465–478

    2018

  70. [78]

    Parameter-free stochastic optimization of variationally coherent functions

    Orabona, F. and Pál, D. (2021). “Parameter-free stochastic optimization of variationally coherent functions.”arXiv:2102.00236

    2021

  71. [79]

    Training deep networks without learning 34Default optimizers for variational inference rates through coin betting

    Orabona, F. and Tommasi, T. (2017). “Training deep networks without learning 34Default optimizers for variational inference rates through coin betting.” InAdvances in Neural Information Processing Systems, volume 30, 2160–2170. Curran Associates, Inc

    2017

  72. [80]

    R package

    outbreak package authors (2024).outbreak: Tools for Simulating and Analyzing Epidemic Outbreaks. R package. URLhttps://CRAN.R-project.org/package=outbreak

    2024

  73. [81]

    Explorations of the mean field theory learning algorithm

    Peterson, C. and Hartman, E. (1989). “Explorations of the mean field theory learning algorithm.”Neural Networks, 2(6): 475–494

    1989

  74. [82]

    On the determination of the step size in stochastic quasigradient methods

    Pflug, G. (1983). “On the determination of the step size in stochastic quasigradient methods.”IIASA Collaborative Paper CP-83-025

    1983

  75. [83]

    Acceleration of stochastic approximation by averaging

    Polyak, B. T. and Juditsky, A. B. (1992). “Acceleration of stochastic approximation by averaging.”SIAM Journal on Control and Optimization, 30(4): 838–855

    1992

  76. [84]

    Blackboxvariationalinference

    Ranganath,R.,Gerrish,S.,andBlei,D.(2014). “Blackboxvariationalinference.” In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 33 ofPMLR, 814–822. JMLR

    2014

  77. [85]

    Hierarchical variational models

    Ranganath, R., Tran, D., and Blei, D. (2016). “Hierarchical variational models.” InProceedings of the International Conference on Machine Learning, volume 48 ofPMLR, 324–333. JMLR

    2016

  78. [86]

    On the convergence of Adam and beyond

    Reddi, S., Kale, S., and Kumar, S. (2018). “On the convergence of Adam and beyond.” InProceedings of the International Conference on Learning Representa- tions

    2018

  79. [87]

    Variational Inference with Normalizing Flows

    Rezende, D. and Mohamed, S. (2015). “Variational Inference with Normalizing Flows.” InProceedings of the International Conference on Machine Learning, volume 37 ofPMLR, 1530–1538. JMLR

    2015

  80. [88]

    Stochastic Backpropaga- tion and Approximate Inference in Deep Generative Models

    Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). “Stochastic Backpropaga- tion and Approximate Inference in Deep Generative Models.” InProceedings of the International Conference on Machine Learning, volume 32 ofPMLR, 1278–1286. JMLR

    2014

  81. [89]

    A stochastic approximation method

    Robbins, H. and Monro, S. (1951). “A stochastic approximation method.”Annals of Mathematical Statistics, 22(3): 400–407

    1951

  82. [90]

    Robert, C. P. and Casella, G. (2004).Monte Carlo Statistical Methods. Springer Texts in Statistics. Springer

    2004

  83. [91]

    Sticking the landing: Simple, lower-variance gradient estimators for variational inference

    Roeder, G., Wu, Y., and Duvenaud, D. K. (2017). “Sticking the landing: Simple, lower-variance gradient estimators for variational inference.” InAdvances in Neural Information Processing Systems, volume 30, 6928–6937. Curran Associates, Inc

    2017

  84. [92]

    BridgeStan: Efficient in-memory access to the methods of a Stan model

    Roualdes, E. A., Ward, B., Carpenter, B., Seyboldt, A., and Axen, S. D. (2023). “BridgeStan: Efficient in-memory access to the methods of a Stan model.”Journal of Open Source Software, 8(87): 5236

    2023

  85. [93]

    Sensitivity analysis of discrete event systems by the “push out

    Rubinstein, R. Y. (1992). “Sensitivity analysis of discrete event systems by the “push out” method.”Annals of Operations Research, 39(1): 229–250. T. Campbell, J. H. Huggins, K. Kim, C. C. Margossian35

    1992

  86. [94]

    Markov Chain Monte Carlo and Variational Inference: Bridging the Gap

    Salimans, T., Kingma, D., and Welling, M. (2015). “Markov Chain Monte Carlo and Variational Inference: Bridging the Gap.” InProceedings of the International Conference on Machine Learning, volume 37 ofPMLR, 1218–1226. JMLR

    2015

  87. [95]

    Mean field theory for sigmoid belief networks

    Saul, L. K., Jaakkola, T., and Jordan, M. I. (1996). “Mean field theory for sigmoid belief networks.”Journal of Artificial Intelligence Research, 4: 61–76

    1996

  88. [96]

    Descending through a crowded valley—Benchmarking deep learning optimizers

    Schmidt, R. M., Schneider, F., and Hennig, P. (2021). “Descending through a crowded valley—Benchmarking deep learning optimizers.” InProceedings of the International Conference on Machine Learning, volume 139 ofPMLR, 9367–9376. JMLR

    2021

  89. [97]

    Stan Reference Manual, v2.38

    Stan Development Team (2026). “Stan Reference Manual, v2.38.”https:// mc-stan.org

    2026

  90. [98]

    Analytic natural gradient updates for Cholesky factor in Gaussian variational approximation

    Tan, L. S. L. (2025). “Analytic natural gradient updates for Cholesky factor in Gaussian variational approximation.”Journal of the Royal Statistical Society Series B: Statistical Methodology, 87(4): 930–956

    2025

  91. [99]

    Gaussian variational approximation with sparse precision matrices

    Tan, L. S. L. and Nott, D. J. (2018). “Gaussian variational approximation with sparse precision matrices.”Statistics and Computing, 28(2): 259–275

    2018

  92. [100]

    Doubly stochastic variational Bayes for non-conjugate inference

    Titsias, M. and Lázaro-Gredilla, M. (2014). “Doubly stochastic variational Bayes for non-conjugate inference.” InProceedings of the International Conference on Machine Learning, volume 32 ofPMLR, 1971–1979. JMLR

    2014

  93. [101]

    Unbiased implicit variational inference

    Titsias, M. K. and Ruiz, F. (2019). “Unbiased implicit variational inference.” In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 89 ofPMLR, 167–176. JMLR

    2019

  94. [102]

    Towards noise- adaptive, problem-adaptive (accelerated) SGD

    Vaswani, S., Dubois-Taine, B., and Babanezhad, R. (2022). “Towards noise- adaptive, problem-adaptive (accelerated) SGD.” InProceedings of the International Conference on Machine Learning, volume 162 ofPMLR, 22015–22059

    2022

  95. [103]

    Adaptive gradient methods converge faster with over-parametrization (but you should do a line-search)

    Vaswani, S., Laradji, I., Kunstner, F., Meng, S. Y., Schmidt, M., and Lacoste-Julien, S. (2020). “Adaptive gradient methods converge faster with over-parametrization (but you should do a line-search).” InOptimization for Machine Learning NeurIPS Workshop

    2020

  96. [104]

    Painless stochastic gradient: Interpolation, line-search, and convergence rates

    Vaswani, S., Mishkin, A., Laradji, I., Schmidt, M., Gidel, G., and Lacoste-Julien, S. (2019). “Painless stochastic gradient: Interpolation, line-search, and convergence rates.” InAdvances in Neural Information Processing Systems, volume 32, 3732–

    2019

  97. [105]

    Curran Associates, Inc

  98. [106]

    Multivariate output analysis for Markov chain Monte Carlo

    Vats, D., Flegal, J. M., and Jones, G. L. (2019). “Multivariate output analysis for Markov chain Monte Carlo.”Biometrika, 106(2): 321–337

    2019

  99. [107]

    Rank-normalization, folding, and localization: An improvedˆR for assessing con- vergence of MCMC (with Discussion)

    Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., and Bürkner, P.-C. (2021). “Rank-normalization, folding, and localization: An improvedˆR for assessing con- vergence of MCMC (with Discussion).”Bayesian Analysis, 16(2): 667–718

    2021

  100. [108]

    VIABEL: Variational inference and approximation 36Default optimizers for variational inference bounds that are efficient and lightweight

    VIABEL Developers (2026). “VIABEL: Variational inference and approximation 36Default optimizers for variational inference bounds that are efficient and lightweight.” GitHub Repository:https://github. com/jhuggins/viabel. Commit 46dd21a

    2026

  101. [109]

    A frame- work for improving the reliability of black-box variational inference

    Welandawe, M., Andersen, M. R., Vehtari, A., and Huggins, J. (2024). “A frame- work for improving the reliability of black-box variational inference.”Journal of Machine Learning Research, 25(219): 1–71

    2024

  102. [110]

    Note on a method for calculating corrected sums of squares and products

    Welford, B. P. (1962). “Note on a method for calculating corrected sums of squares and products.”Technometrics, 4(3): 419–420

    1962

  103. [111]

    Automated variational inference in proba- bilistic programming

    Wingate, D. and Weber, T. (2013). “Automated variational inference in proba- bilistic programming.”arXiv:1301.1299

    2013 arXiv

  104. [112]

    Robust discovery of mutational signatures using power posteriors

    Xue, C., Miller, J. W., Carter, S. L., and Huggins, J. H. (2024). “Robust discovery of mutational signatures using power posteriors.”bioRxiv, 2024.10.23.619958

    2024

  105. [113]

    Yes, but did it work?: Evaluating variational inference

    Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2018). “Yes, but did it work?: Evaluating variational inference.” InProceedings of the International Conference on Machine Learning, volume 80 ofPMLR, 5581–5590. JMLR

    2018

  106. [114]

    Semi-implicit variational inference via score match- ing

    Yu, L. and Zhang, C. (2023). “Semi-implicit variational inference via score match- ing.” InProceedings of the International Conference on Learning Representations

    2023

  107. [115]

    Differentiable an- nealed importance sampling and the perils of gradient noise

    Zhang, G., Hsu, K., Li, J., Finn, C., and Grosse, R. B. (2021). “Differentiable an- nealed importance sampling and the perils of gradient noise.” InAdvances in Neural Information Processing Systems, volume 34, 19398–19410. Curran Associates, Inc

    2021

  108. [116]

    Compressive Bayesian non-negative matrix factorization for mutational signatures analysis

    Zito, A. and Miller, J. W. (2024). “Compressive Bayesian non-negative matrix factorization for mutational signatures analysis.”arXiv:2404.10974. T. Campbell, J. H. Huggins, K. Kim, C. C. Margossian37 Appendix A: Subsampled PosteriorDB As part of our benchmark problem suite, we...

    2024

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.