Online Bootstrap Inference with Nonconvex Stochastic Gradient Descent Estimator
Reviewed by Pithpith:ZAGFOA2Jopen to challenge →
read the original abstract
In this paper, we investigate the theoretical properties of stochastic gradient descent (SGD) for statistical inference in the context of nonconvex optimization problems, which have been relatively unexplored compared to convex settings. Our study is the first to establish provable inferential procedures using the SGD estimator for general nonconvex objective functions, which may contain multiple local minima. We propose two novel online inferential procedures that combine SGD and the multiplier bootstrap technique. The first procedure employs a consistent covariance matrix estimator, and we establish its error convergence rate. The second procedure approximates the limit distribution using bootstrap SGD estimators, yielding asymptotically valid bootstrap confidence intervals. We validate the effectiveness of both approaches through numerical experiments. Furthermore, our analysis yields an intermediate result: the in-expectation error convergence rate for the original SGD estimator in nonconvex settings, which is comparable to existing results for convex problems. We believe this novel finding holds independent interest and enriches the literature on optimization and statistical inference.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
Gaussian Approximation and Multiplier Bootstrap for Federated Linear Stochastic Approximation
Establishes non-asymptotic Gaussian approximation bounds for federated LSA with explicit communication-heterogeneity trade-offs and introduces an online multiplier bootstrap for last-iterate inference with validity gu...
-
Online Covariance Matrix Estimation in Sketched Newton Methods
Develops a batch-free online covariance estimator for sketched Newton methods, proves its consistency, and demonstrates use for online statistical inference on regression and CUTEst problems.
-
Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction
A novel bias-reduced online covariance estimator for SGD achieves convergence rate n to the power (α-1)/2 times square root of log n without second-order derivatives.
-
Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descent
Proves the first fully non-asymptotic bound on the accuracy of multiplier bootstrap for constructing confidence sets from Polyak-Ruppert SGD iterates, achieving convex-distance rates up to 1/sqrt(n) under regularity c...
-
Statistical Inference for Stochastic Gradient Descent Beyond Finite Variance
Presents a self-normalized subsampling procedure for asymptotically valid confidence regions from SGD iterates under both finite and infinite variance assumptions.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.