Fully stochastic trust-region methods with Barzilai-Borwein steplengths

Benedetta Morini; Mahsa Yousefi; Stefania Bellavia

REVIEW 3 major objections 4 minor 1 cited by

Fully stochastic trust-region methods with Barzilai-Borwein steplengths

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fully stochastic trust-region method with Barzilai-Borwein steplengths converges without diminishing step sizes or full-gradient passes.

desk verdict A useful new combination of TRish and stochastic BB stepsizes, but the main theorems overclaim: the proofs establish only limsup bounds, and the stated exact limits are false. read the letter →

arxiv 2412.12180 v2 pith:6UUDLCCF submitted 2024-12-13 math.OC cs.NAmath.NA

Stefania Bellavia , Benedetta Morini , Mahsa Yousefi This is my paper · ORCID

classification math.OCcs.NAmath.NA MSC 65K0590C30

keywords finite-summinimizationstochastictrust-regionmethodsBarzilai-BorweinmethodgradientnonconvexoptimizationPolyak-Lojasiewiczconditionmachinelearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes TRishBB, a class of stochastic trust-region methods for finite-sum minimization that uses stochastic Barzilai-Borwein (BB) steplengths to inject cheap second-order information. Its aim is to show that these steps can be selected by the trust-region-ish rule and still have rigorous convergence guarantees: under smoothness and bounded-variance assumptions, the expected optimality gap contracts to an explicit neighborhood of the optimum under the Polyak-Lojasiewicz condition, and the expected average squared gradient norm converges to an explicit constant for nonconvex objectives. The theorems cover both unbiased and biased stochastic gradients, and they require neither shrinking step sizes nor full gradient evaluations. The authors also give three practical variants and experiments on logistic regression and neural-network classification in which the BB-driven methods match or improve on the first-order TRish baseline and compare well with tuned stochastic quasi-Newton methods.

What carries the argument

The central object is the TRishBB step: at iterate $x_k$, compute a mini-batch gradient $g_k$, set $H_k = \mu_k^{-1}I$ with $\mu_k$ a stochastic BB steplength, and solve the quadratic trust-region subproblem with radius $\Delta_k$ chosen as $\alpha\gamma_1\|g_k\|$, $\alpha$, or $\alpha\gamma_2\|g_k\|$ according to the size of $\|g_k\|$. The solution is either the unconstrained step $-\mu_k g_k$ inside the radius or the normalized step $-\Delta_k g_k/\|g_k\|$ on the boundary, so the cost per iteration stays first-order. The stochastic BB steplength is obtained from secant pairs $(s_k, y_k)$ built from mini-batch gradient differences, with three variants based on the current batch, on a moving average of gradients over $m$ iterations, and on an accumulated Fisher information matrix; the resulting value is clipped to $[\mu_{\min}, \mu_{\max}]$. The proof works by bounding the conditional one-step decrease of $f$ with the $L$-smooth descent inequality and the specially chosen radius rule, then summing the inequalities; the radius rule is what controls the bad event where the stochastic gradient has positive correlation with the true gradient.

What would settle it

Run TRishBB v1 on $f(x)=\tfrac12\|x\|^2$ with exact gradient $g_k=x_k$, taking $\gamma_1=\gamma_2=1$ and $\alpha<1/8$ with $\mu_{\min}\ge 4\alpha/5$. The theorem predicts the expected average squared gradient norm converges to $(16\gamma_1^2/\gamma_2^2 - 1)M_g = 0$, since $M_g=0$; observing a nonzero plateau or divergence would falsify the nonconvex bound. Alternatively, add synthetic unbiased noise with known variance $M_g$ to the same quadratic and check whether the long-run average equals $(16\gamma_1^2/\gamma_2^2 - 1)M_g$ within Monte Carlo error.

Watch

Extended reading notes

Core claim

The central claim is that the trust-region-ish step-selection rule can carry stochastic BB steplengths and still produce provable convergence in expectation. Concretely, when $f$ is $L$-smooth and bounded below and the stochastic gradient $g_k$ is unbiased with variance bounded by $M_g$, Theorem 3.3 states that under condition (31) and the PL condition, $E[f(x_{k+1})]-f^* \to 8\theta_3 M_g/(\gamma_2 c)$, with $\theta_3 = \gamma_1^2/\gamma_2 - \gamma_2/16$ and $c$ the PL constant; Theorem 3.4 states that for nonconvex $f$, under condition (35), the expected average squared gradient norm converges to $(16\gamma_1^2/\gamma_2^2 - 1)M_g$. The biased-gradient results, Theorems 3.1 and 3.2, give analogous limits $\theta_2/(2c\alpha\theta_1)$ and $\theta_2/(\alpha\theta_1)$ under correlation bounds (12)-(13) and parameter condition (18). The practical message is that the algorithm needs no hand-tuned diminishing schedule and no full gradient, and the asymptotic accuracy is limited only by the gradient noise $M_g$ and by the ratio $\gamma_1/\gamma_2$.

Load-bearing premise

The load-bearing premise is that the objective is smooth and bounded below and that the stochastic gradient noise is controlled; the PL results need an additional gradient-dominance condition, and the ReLU-network experiments do not satisfy smoothness.

Editorial extensions

If this is right

For finite-sum training, TRishBB removes the need to tune or shrink step sizes: the BB steplength adapts curvature information, and convergence holds with fixed algorithm parameters.
The nonconvex limit gives an explicit prediction for the stationarity plateau, $(16\gamma_1^2/\gamma_2^2 - 1)M_g$, so a user can trade off mini-batch size (which controls $M_g$) and the $\gamma_1/\gamma_2$ ratio to target a desired accuracy.
Under the PL condition, the method converges linearly to a neighborhood whose radius is proportional to $M_g/c$, so reducing gradient variance directly translates into a tighter optimality gap.
Because the algorithms never evaluate $f$, they remain applicable when objective values are expensive or unavailable, as long as mini-batch gradients are available.
The variants with accumulated BB parameters show low sensitivity to $(\gamma_1, \gamma_2)$ in the reported experiments, suggesting that the additional tuning burden over first-order TRish is mild.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The saturation constant $(16\gamma_1^2/\gamma_2^2 - 1)M_g$ suggests a variance-reduction strategy that the paper does not explore: instead of shrinking the step, one can grow the mini-batch as $\|\nabla f\|$ approaches the noise floor, pushing $M_g$ toward zero and improving the plateau.
If $\gamma_1/\gamma_2$ is chosen close to $1/4$, the predicted nonconvex plateau approaches zero even with nonzero gradient noise; testing this ratio regime on a synthetic quadratic would be a direct check of the theory's practical relevance.
The convergence theorems assume an $L$-smooth, continuously differentiable objective, but the MNIST and CIFAR10 experiments use ReLU networks that are not smooth; a natural extension is to ask whether the same limit formulas survive for nonsmooth losses or whether a smoothed surrogate is required.
The theory treats $M_g$ as a fixed bound, but in practice $M_g$ shrinks as the iterate approaches a minimizer; coupling the BB steplength with an online variance estimate could yield tighter, data-dependent convergence statements.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper combines TRish with stochastic Barzilai-Borwein stepsizes and proves convergence bounds for biased and unbiased gradients without diminishing stepsizes. That combination is new. But Theorems 3.1–3.4 state exact limits that the proofs do not establish, and the stated limits are false. The correct conclusions are one-sided "limsup ≤ constant" bounds.

What is good: the three variants are non-trivial constructions, the numerical comparison with TRish, SGD-BB, and AdaQN is extensive, and the claim that TRishBB v2 is robust across hyperparameter choices is believable. The practical value of the framework is plausible, especially for logistic regression problems.

Soft spots, in order. First, the overclaimed theorems. Inequality (22), (34), and (37) are one-sided; they give limsup ≤ constant, not convergence to the constant. A simple counterexample with f = 0 (average of sin and -sin, batch size 1) satisfies Assumptions 3.1, 3.2, and 3.5, but the expected optimality gap and average squared gradient are identically zero while the stated limits are positive. This is not cosmetic, but it is fixable: restate the theorems as limsup bounds and the proofs go through. Second, the convergence analysis does not use the BB formulas at all; it applies to any step-size sequence in the TRish family. So the theoretical support for the BB-specific update is weak. Third, Assumption 3.1 requires L-smoothness, which the ReLU networks in the experiments violate; the theory covers only the logistic regression cases. Fourth, the numerical results have no error bars, and the hyperparameter sweep is wide, so the performance comparisons are suggestive rather than conclusive.

Verdict: the paper deserves a serious referee, but it needs major revision. The theorems must be corrected to limsup statements, and the authors should either soften the claim that the theory covers the deep learning experiments or add a nonsmooth extension. I would not cite it in its current form, but after the theorem corrections it would be a useful reference for stochastic trust-region methods.

Referee Report

3 major / 4 minor

Summary. This paper proposes TRishBB, a family of stochastic trust-region-ish methods in which the trust-region subproblem uses H_k = mu_k^{-1} I and the steplength mu_k is updated by three stochastic Barzilai-Borwein rules (Algorithms 2-4). The authors prove convergence under biased and unbiased gradient models: for PL objectives they claim convergence of the expected optimality gap to a positive constant proportional to the noise level, and for nonconvex objectives they claim convergence of the average squared gradient norm to a positive constant. The analysis requires neither diminishing step sizes nor full gradient evaluations. Numerical experiments on logistic regression and on small neural networks compare the variants with TRish, SGD-BB, and AdaQN and report accuracy improvements for TRishBB v2.

Significance. The framework is a reasonable and cheap extension of TRish, and the three variants are well-motivated methodologically; the experiments are extensive, include standard datasets, and show that TRishBB v2 and v3 are less sensitive to (alpha,gamma1,gamma2) than TRish and often more accurate. The convergence analysis is nontrivial in that it covers biased gradients and nonconvex objectives. However, the exact-limit theorems are stronger than the proofs support, and the analysis is not specific to the BB formulas. With the statements corrected to limsup upper bounds and the scope of the experimental claims clarified, the paper would be a useful contribution to the stochastic trust-region and spectral-step literature.

major comments (3)

[§3, Theorems 3.1–3.4] The four theorems state exact limits (Eqs. (20), (24), (32), and (37) respectively), but the proofs establish only one-sided limsup upper bounds. In Theorem 3.3, inequality (34) has the form z_{k+1} ≤ (1 − δ)z_k + C with δ = γ2αc/8 and C = αθ3Mg; iterating gives limsup_{k→∞} z_k ≤ C/δ = 8θ3Mg/(γ2c), with no matching lower bound. Similarly, Theorem 3.4 sums (37) to obtain (1/K)Σ E‖∇f(x_k)‖² ≤ (f(x0)−f*)/(Kγ2α/16) + (16γ1²/γ2² −1)Mg, which is a limsup bound, not a limit. The same structure appears in Theorems 3.1 and 3.2 via (22) and (25). This distinction is not cosmetic. Let N=2, f1(x)=sin x, f2(x)=−sin x, with batch size 1. Then f≡0, ∇f≡0, and g_k=±cos(x_k) is unbiased with variance cos²(x_k) ≤ 1, so Assumptions 3.1, 3.2, and 3.5 hold with L=1, c=1, Mg=1. For parameter choices satisfying (31) or (35) (e.g., γ1=2, γ2=1, α small, μmin≥4γ1α/5), the expected optimality gap and the average squared gradient are exactly 0, while the limits claimed in Theorems 3.3 and 3.4 are strictly positive, namely 8θ3Mg/(γ2c) > 0 and (16γ1²/γ2² −1)Mg > 0. The same construction, viewed under Assumption 3.3 with ω=1, M1=Mg, M2=1, refutes the exact-limit forms of Theorems 3.1 and 3.2. The theorems should be restated as limsup inequalities.
[§3, Lemmas 3.1–3.3] The convergence analysis does not use the Barzilai-Borwein updating formulas of Algorithms 2–4. Every displayed argument in Section 3 bounds the step using only μk ≥ μmin (or μmin ≥ 4γ1α/5) and the normalization rule (4); the secant equations defining μ_{k+1} never appear. Consequently Theorems 3.1–3.4 hold for any sequence μk with μk ∈ [μmin, μmax] (and satisfying Assumption 3.4), not specifically for stochastic BB steps. The authors should state this generality explicitly and discuss what theoretical or practical role the BB construction plays; otherwise the claimed connection between the theory and the new BB mechanisms is unsupported.
[§4 and Table C.8] The numerical experiments on MNIST and CIFAR10 use networks with ReLU activations, max pooling, and average pooling (Table C.8). The resulting empirical loss is not continuously differentiable and does not verify the L-smoothness-type inequality in Assumption 3.1, so the convergence theorems do not cover these experiments. The theoretical support in Section 3 applies directly only to the logistic-regression problems (a1a, w1a, cina). The paper should either restrict the claims made for the deep-learning experiments or provide an argument (e.g., smoothing or an extension of the analysis) that covers them; as written, the experiments are heuristic evidence beyond the proven scope.

minor comments (4)

[Algorithm 1, lines 1–2] The input list chooses μmin together with (α,γ1,γ2), but μmax is only introduced in Algorithms 2–4; include μmax in the general algorithm statement or explain why it is not needed there.
[Algorithm 2, line 6] The quotient s_k^T s_k / (s_k^T y_k) can have zero or near-zero denominator when the objective is flat along the step; the text should state how this case is handled in practice, since thresholding with μmax does not remove a division by zero.
[Table 2 and Figure 9] Table 2 labels the a1a panel as 'ala' and Figure 9 spells 'MNSIT' in the caption; these typos should be corrected.
[Section 2.1] The spectral interpretation in (6) is invoked for stochastic gradients, but it requires twice continuous differentiability on the segment, whereas Assumption 3.1 only assumes continuous differentiability and inequality (10); this additional requirement should be stated explicitly.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the TRishBB convergence theorems are derived from stated Assumptions 3.1-3.5 via explicit lemmas, and the self-citations present are background/implementation references, not load-bearing.

full rationale

The derivation chain is self-contained modulo the external TRish framework of [7] and [9]: Lemma 3.1 follows from Assumptions 3.1 and 3.4 using the explicit step formula (7)-(8), Lemma 3.3 follows from Assumptions 3.1 and 3.5 by case analysis on the events C_{i,k}, and Theorems 3.1-3.4 are direct algebraic consequences of inequalities (16), (29), (30), and the PL condition (11). No parameter is fitted to data and then renamed as a prediction: the limiting constants are composed of assumption constants and algorithm parameters, not estimated quantities. The self-citations present ([8], [14], [39]) support background claims, prior TRish results, and implementation details; they are not load-bearing in the convergence proofs, so they do not raise the circularity score. There is a genuine correctness concern, though not a circularity: Theorems 3.2-3.4 state exact limits while the proofs establish only one-sided limsup upper bounds, and the constant-objective example shows the stated exact limits can be false; this is an overclaim of the derived inequality, not a reduction of the conclusion to an input. The numerical experiments on MNIST and CIFAR10 use ReLU networks outside Assumption 3.1's smoothness, but this is a scope limitation, not circular reasoning.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central theoretical claims rest on standard smoothness and noise assumptions plus the TRish normalization constraints. The BB-specific update formulas in Algorithms 2-4 are not used in the convergence proofs, so the ledger's main gap is that the theoretical conclusion is for the generic step-size family, not for the BB rules themselves.

free parameters (8)

alpha = 10^-1 to 10 for binary; 10^-3 to 1 for MNIST/CIFAR10
Base trust-region radius; theory imposes upper bounds (19), (23), (31), (35) and pairs with gamma2 through Assumption 3.4.
gamma1, gamma2 = gamma1 in {4/G, 8/G, 16/G, 32/G}, gamma2 in {1/(2G), 1/G, 2/G} for binary; scaled sets for DL
Step-normalization interval endpoints, set from a one-epoch SG estimate G; conditions gamma1 >= gamma2 and mu_min <= gamma2*alpha are used in proofs.
mu_min, mu_max = 1e-5 and 1e5
BB steplength clipping thresholds; mu_min enters Assumption 3.4 and Lemma 3.1.
G (gradient magnitude estimate) = 0.3477, 0.0887, 0.0497, 0.2517, 1.8375 per dataset
Average stochastic gradient norm from one SG epoch with learning rate l; directly determines the gamma1 and gamma2 grids.
m = 20 for v1; Nb = floor(N/128) = 25, 38, 156, 468, 390 for v2/v3
Cycle length over which the BB pair is accumulated; beta=(m-1)/m and update frequency follow from it.
eta = 0.9
Convex weight in the v2/v3 moving average of the BB parameter.
mF = 100
Limited-memory Fisher matrix size in v3, taken from [34].
mu0 = 1 for all TRishBB variants; 1e-2 or 1e-3 after tuning for SGD-BB baseline
Initial BB step size; affects early iterations and is manually adjusted for the SGD-BB comparison.

assumptions (6)

standard math Spherical trust-region subproblem solution: pk = -mu_k g_k if in radius, else -Delta_k g_k/||g_k|| (Eq. 7)
Used throughout as the model minimizer; classical Cauchy point result for the diagonal model.
domain assumption Assumption 3.1 (Eq. 10): f is continuously differentiable, bounded below and L-smooth
Basis of all descent lemmas; not satisfied by the ReLU networks in Section 4 experiments.
domain assumption Assumption 3.2: PL condition with c > 0 (Eq. 11)
Used in Theorems 3.1 and 3.3 for linear convergence to a noise neighborhood.
domain assumption Assumption 3.3: biased-gradient conditions (12)-(13)
Defines the biased-gradient setting and enters theta1, theta2 and Theorems 3.1 and 3.2.
ad hoc to paper Assumption 3.4: mu_min <= gamma2*alpha
Imposed to make the (gamma1-gamma2)*alpha <= gamma1*alpha-mu_min inequality in Lemma 3.1 hold; a parameter constraint, not an intrinsic property of the problem.
domain assumption Assumption 3.5: unbiased gradients with bounded variance Mg
Used in Lemma 3.3 and Theorems 3.3 and 3.4.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Fully stochastic trust-region methods with Barzilai-Borwein steplengths." pith.science (2026). https://pith.science/paper/6UUDLCCF

@misc{pith2026241212180,
  author       = {Pith},
  title        = {Pith review of: Fully stochastic trust-region methods with Barzilai-Borwein steplengths},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6UUDLCCF}},
  note         = {Machine review of arXiv:2412.12180}
}

read the original abstract

We investigate stochastic gradient methods and stochastic counterparts of the Barzilai-Borwein steplengths and their application to finite-sum minimization problems. Our proposal is based on the Trust-Region-ish (TRish) framework introduced in [F. E. Curtis, K. Scheinberg, R. Shi, {\it A stochastic trust region algorithm based on careful step normalization}, Informs Journal on Optimization, 1, 2019]. The new framework, named TRishBB, aims to enhance the performance of TRish and at reducing the computational cost of the second-order TRish variant. We propose three different methods belonging to the TRishBB framework and present the convergence analysis for possibly nonconvex objective functions, considering biased and unbiased gradient approximations. Our analysis requires neither diminishing step-sizes nor full gradient evaluation. The numerical experiments in machine learning applications demonstrate the effectiveness of applying the Barzilai-Borwein steplength with stochastic gradients and show improved testing accuracy compared to the TRish method.

Figures

Figures reproduced from arXiv: 2412.12180 by the authors.

**Figure 2.** w1a: Average accuracy, (α, γ1, γ2) = (10, γ1, γ2), |Nk| = 64. Top: TRishBB v1, m = 20. Middle: TRishBB v2, m = 38. Bottom: TRishBB v3, m = 38. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_2.png] view at source ↗

**Figure 3.** cina: Average accuracy, (α, γ1, γ2) = (10, γ1, γ2), |Nk| = 64. Top: TRishBB v1, m = 20; Middle: TRishBB v2, m = 156. Bottom: TRishBB v3, m = 156. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Average values of µk with triplets (α, γ1, γ2) along the iterations. Top: a1a, w1a and cina with α = 10. Bottom: MNIST and CIFAR10 with α = 1. 4.3.3. Average testing loss obtained by TRishBB v2 Sections 4.3.1 and 4.3.2 indicate that TRishBB v2 is the overall best varia…

**Figure 5.** Figure 5: MNIST: Testing accuracy, (α, γ1, γ2) = (1, γ1, γ2), |Nk| = 128. Top: TRishBB v1, m = 20. Middle: TRishBB v2, m = 468. Bottom: TRishBB v3, m = 468 28 [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]

**Figure 6.** Figure 6: CIFAR10: Testing accuracy, (α, γ1, γ2) = (1, γ1, γ2), |Nk| = 128. Top: TRishBB v1, m = 20. Middle: TRishBB v2, m = 390. Bottom: TRishBB v3, m = 390. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]

**Figure 7.** Figure 7: Average testing loss of TRishBB v2 vs. number of gradient evaluations (NGE) compared to TRish across five datasets. loss at termination is largely different and in favour of TRishBB v2. 4.3.4. Comparison to other algorithms As discussed in Sections 2.2 and 2.3, TRishBB…

**Figure 8.** Figure 8: Testing average accuracy of TRishBB v3 with (α, γ1, γ2) = (10, γ1, γ2), m ∈ {5, 20, 64, Nb}. Top: a1a, Nb = 25. Middle: w1a, Nb = 38. Bottom: cina, Nb = 156. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_8.png]

**Figure 9.** Figure 9: Testing average accuracy of TRishBB v3 with (α, γ1, γ2) = (1, γ1, γ2), m ∈ {5, 20, 64, Nb}. Top: MNSIT, Nb = 468. Bottom: CIFAR10, Nb = 390. A. Some intermediate results We present some theoretical results required in Appendix B. Lemma A.1. Suppose Assumption 3.1 holds…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Fast Stochastic Second-Order Adagrad for Nonconvex Bound-Constrained Optimization
math.OC 2025-05 conditional novelty 6.0 of 10

ADAGB2 achieves O(epsilon^{-2}) probabilistic iteration complexity for nonconvex bound-constrained stochastic optimization under a directional gradient-noise condition, with a counterexample showing unbiased oracles n...

Reference graph

Works this paper leans on

41 extracted references · 41 canonical work pages · cited by 1 Pith paper

[1]

Barzilai, J

J. Barzilai, J. M. Borwein, Two-point step size gradient methods, IMA Journal of Numerical Analysis 8 (1) (1988) 141–148. 42

work page 1988
[2]

Nocedal, S

J. Nocedal, S. Wright, Lecture notes, Springer, Nature Switze rland, 2006

work page 2006
[3]

Nesterov, Lectures on convex optimization, Springer, New Y ork, NY, 2018

Y. Nesterov, Lectures on convex optimization, Springer, New Y ork, NY, 2018

work page 2018
[4]

Robbins, S Monro, A stochastic approximation method, The An nals of Mathematical Statistics 22 (1951) 1400–407

H. Robbins, S Monro, A stochastic approximation method, The An nals of Mathematical Statistics 22 (1951) 1400–407

work page 1951
[5]

Bottou, Y

L. Bottou, Y. LeCun, Large scale online learning, in: Neural Info rmation Processing Systems, Vol. 16, 2004, pp. 217–224

work page 2004
[6]

Bottou, F

L. Bottou, F. E. Curtis, J. Nocedal, Optimization methods for lar ge- scale machine learning, Siam Review 60 (2) (2018) 223–311

work page 2018
[7]

F. E. Curtis, K. Scheinberg, R. Shi, A stochastic trust region alg orithm based on careful step normalization, Informs Journal on Optimiza tion 1 (3) (2019) 200–220

work page 2019
[8]

Bellavia, B

S. Bellavia, B. Morini, S. Rebegoldi, An investigation of stochastic trust-region based algorithms for ﬁnite-sum minimization, Optimizat ion Methods and Software (2024) 1–30

work page 2024

Show all 41 references

[9]

F. E. Curtis, R. Shi, A fully stochastic second-order trust regio n method, Optimization Methods and Software 37 (3) (2022) 844–877

2022
[10]

Y. Fang, S. Na, M. W. Mahoney, M. Kolar, Fully stochastic trust -region sequential quadratic programming for equality-constrained optim ization problems, SIAM Journal on Optimization 34 (2) (2024) 2007–2037

2024
[11]

C. Tan, S. Ma, Y. Dai, Y. Qian, Barzilai-Borwein step size for stoc hastic gradient descent, in: Neural Information Processing Systems, 2 016
[12]

L. Wang, H. Wu, I. Matveev, Stochastic gradient method with B arzilai- Borwein step for unconstrained nonlinear optimization, Journal of Com- puter and Systems Sciences International 60 (1) (2021) 75–86

2021
[13]

Liang, Y

J. Liang, Y. Xu, C. Bao, Y. Quan, H. Ji, Barzilai-Borwein-based a dap- tive learning rate for deep learning, Pattern Recognition Letters 1 28 (2019) 197–203. 43

2019
[14]

Bellavia, N

S. Bellavia, N. Kreji´ c, N. K. Jerinki´ c, M. Raydan, SLiSeS: Su bsam- pled line search spectral gradient method for ﬁnite sums, Optimizat ion Methods and Software (2024)

2024
[15]

Kreji´ c, N

N. Kreji´ c, N. Krklec Jerinki´ c, Spectral projected gradie nt method for stochastic optimization, Journal of Global Optimization 73 (2018) 5 9– 81

2018
[16]

A. S. Berahas, M. Tak´ aˇ c, A robust multi-batch L-BFGS meth od for machine learning, Optimization Methods and Software 35 (1) (2020) 191–219

2020
[17]

Bollapragada, J

R. Bollapragada, J. Nocedal, D. Mudigere, H.-J. Shi, P. T. P. Tan g, A progressive batching L-BFGS method for machine learning, in: Inte rna- tional Conference on Machine Learning, PMLR, 2018, pp. 620–629

2018
[18]

Gower, D

R. Gower, D. Goldfarb, P. Richt´ arik, Stochastic block BFGS: S queezing more curvature out of data, in: International Conference on Ma chine Learning, PMLR, 2016, pp. 1869–1878

2016
[19]

Mokhtari, A

A. Mokhtari, A. Ribeiro, Global convergence of online limited memo ry BFGS, The Journal of Machine Learning Research 16 (1) (2015) 31 51– 3181

2015
[20]

X. Wang, S. Ma, D. Goldfarb, W. Liu, Stochastic quasi-Newton m ethods for nonconvex stochastic optimization, SIAM Journal on Optimizat ion 27 (2) (2017) 927–956

2017
[21]

J. B. Erway, J. Griﬃn, R. F. Marcia, R. Omheni, Trust-region alg o- rithms for training responses: machine learning methods using indeﬁ - nite Hessian approximations, Optimization Methods and Software 35 (3) (2020) 460–487

2020
[22]

Youseﬁ, ´A

M. Youseﬁ, ´A. Mart ´ ınez, Deep neural networks training by stochastic quasi-newton trust-region methods, Algorithms 16 (10) (2023) 4 90

2023
[23]

Johnson, T

R. Johnson, T. Zhang, Accelerating stochastic gradient desc ent using predictive variance reduction, in: Neural Information Processing Sys- tems, Vol. 26, 2013, pp. 315–323. 44

2013
[24]

Y. Dai, J. Yuan, Y.-X. Yuan, Modiﬁed two-point stepsize gradien t meth- ods for unconstrained optimization, Computational Optimization an d Applications 22 (2002) 103–109

2002
[25]

Raydan, On the Barzilai and Borwein choice of steplength for the gradient method, IMA J

M. Raydan, On the Barzilai and Borwein choice of steplength for the gradient method, IMA J. Numerical Analysis 13 (1993) 321–326

1993
[26]

E. G. Birgin, J. M. Mart ´ ınez, M. Raydan, Spectral projected gradient methods: Review and perspectives, Journal of Statistical Softw are 60 (3) (2014) 1–21

2014
[27]

Di Seraﬁno, V

D. Di Seraﬁno, V. Ruggiero, G. Toraldo, L. Zanni, On the steple ngth selection in gradient methods for unconstrained optimization, Applie d Mathematics and Computation 318 (2018) 176–195

2018
[28]

A. Conn, N. Gould, P. Toint, Trust-region methods, SIAM, Phila delphia, PA, 2000

2000
[29]

N. N. Schraudolph, J. Yu, S. G¨ unter, A stochastic Quasi-Newton method for online convex optimization, in: Artiﬁcial intelligence and statistics , JMLR, 2007, pp. 436–443

2007
[30]

Y. H. Dai, W. W. Hager, K. Schittkowski, H. Zhang, The cyclic Bar zilai- Borwein method for unconstrained optimization, IMA Journal of Nu - merical Analysis 26 (3) (2006) 604–627

2006
[31]

D. P. Kingma, J. Ba, Adam: A method for stochastic optimization , in: 3rd International Conference on Learning Representations, IC LR 2015 - Conference Track Proceedings, 2015

2015
[32]

R. H. Byrd, S. L. Hansen, J. Nocedal, Y. Singer, A stochastic q uasi- Newton method for large-scale optimization, SIAM Journal on Opti- mization 26 (2) (2016) 1008–1031

2016
[33]

Martens, New insights and perspectives on the natural gra dient method, The Journal of Machine Learning Research 21 (1) (2020) 5776– 5851

J. Martens, New insights and perspectives on the natural gra dient method, The Journal of Machine Learning Research 21 (1) (2020) 5776– 5851

2020
[34]

N. S. Keskar, A. S. Berahas, adaQN: An adaptive quasi-newto n algo- rithm for training RNNs, in: Machine Learning and Knowledge Discov- ery in Databases, Springer, 2016, pp. 1–16. 45

2016
[35]

J. D. Faria, R. Assun¸ c˜ ao, F. Murai, Fisher scoring method fo r neural networks optimization, in: International Conference on Data Minin g (SDM), SIAM, 2023, pp. 748–756

2023
[36]

Chang, C.J

C.C. Chang, C.J. Lin, LIBSVM: a library for support vector machines , Transactions on Intelligent Systems and Technology 2 (3) (2011) 1 –27. https://www.csie.ntu.edu.tw/~cjlin/libsvm/

2011
[37]

Deng, The MNIST database of handwritten digit images for machine learning research, IEEE signal processing magazine 29 (6) (2012) 141–142

L. Deng, The MNIST database of handwritten digit images for machine learning research, IEEE signal processing magazine 29 (6) (2012) 141–142. https://www.kaggle.com/datasets/hojjatk/mnist-dataset

2012
[38]

Krizhevsky, G

A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images , Tech. rep., University of Toronto (2009). https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf

2009
[39]

Kreji´ c, N

N. Kreji´ c, N. Krklec Jerinki´ c,´A. Mart ´ ınez, M. Youseﬁ, A non-monotone trust-region method with noisy oracles and additional sampling, Com - putational Optimization and Applications 89 (1) (2024) 247 – 278

2024
[40]

Glorot, Y

X. Glorot, Y. Bengio, Understanding the diﬃculty of training dee p feed- forward neural networks, in: Artiﬁcial intelligence and statistics, JMLR, 2010, pp. 249–256

2010
[41]

Franchini, F

G. Franchini, F. Porta, V. Ruggiero, I. Trombini, L. Zanni, Diago nal barzilai-borwein rules in stochastic gradient-like methods, in: Inter na- tional Conference on Optimization and Learning, Springer, 2023, p p. 21–35. 46

2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Barzilai, J

J. Barzilai, J. M. Borwein, Two-point step size gradient methods, IMA Journal of Numerical Analysis 8 (1) (1988) 141–148. 42

work page 1988

[2] [2]

Nocedal, S

J. Nocedal, S. Wright, Lecture notes, Springer, Nature Switze rland, 2006

work page 2006

[3] [3]

Nesterov, Lectures on convex optimization, Springer, New Y ork, NY, 2018

Y. Nesterov, Lectures on convex optimization, Springer, New Y ork, NY, 2018

work page 2018

[4] [4]

Robbins, S Monro, A stochastic approximation method, The An nals of Mathematical Statistics 22 (1951) 1400–407

H. Robbins, S Monro, A stochastic approximation method, The An nals of Mathematical Statistics 22 (1951) 1400–407

work page 1951

[5] [5]

Bottou, Y

L. Bottou, Y. LeCun, Large scale online learning, in: Neural Info rmation Processing Systems, Vol. 16, 2004, pp. 217–224

work page 2004

[6] [6]

Bottou, F

L. Bottou, F. E. Curtis, J. Nocedal, Optimization methods for lar ge- scale machine learning, Siam Review 60 (2) (2018) 223–311

work page 2018

[7] [7]

F. E. Curtis, K. Scheinberg, R. Shi, A stochastic trust region alg orithm based on careful step normalization, Informs Journal on Optimiza tion 1 (3) (2019) 200–220

work page 2019

[8] [8]

Bellavia, B

S. Bellavia, B. Morini, S. Rebegoldi, An investigation of stochastic trust-region based algorithms for ﬁnite-sum minimization, Optimizat ion Methods and Software (2024) 1–30

work page 2024