Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

qNBO: quasi-Newton Meets Bilevel Optimization

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read qNBO uses a single BFGS recursion both to solve the lower-level problem and to estimate the inverse-Hessian-vector product, proving that the average squared hypergradient norm decays as $\mathcal{O}(\kappa^3 \ln K / K)$, so an…

desk verdict Useful algorithm and a clean quadratic-case proof, but the general-case rate rests on an unproved BFGS telescoping step and should be treated as conjectural. read the letter →

arxiv 2502.01076 v1 pith:L6JGL6NV submitted 2025-02-03 cs.LG math.OC

Sheng Fang , Yong-Jin Liu , Wei Yao , Chengming Yu , Jin Zhang This is my paper · ORCID
classification cs.LGmath.OC MSC 90C5390C26
keywords bileveloptimizationquasi-NewtonmethodsBFGShypergradientnon-asymptoticconvergenceinverseHessianapproximationhyperparametermeta-learning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces qNBO, a framework that attacks the two bottlenecks of bilevel optimization with one quasi-Newton engine: reaching the lower-level minimizer $y^*(x)$ and forming the inverse-Hessian-vector product $[\nabla^2_{yy} f]^{-1}\nabla_y F$ that the hypergradient needs. For the BFGS variant, the paper proves that under standard strong-convexity and smoothness assumptions, the running average of squared hypergradient norms over $K$ outer steps is $\mathcal{O}(\kappa^3 \ln K / K)$, equivalently $\widetilde{\mathcal{O}}(\kappa^3/\epsilon)$ outer iterations to an $\epsilon$-stationary point. This is the first non-asymptotic guarantee on the iterates themselves for a quasi-Newton hypergradient method, going beyond earlier work that only showed asymptotic consistency of the hypergradient estimate. The paper also validates the method on toy problems, hyperparameter optimization, data hyper-cleaning, and few-shot meta-learning.

What carries the argument

The workhorse is the BFGS two-loop recursion (Algorithm 4), which applies an inverse Hessian approximation $H_t$ to a vector without ever storing or inverting a matrix. qNBO runs this recursion in two roles: in subroutine A it produces quasi-Newton steps for the lower-level problem, and in subroutine B it produces the direction-adapted vector $u_k$ approximating $[\nabla^2_{yy} f]^{-1}\nabla_y F$, avoiding the incorrect inversion that occurs when sharing a lower-level secant history directly with the hypergradient direction. The proof leans on the superlinear BFGS convergence lemma (Lemma D.5), which gives a $(1/T)^T$ contraction factor per inner sweep, and on a potential-function inequality (Lemma D.11) that bounds the accumulated error of the $Q_k$ direction-adapted quasi-Newton steps. The schedule $Q_k = k+1$ is what turns the residual sum $\sum_k 1/Q_k$ into the harmless $\ln K$ factor.

What would settle it

Run qNBO(BFGS) on the quadratic problem (8) with $n=1000$ and $Q_k=k+1$, recording the running average $\frac{1}{K}\sum_{k=0}^{K-1}\|\nabla\Phi(x_k)\|^2$; Theorem 3.4 predicts $\mathcal{O}(\ln K/K)$, so observing a plateau or a slower rate after the transient would falsify the bound. For the general setting, monitor $(1-\beta\mu)^P\|y_k-y^*_k\|$ at each outer step: if it ever exceeds $1/(300\sqrt{\mu})$, the superlinear lemma on which the theorem rests no longer applies.

Watch

Extended reading notes

Core claim

The central claim is that the two traditionally separate computations in bilevel optimization—solving the lower-level problem and inverting its Hessian along the upper-level gradient—can be carried by a single quasi-Newton recursion without losing convergence guarantees. For qNBO(BFGS), the proof tracks two coupled errors: how far the inner iterate $y_{k,T}$ lies from $y^*(x_k)$, and how far the auxiliary vector $u_{k,Q_k}$ lies from $[\nabla^2_{yy} f]^{-1}\nabla_y F$ at the current outer iterate. The superlinear contraction of BFGS makes the first error shrink by a factor $(1/T)^T$ per inner sweep, while choosing $Q_k = k+1$ makes the accumulated $u$-error contribute only $\mathcal{O}(\ln K / K)$. The general-case Theorem 3.7 states the resulting bound explicitly, yielding $\widetilde{\mathcal{O}}(\kappa^3/\epsilon)$ outer iterations and $\widetilde{\mathcal{O}}(\kappa^3/\epsilon)$ Jacobian-vector products to reach an $\epsilon$-stationary point; the quadratic-case Theorem 3.4 gives an analogous rate for SR1 as well.

Load-bearing premise

The proof requires every outer iterate to land inside a fixed-radius basin where $(1-\beta\mu)^P\|y_k-y^*_k\| \le 1/(300\sqrt{\mu})$ and the initial inverse-Hessian approximation is within $1/7$ of the true lower-level Hessian, a condition the paper assumes rather than derives from the stated smoothness assumptions.

Editorial extensions

If this is right

  • qNBO(BFGS) reaches an $\epsilon$-stationary point in $\widetilde{\mathcal{O}}(\kappa^3/\epsilon)$ outer iterations with only $\widetilde{\mathcal{O}}(\kappa^3/\epsilon)$ Jacobian-vector products, matching the order of the best deterministic bilevel rates while using a quasi-Newton inner solver.
  • The proof bounds the average squared norm of the true hypergradient $\nabla\Phi(x_k)$ directly, not just the distance between an estimated and an exact hypergradient, which is a stronger guarantee than the asymptotic consistency established for earlier quasi-Newton sharing schemes.
  • When the lower-level objective is quadratic, the SR1 variant inherits a similar non-asymptotic rate; for general lower-level objectives the authors note that SR1 needs numerical-stability corrections before it can be guaranteed.
  • In the reported experiments, qNBO(BFGS) reaches target test accuracy on MNIST data hyper-cleaning in less than a tenth of the time of the next-fastest method, and it attains higher few-shot meta-learning accuracy than the compared baseline in substantially less time.
  • The framework is modular: other quasi-Newton updates or L-BFGS can be plugged into the same two-subroutine structure without altering the outer-loop logic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof only needs superlinear contraction plus the inverse-product recursion, so a similar argument likely extends to L-BFGS or stochastic quasi-Newton variants; the paper leaves that analysis open.
  • As written, the guarantee is local in the lower-level variable: if the basin condition $(1-\beta\mu)^P\|y_k-y^*_k\| \le 1/(300\sqrt{\mu})$ fails at any outer step, the proof's superlinear lemma no longer applies, so a globalization strategy such as restarts or trust regions would be needed to make the rate unconditional.
  • The rate's dimension dependence enters only through an $n \ln K / K$ term, suggesting the approach may be attractive for high-dimensional lower-level problems where forming the full Hessian or solving an independent linear system is prohibitive.
  • The specific choice $Q_k = k+1$ is convenient because it makes $\sum 1/Q_k$ logarithmic; any $Q_k$ with a divergent harmonic sum would likely give the same rate up to constants, so the proof does not single out a unique schedule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes qNBO, a bilevel optimization framework that uses quasi-Newton recursions both for the lower-level solve and for approximating the inverse-Hessian-vector product in the hypergradient. Two instantiations are given (BFGS and SR1). For the BFGS variant, the paper claims a non-asymptotic stationary-point convergence rate O(κ³ ln K / K) for the general strongly convex lower-level case (Theorem 3.7) and a similar rate for the quadratic case (Theorem 3.4). The proof decomposes the hypergradient error into lower-level and u-subroutine errors and uses BFGS superlinear convergence lemmas. Numerical experiments cover toy problems, hyperparameter optimization, data hyper-cleaning, and few-shot meta-learning.

Significance. The algorithmic idea is natural and the experiments are broad; if the general-case rate were established, it would be a meaningful step beyond SHINE's asymptotic hypergradient consistency. The quadratic-case result is more solid because the Hessian is constant. The paper imports standard BFGS lemmas, and no parameter is fitted to the target result; these are strengths. However, the general-case theorem currently rests on an unproved lemma and an unverified initialization condition, and both theorems rely on an argmin selection that Algorithm 3 does not implement.

major comments (3)
  1. [Appendix D.4, Lemma D.22 (inequality (62))] The proof of the key bound (62) is not a derivation. It invokes Lemma D.11, whose hypotheses include the sequential-iterate condition y_{i+1}=y_i+s_i and the explicit bound max_i ξ_{i+1} ≤ 2. In Algorithm 3, all steps s_i = ζ_i u_i emanate from the fixed base point y_{k+1}, so the virtual points do not chain, and the paper never specifies ζ_i so that ξ_{i+1} ≤ 2. The statement 'When the step size ζ_i is chosen appropriately... it can be concluded that J_i is nearly equal to J_{i+1}, i.e., Δ_i ≈ 0' replaces a needed quantitative estimate and is not demonstrated. Since (67) and therefore the O(ln K/K) term in (7) are the only place the u-subroutine error is controlled, the general-case non-asymptotic rate of Theorem 3.7 is not supported.
  2. [Theorems 3.7 and D.24, Lemma D.15] The local initialization condition (1−βμ)^P ∥y_k − y*_k∥ ≤ 1/(300√μ) and the closeness condition on H0 are stated as choices of β, P and H0, but the first involves the iterate y_k and the second involves the Hessian at (x_k, y*(x_k)), which changes with k. The proof invokes Lemma D.5 at every outer iteration via Lemma D.15 without showing that these conditions hold along the trajectory; no inductive argument is given. As stated, the theorem is conditional on unverified properties of the iterates, so the assumptions are not derived from Assumptions 3.1–3.3.
  3. [Algorithms 1–3 vs. Lemmas D.18/D.23] Algorithm 3 returns u_{Q-1}, the last quasi-Newton iterate, while Lemmas D.18 and D.23 analyze u_{k,Q_k} = \bar{u}_k = argmin_i ∥∇²_{yy}f(x_k,y_{k+1})^{-1}∇_yF(x_k,y_{k+1}) − u_{k,i}∥. The proofs of Theorems 3.4 and 3.7 use (47) and (67), which require the argmin property. Thus the proven rates apply to a variant of qNBO that returns the best of the Q_k iterates, not to the algorithm as specified. Either modify Algorithm 3 (and the experimental implementation) to store and return the best iterate, or extend the proofs to bound the last iterate.
minor comments (5)
  1. [Section 1 heading] The heading 'Introductions' should be 'Introduction'.
  2. [Figure 1 and Section 4.1] The sentence 'As depicted in Figure 1(d) shows that the hypergradient...' is grammatically awkward; consider rewriting.
  3. [Lemma D.11 vs Lemma D.22] The symbol \tildeξ is used with two different definitions: in Lemma D.11 it is 1/(2(\barξ²+\barξ)) with \barξ = max_i ξ_{i+1}, while in Lemma D.22 it is min_i 1/(2(ξ_i²+ξ_i)). These conflict and should be renamed to avoid ambiguity.
  4. [Theorem 3.4 statement] The constants in the main-text statement of Theorem 3.4 (ω involving c1 and the ln K term without M²_{fxy}) differ in appearance from the full statement in Theorem D.19 (where explicit 6(...) terms and M²_{fxy} appear). Please align the statements or explain the notational compression.
  5. [Theorem 3.7, equation (7)] The rate in (7) depends on \tildeξ, which is defined through the unspecified step sizes ζ_i in Algorithm 3. To make the claimed O(κ³ ln K/K) rate explicit, the paper should either give a uniform lower bound on \tildeξ or state explicit constraints on ζ_i.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the only flagged issue is an unproved small-step assertion in Lemma D.22, which is a proof gap rather than a circular reduction.

full rationale

The claimed non-asymptotic rate in Theorem 3.7 is not equivalent to its inputs by construction. The proof chains a standard hypergradient error decomposition (Lemma D.13), upper bounds for the lower-level solve error (Lemma D.15, invoking the external BFGS superlinear lemma D.5 from Jin & Mokhtari 2023), and an upper bound for the u-subroutine error (Lemmas D.22-D.23), with the telescoping argument importing Lemma D.11 from Rodomanov & Nesterov. No parameter is fitted to the target stationary-point bound, and Q_k = k+1 is an algorithm setting rather than a value inferred from the conclusion. The cited BFGS theory is external and parameter-free under stated assumptions that do not include the target result, so this is legitimate reuse rather than circularity. The general-case analysis does contain a genuine unproved step: Lemma D.22 asserts that for 'appropriately chosen' step sizes zeta_i, the averaged Hessians J_i and J_{i+1} are nearly equal so that Delta_i is approximately zero, and that 'if the parameter M of function f or zeta_i u_{k,i} is sufficiently small,' J_i approximates the true Hessian; no explicit condition on zeta_i is derived, and Algorithm 3's fixed base point y_{k+1} makes the chaining assumed in Lemma D.11 nontrivial. This is a correctness and rigor gap, not circularity, because the final bound is not assumed as an input and does not reduce to the theorem's assumptions by definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central convergence proof imports BFGS superlinear lemmas as black boxes and adds two ad hoc assumptions: a locality condition on y_k and H0, and an informal sufficiently-small condition in Lemma D.22. The experiments introduce hand-tuned H0 scalings and iteration budgets. No new physical or conceptual entity is postulated.

free parameters (3)
  • H0 scaling = cI with c in {1, 0.1, 0.01} depending on experiment
    Initial inverse Hessian approximation. In theory it is assumed to satisfy a spectral closeness condition that is not verified; in experiments it is tuned per dataset.
  • iteration budgets P, T, Q_k = P=1..9, T=6..47, Q_k=1, 3, or k+1 depending on experiment
    Control the lower-level solve and the hypergradient refinement. Theory sets Q_k=k+1, while the practical algorithms and most experiments use Q_k=1 or 3.
  • step sizes alpha, beta, gamma, zeta_i = alpha=0.1 or 100, beta=0.0001/(j+1) or 0.1, gamma=0.1 or 1, zeta_i=1 or ||u_i||
    Chosen by hand in experiments. The theory requires alpha=Theta(kappa^{-3}) and T=Theta(ln kappa), which do not match the experimental settings.
assumptions (4)
  • domain assumption Assumptions 3.1-3.3: Lipschitz upper-level gradients, bounded grad_y F, strongly convex lower-level objective with Lipschitz Hessians, bounded grad^2_xy f, and bounded-below Phi.
    Standard regularity conditions introduced in Section 3.1 and used throughout the hypergradient error decomposition and the descent lemma.
  • standard math External BFGS superlinear convergence theorems from Rodomanov and Nesterov and Jin and Mokhtari, including Lemma D.2 for quadratic objectives and Lemma D.5 for the local general case.
    Used as black boxes in Lemmas D.15, D.16 and D.17 to bound ||y_{k,T} - y*_k||. The paper does not re-derive these results.
  • ad hoc to paper Local initialization condition in Theorem 3.7: (1-beta mu)^P ||y_k - y*_k|| <= 1/(300 sqrt(mu)) and the H0 Frobenius-norm closeness condition.
    Needed to apply Lemma D.5 at every outer iteration, but it is neither derived from Assumptions 3.1-3.3 nor verified in experiments. It is presented as a parameter choice rather than as an additional assumption.
  • ad hoc to paper In Lemma D.22, the averaged Hessians J_i are effectively treated as equal to nabla^2_yy f(x_k, y_{k+1}) under the informal condition that zeta_i u_{k,i} is sufficiently small.
    This underlies the Q_k error bound (67) and the general theorem. No quantitative condition is stated in Theorem 3.7, so the bound is not rigorously established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of qNBO: quasi-Newton Meets Bilevel Optimization." pith.science (2026). https://pith.science/paper/L6JGL6NV

@misc{pith2026250201076,
  author       = {Pith},
  title        = {Pith review of: qNBO: quasi-Newton Meets Bilevel Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L6JGL6NV}},
  note         = {Machine review of arXiv:2502.01076}
}
read the original abstract

Bilevel optimization, addressing challenges in hierarchical learning tasks, has gained significant interest in machine learning. The practical implementation of the gradient descent method to bilevel optimization encounters computational hurdles, notably the computation of the exact lower-level solution and the inverse Hessian of the lower-level objective. Although these two aspects are inherently connected, existing methods typically handle them separately by solving the lower-level problem and a linear system for the inverse Hessian-vector product. In this paper, we introduce a general framework to address these computational challenges in a coordinated manner. Specifically, we leverage quasi-Newton algorithms to accelerate the resolution of the lower-level problem while efficiently approximating the inverse Hessian-vector product. Furthermore, by exploiting the superlinear convergence properties of BFGS, we establish the non-asymptotic convergence analysis of the BFGS adaptation within our framework. Numerical experiments demonstrate the comparable or superior performance of the proposed algorithms in real-world learning tasks, including hyperparameter optimization, data hyper-cleaning, and few-shot meta-learning.

Figures

Figures reproduced from arXiv: 2502.01076 by the authors.

Figure 1
Figure 1. Numerical results on toy example. (d) Testing results on the impact of the parameter {Qk} K−1 k=0 in qNBO (BFGS) [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Hyperparameter optimization experiments for l2-regularized logistic regression on two datasets (Left: 20News; Right: Real-sim). where (ai , bi) ∈ Dtrain and (a ′ i , b′ i ) ∈ Dval are the training data and validation data respectively, and ℓ(ai , bi , y) := log 1 + exp − biai T y . The LL variable y is the model’s parameter, while the UL variable x refers to the regularization hyperparameter. 8 [PITH_FULL_IMAGE:f… view at source ↗
Figure 3
Figure 3. Data hyper-cleaning results on two datasets. (Left: MNIST; Right: FashionMNIST). 4.3 DATA HYPER-CLEANING This subsection focuses on data hyper-cleaning for the MNIST (Deng, 2012) and FashionMNIST (Xiao et al., 2017) to enhance model accuracy, using a noisy training set Dtrain := {ai , bi} m i=1 and a clean validation set Dval. The objective is to adjust the training data weights to improve performance on Dval. This … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: 5-way 5-shot experiments on two datasets. (Left: [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Numerical results on toy example: Comparison of qNBO (SR1), qNBO (BFGS) and qNBO (BFGS)-ws with other bilevel optimization methods in a toy experiment. Here, “ws” indicates that a warm-start strategy is applied for uk, with Qk = min(k + 1, 60). BOME: The maximum number…
Figure 6
Figure 6. Figure 6: Testing results on the impact of the parameter Qk in qNBO (BFGS) in the toy example. (The qNBO (BFGS) parameters are the same as those in [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Data hyper-cleaning on two datasets. (First row: MNIST; Second row: FashionMNIST. All results are averaged over 10 random trials. The exclusion of BSG1’s performance in this experiment is due to its ineffectiveness in addressing these data hyper-cleaning problems.) qNB…
Figure 8
Figure 8. Figure 8: 5way-5shot on FC100 datasets. Results are averaged over 5 runs, with all algorithms [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Ablation study on the iteration number T of qNBO (BFGS). (The first row illustrates the test accuracy and test loss for MNIST, and the second row shows these values for FashionMNIST.) C.5 ABLATION STUDY C.5.1 TOY EXAMPLE In this subsection, we conduct an ablation study…
Figure 10
Figure 10. Figure 10: Ablation study on the iteration number Q of qNBO (BFGS). (The first row shows the test accuracy and test loss for the MNIST dataset, and the second row shows the same metrics for the FashionMNIST dataset.) D PROOF OF THE RESULTS IN SECTION 3.2 D.1 RESULTS OF QUASI-NEW…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sharper Analysis of Single-Loop Methods for Bilevel Optimization

    cs.LG 2026-07 accept novelty 6.0 of 10

    Decoupled-norm analysis improves single-loop AID to O(κ⁵/K) and shows single-loop ITD's asymptotic error is exactly O(κ²), matching the known lower bound.

Reference graph

Works this paper leans on

68 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Amortized implicit differentiation for stochastic bilevel optimization

    Michael Arbel and Julien Mairal. Amortized implicit differentiation for stochastic bilevel optimization. In International Conference on Learning Representations, 2022

    work page 2022

  3. [3]

    learn2learn: A library for meta-learning research

    S \'e bastien MR Arnold, Praateek Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. learn2learn: A library for meta-learning research. arXiv preprint arXiv:2008.12284, 2020

    arXiv 2008

  4. [4]

    Deep equilibrium models

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in neural information processing systems, 32, 2019

    2019

  5. [5]

    Multiscale deep equilibrium models

    Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. Advances in neural information processing systems, 33: 0 5238--5250, 2020

    work page 2020

  6. [6]

    Quasi-newton methods and their application to function minimisation

    Charles G Broyden. Quasi-newton methods and their application to function minimisation. Mathematics of Computation, 21 0 (99): 0 368--381, 1967

    work page 1967

  7. [7]

    The convergence of a class of double-rank minimization algorithms: 2

    Charles G Broyden. The convergence of a class of double-rank minimization algorithms: 2. the new algorithm. IMA Journal of Applied Mathematics, 6 0 (3): 0 222--231, 1970 a

    work page 1970

  8. [8]

    The convergence of a class of double-rank minimization algorithms 1

    Charles George Broyden. The convergence of a class of double-rank minimization algorithms 1. general considerations. IMA Journal of Applied Mathematics, 6 0 (1): 0 76--90, 1970 b

    work page 1970

Show all 68 references
  1. [9]

    Representations of quasi- N ewton matrices and their use in limited memory methods

    Richard H Byrd, Jorge Nocedal, and Robert B Schnabel. Representations of quasi- N ewton matrices and their use in limited memory methods. Mathematical Programming, 63 0 (1): 0 129--156, 1994

    1994

  2. [10]

    Libsvm: A library for support vector machines

    Chih-Chung Chang and Chih-Jen Lin. Libsvm: A library for support vector machines. URL https://www.csie.ntu.edu.tw/ cjlin/libsvmtools/datasets/. Accessed: 2021-05-06

    2021

  3. [11]

    A single-timescale method for stochastic bilevel optimization

    Tianyi Chen, Yuejiao Sun, Quan Xiao, and Wotao Yin. A single-timescale method for stochastic bilevel optimization. In International Conference on Artificial Intelligence and Statistics, pp.\ 2466--2488. PMLR, 2022

    2022

  4. [12]

    Progressive differentiable architecture search: Bridging the depth gap between search and evaluation

    Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 1294--1303, 2019

    2019

  5. [13]

    A framework for bilevel optimization that enables stochastic and global variance reduction algorithms

    Mathieu Dagr \'e ou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. Advances in Neural Information Processing Systems, 35: 0 26698--26710, 2022

    2022

  6. [14]

    Variable metric method for minimization

    William C Davidon. Variable metric method for minimization. SIAM Journal on Optimization, 1 0 (1): 0 1--17, 1991

    1991

  7. [15]

    The mnist database of handwritten digit images for machine learning research [best of the web]

    Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29 0 (6): 0 141--142, 2012

    2012

  8. [16]

    Meta-learning of neural architectures for few-shot learning

    Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp.\ 12365--12375, 2020

    2020

  9. [17]

    On solving large-scale limited-memory quasi-newton equations

    Jennifer B Erway and Roummel F Marcia. On solving large-scale limited-memory quasi-newton equations. Linear Algebra and its Applications, 515: 0 196--225, 2017

    2017

  10. [18]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp.\ 1126--1135. PMLR, 2017

    2017

  11. [19]

    A new approach to variable metric algorithms

    Roger Fletcher. A new approach to variable metric algorithms. The Computer Journal, 13 0 (3): 0 317--322, 1970

    1970

  12. [20]

    Forward and reverse gradient-based hyperparameter optimization

    Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning, pp.\ 1165--1173. PMLR, 2017

    2017

  13. [21]

    Bilevel programming for hyperparameter optimization and meta-learning

    Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pp.\ 1568--1577. PMLR, 2018

    2018

  14. [22]

    Fixed point networks: Implicit depth models with jacobian-free backprop

    Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. Fixed point networks: Implicit depth models with jacobian-free backprop. arXiv preprint arXiv:2103.12803, 2021

    2021 arXiv

  15. [23]

    Approximation methods for bilevel programming

    Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246, 2018

    2018 arXiv

  16. [24]

    Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems

    Tommaso Giovannelli, Griffin Kent, and Luis Nunes Vicente. Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems. arXiv preprint arXiv:2110.00604, 2021

    2021 arXiv

  17. [25]

    A family of variable-metric methods derived by variational means

    Donald Goldfarb. A family of variable-metric methods derived by variational means. Mathematics of Computation, 24 0 (109): 0 23--26, 1970

    1970

  18. [26]

    Practical quasi-newton methods for training deep neural networks

    Donald Goldfarb, Yi Ren, and Achraf Bahamou. Practical quasi-newton methods for training deep neural networks. Advances in Neural Information Processing Systems, 33: 0 2386--2396, 2020

    2020

  19. [27]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014

    2014

  20. [28]

    Stochastic block BFGS : Squeezing more curvature out of data

    Robert Gower, Donald Goldfarb, and Peter Richt \'a rik. Stochastic block BFGS : Squeezing more curvature out of data. In International Conference on Machine Learning, pp.\ 1869--1878. PMLR, 2016

    2016

  21. [29]

    Optimizing large-scale hyperparameters via automated learning algorithm

    Bin Gu, Guodong Liu, Yanfu Zhang, Xiang Geng, and Heng Huang. Optimizing large-scale hyperparameters via automated learning algorithm. arXiv preprint arXiv:2102.09026, 2021

    2021 arXiv

  22. [30]

    A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic

    Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33 0 (1): 0 147--180, 2023

    2023

  23. [31]

    Bilevel optimization: Convergence analysis and enhanced design

    Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning, pp.\ 4882--4892. PMLR, 2021

    2021

  24. [32]

    Will bilevel optimizers benefit from loops

    Kaiyi Ji, Mingrui Liu, Yingbin Liang, and Lei Ying. Will bilevel optimizers benefit from loops. Advances in Neural Information Processing Systems, 35: 0 3011--3023, 2022

    2022

  25. [33]

    Non-asymptotic superlinear convergence of standard quasi-newton methods

    Qiujiang Jin and Aryan Mokhtari. Non-asymptotic superlinear convergence of standard quasi-newton methods. Mathematical Programming, 200 0 (1): 0 425--473, 2023

    2023

  26. [34]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009

    2009

  27. [35]

    A fully first-order method for stochastic bilevel optimization

    Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert D Nowak. A fully first-order method for stochastic bilevel optimization. In International Conference on Machine Learning, pp.\ 18083--18113. PMLR, 2023

    2023

  28. [36]

    On penalty methods for nonconvex bilevel optimization and first-order stochastic approximation

    Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert D Nowak. On penalty methods for nonconvex bilevel optimization and first-order stochastic approximation. In International Conference on Learning Representations, 2024

    2024

  29. [37]

    Human-level concept learning through probabilistic program induction

    Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350 0 (6266): 0 1332--1338, 2015

    2015

  30. [38]

    Newsweeder: Learning to filter netnews

    Ken Lang. Newsweeder: Learning to filter netnews. In Machine learning proceedings 1995, pp.\ 331--339. Elsevier, 1995

    1995

  31. [39]

    Bome! Bilevel Optimization Made Easy: A simple first-order approach

    Bo Liu, Mao Ye, Stephen Wright, Peter Stone, and Qiang Liu. Bome! Bilevel Optimization Made Easy: A simple first-order approach. Advances in Neural Information Processing Systems, 35: 0 17248--17262, 2022

    2022

  32. [40]

    On the limited memory BFGS method for large scale optimization

    Dong C Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45 0 (1): 0 503--528, 1989

    1989

  33. [41]

    Darts: Differentiable architecture search

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2018

    2018

  34. [42]

    Value-function-based sequential minimization for bi-level optimization

    Risheng Liu, Xuan Liu, Shangzhi Zeng, Jin Zhang, and Yixuan Zhang. Value-function-based sequential minimization for bi-level optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  35. [43]

    On first-order meta-learning algorithms

    Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018

    2018 arXiv

  36. [44]

    Updating quasi-newton matrices with limited storage

    Jorge Nocedal. Updating quasi-newton matrices with limited storage. Mathematics of computation, 35 0 (151): 0 773--782, 1980

    1980

  37. [45]

    Numerical Optimization

    Jorge Nocedal and Stephen Wright. Numerical Optimization. Springer Science & Business Media, 2006

    2006

  38. [46]

    On _p -hyperparameter learning via bilevel nonsmooth optimization

    Takayuki Okuno, Akiko Takeda, Akihiro Kawana, and Motokazu Watanabe. On _p -hyperparameter learning via bilevel nonsmooth optimization. The Journal of Machine Learning Research, 22 0 (1): 0 11093--11139, 2021

    2021

  39. [47]

    Tadam: Task dependent adaptive metric for improved few-shot learning

    Boris Oreshkin, Pau Rodr \' guez L \'o pez, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in Neural Information Processing Systems, 31, 2018

    2018

  40. [48]

    Hyperparameter optimization with approximate gradient

    Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International Conference on Machine Learning, pp.\ 737--746. PMLR, 2016

    2016

  41. [49]

    Connecting generative adversarial networks and actor-critic methods

    David Pfau and Oriol Vinyals. Connecting generative adversarial networks and actor-critic methods. arXiv preprint arXiv:1610.01945, 2016

    2016 arXiv

  42. [50]

    Rapid learning or feature reuse? T owards understanding the effectiveness of maml

    Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? T owards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157, 2019

    1909 arXiv

  43. [51]

    Meta-learning with implicit gradients

    Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in Neural Information Processing Systems, 32, 2019

    2019

  44. [52]

    SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models

    Zaccharie Ramzi, Florian Mannel, Shaojie Bai, Jean-Luc Starck, Philippe Ciuciu, and Thomas Moreau. SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models. In International Conference on Learning Representations, 2022

    2022

  45. [53]

    Greedy quasi-newton methods with explicit superlinear convergence

    Anton Rodomanov and Yurii Nesterov. Greedy quasi-newton methods with explicit superlinear convergence. SIAM Journal on Optimization, 31 0 (1): 0 785--811, 2021 a

    2021

  46. [54]

    New results on superlinear convergence of classical quasi-newton methods

    Anton Rodomanov and Yurii Nesterov. New results on superlinear convergence of classical quasi-newton methods. Journal of Optimization Theory and Applications, 188: 0 744--769, 2021 b

    2021

  47. [55]

    Rates of superlinear convergence for classical quasi-newton methods

    Anton Rodomanov and Yurii Nesterov. Rates of superlinear convergence for classical quasi-newton methods. Mathematical Programming, 194: 0 159--190, 2022

    2022

  48. [56]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115: 0 211--252, 2015

    2015

  49. [57]

    Conditioning of quasi-newton methods for function minimization

    David F Shanno. Conditioning of quasi-newton methods for function minimization. Mathematics of Computation, 24 0 (111): 0 647--656, 1970

    1970

  50. [58]

    On penalty-based bilevel gradient descent method

    Han Shen and Tianyi Chen. On penalty-based bilevel gradient descent method. In International Conference on Machine Learning, pp.\ 30992--31015. PMLR, 2023

    2023

  51. [59]

    Es-maml: Simple hessian-free meta learning

    Xingyou Song, Wenbo Gao, Yuxiang Yang, Krzysztof Choromanski, Aldo Pacchiano, and Yunhao Tang. Es-maml: Simple hessian-free meta learning. In International Conference on Learning Representations, 2019

    2019

  52. [60]

    A constrained optimization approach to bilevel optimization with multiple inner minima

    Daouda Sow, Kaiyi Ji, Ziwei Guan, and Yingbin Liang. A constrained optimization approach to bilevel optimization with multiple inner minima. arXiv preprint arXiv:2203.01123, 2022 a

    2022 arXiv

  53. [61]

    On the convergence theory for hessian-free bilevel algorithms

    Daouda Sow, Kaiyi Ji, and Yingbin Liang. On the convergence theory for hessian-free bilevel algorithms. Advances in Neural Information Processing Systems, 35: 0 4136--4149, 2022 b

    2022

  54. [62]

    Matching networks for one shot learning

    Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29, 2016

    2016

  55. [63]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

    Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  56. [64]

    Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost

    Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost. Advances in Neural Information Processing Systems, 32, 2019

    2019

  57. [65]

    Towards explicit superlinear convergence rate for SR1

    Haishan Ye, Dachao Lin, Xiangyu Chang, and Zhihua Zhang. Towards explicit superlinear convergence rate for SR1 . Mathematical Programming, 199 0 (1): 0 1273--1303, 2023

    2023

  58. [66]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  59. [67]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  60. [68]

    !1A Qa

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.