Pith. sign in

REVIEW 2 major objections 5 minor 299 references

Optimization under Persistent State-Dependent Bias: Gradient-based Method and Complexity Analysis

T0 review · 2 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that exact convergence to the true optimum is achievable despite persistent state-dependent update bias, and that a bilevel 'residual learning' reformulation does it with only a polynomial price in the hardware condition n

desk verdict The Residual Learning idea is genuinely interesting, but the central equivalence theorem is false under the stated assumptions and the convergence proof analyzes the wrong P*; this needs major revision before it can be trusted. read the letter →

arxiv 2607.26032 v1 pith:QCQ7GCRN submitted 2026-07-28 math.OC

Zhaoxian Wu , Quan Xiao , Tayfun Gokmen , Tianyi Chen This is my paper · ORCID
classification math.OC MSC 90C2690C1590C0690C6049M3768Q25
keywords stochasticoptimizationstate-dependentbiasanalogin-memorycomputingbilevelimplicitpenalizationresiduallearninghardwareconditionnumberconvergencecomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies stochastic gradient descent on hardware where every desired update is scaled component-wise by state-dependent response functions, so positive and negative updates are systematically asymmetric. It first shows that this persistent bias is not harmless: naive analog SGD implicitly minimizes a penalized objective whose solution sits between the true optimum and the hardware's symmetric point, and the gap grows with gradient noise. The main constructive claim is that exact convergence to the original optimum is still possible, by reformulating the task as a bilevel problem: an inner variable learns the residual to the optimum, and an outer variable minimizes that residual. The proposed Residual Learning algorithm provably reaches the true minimizer despite the bias, at a last-iterate function gap of order O~(kappa1*kappa2^4*sigma^2/(mu*K)). A lower-bound construction shows a polynomial dependence on the hardware condition number kappa2 is unavoidable.

What carries the argument

The central object is the asymmetry ratio R(W)=G(W)/F(W), where F and G are the average and half-difference of the positive/negative response functions, and its componentwise antiderivative Rc(W). These define the implicit penalty <Sigma,Rc(W)> that explains why naive SGD fails. The constructive machinery is the bilevel reformulation BP: minimize gamma^2/2 ||P*(W)-W_diamond||^2 subject to P*(W)=argmin_P f(W+gamma(P-W_diamond)). Shifting by the symmetric point W_diamond ensures that, when W=W*, the inner minimizer coincides with W_diamond, so the penalty term becomes harmless. The convergence proof combines sufficient descent in both levels through a Lyapunov function and uses the hardware co

What would settle it

Run Residual Learning in one dimension with f(W)=mu/2*(W-W*)^2, response functions q+(w)=4+2tanh(w), q-(w)=4-2tanh(w), and symmetric point W_diamond=0. For these responses F=4, R(w)=-0.5 tanh(w), and Rc(w)=-0.5 log(cosh(w))<0 for w != 0. If the stochastic noise level Sigma satisfies Sigma > 2*mu*gamma^2, the penalized lower-level objective has a local maximum at P=W_diamond, so P*(W*) != W_diamond. If Residual Learning's iterates do not converge to W*, Theorem 3's claimed equivalence is disproved.

Watch

Extended reading notes

Core claim

The paper establishes that Analog SGD does not optimize the original problem: its stationary points are near the minimizer of f(W)+<Sigma,Rc(W)>, an implicit penalty that drags iterates toward the hardware symmetry point W_diamond. To fix this, the authors construct a bilevel problem BP whose upper-level objective is the squared residual ||P*(W)-W_diamond||^2, where P*(W) is the minimizer of a shifted lower-level objective. They prove that BP, and even its implicitly penalized version IBP, has the same unique global minimizer W* as the original problem. Consequently Residual Learning—updating the inner variable with distorted gradients and the outer variable with a deterministic residual ste

Load-bearing premise

The proof that the implicitly penalized bilevel problem has the same unique minimizer as the original assumes the penalty term <Sigma,Rc(W)> is nonnegative for all W, a condition not implied by the stated regularity assumptions on the response functions and false for some natural non-monotone responses.

Editorial extensions

If this is right

  • Naive analog SGD cannot reach the true optimum; its limit is biased toward the symmetry point by an amount proportional to the stochastic-gradient noise at the optimum.
  • Residual Learning restores exact convergence: the last-iterate function gap decays as O~(kappa1*kappa2^4*sigma^2/(mu*K)), preserving the classical log K/K SGD rate up to logarithmic factors.
  • A polynomial dependence on the hardware condition number is fundamental: there exists a hard instance forcing at least Omega~(kappa2^2*sigma^2/(mu*K)).
  • In simulations on image-classification tasks, Residual Learning matches digital SGD accuracy while Analog SGD lags, confirming the theoretical claims on realistic response models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the implicit penalty can be negative—for example with q+(w)=4+2tanh(w), q-(w)=4-2tanh(w), where Rc(w)=-0.5 log(cosh(w))<0—the equivalence between IBP and P is not guaranteed; testing Residual Learning on such non-monotone responses would delimit the theorem's true domain.
  • The gap between the kappa2^4 upper bound and the kappa2^2 lower bound suggests a more refined algorithm, perhaps exploiting the harmonic-mean effective response, could improve the worst-case exponent.
  • The sign-dependent scaling model is not specific to memory hardware: any actuator or quantizer with asymmetric positive/negative response fits the same dynamics, so the residual-learning construction could transfer to control or communication settings with sign-asymmetric distortions.
  • The kappa1 factor in the upper bound might be removable by iterate averaging, but the averaging variable would itself suffer the same state-dependent bias; designing a bias-immune averaging scheme is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper analyzes stochastic gradient methods on hardware where every update is scaled component-wise by state-dependent response functions q_+(·), q_-(·). It first shows that the resulting 'Analog SGD' recursion implicitly optimizes a penalized objective rather than the original problem P. To recover exact convergence, the authors introduce a bilevel reformulation BP and a two-timescale algorithm Residual Learning; they claim that the algorithm's implicit penalized bilevel problem IBP is equivalent to P, so Residual Learning converges exactly to the minimizer of P despite the persistent bias. The paper also gives an O~(κ1κ2^4σ^2/(μK)) upper bound and an Ω~(κ2^2σ^2/(μK)) lower bound, with κ2 the hardware condition number q_max/q_min.

Significance. If the results were correct, the paper would make a valuable contribution to optimization on analog in-memory hardware: it would provide the first exact-convergence guarantee for a gradient method under asymmetric, state-dependent response functions, and a complexity analysis in terms of a hardware condition number. The algorithm is simple, the convergence-rate comparison with digital SGD is informative, and the hard-instance lower bound is a useful step. The paper also includes reproducible simulation code and experiments on real datasets, which are strengths. However, the central equivalence theorem that underpins the paper's interpretation of exact convergence is not valid under the stated assumptions, and the proof of a key technical lemma contains a sign error; the main claims need substantial revision.

major comments (2)
  1. [§3, Theorem 3] The proof of Theorem 3 starts 'Since f(·) ≥ f* and ⟨Σ,Rc(·)⟩ ≥ 0', but the second inequality is not a consequence of Definition 1. Definition 2 only guarantees Rc≥0 when R is strictly monotone, and no such assumption appears in Theorem 3. A concrete allowed instance is q±(w)=4±2tanh(w), for which F=4, R(w)=−0.5tanh(w), and Rc(w)=−0.5log cosh(w)<0 for w≠0. For scalar f(w)=μ/2(w−w*)² and Σ>2μγ², the lower-level IBP objective at W=W* is μγ²P²/2+ΣRc(P), whose second derivative at P=W⋄=0 is μγ²−Σ/2<0; hence P*(W*)≠W⋄ and the proof's asserted equality is false. One can even construct a Lipschitz R with a double-well Rc so that W* is not the global minimizer of IBP. Thus Theorem 3 is false as stated, and the equivalence between IBP and P used in the abstract and §3 is unsupported.
  2. [Appendix A, Lemma 8(iii), Eq. (64)] The displayed chain in the proof of Lemma 8(iii) bounds E⟨∆Z,(|∆Z+δZ|−|∆Z|)⊙G(Z)⟩ from below, but the term appearing in (63) is ⟨∆Z,(|∆Z|−|∆Z+δZ|)⊙G(Z)⟩; the required estimate is an upper bound on E⟨∆Z,(|∆Z+δZ|−|∆Z|)⊙G(Z)⟩. The inequality therefore has the wrong direction. The lemma statement itself appears true — it can be proved directly via E⟨a,A(a+δ)⟩ = F||a||² − ⟨a,G⟩E|a+δ| and Young's inequality — but the written proof is invalid. Since Lemma 1 and hence Theorem 4 rely on Lemma 8(iii), the proof needs a corrected argument.
minor comments (5)
  1. [§3 vs §4] The exact-convergence claim should be disentangled from Theorem 3. Theorem 4's Lyapunov proof bounds f(¯W_K)−f* + C||P*(W_K)−W⋄||² directly and does not invoke IBP equivalence. The text should state that exact convergence is established by Theorem 4 itself, and present Theorem 3 only as a separate (corrected) interpretation. As written, the failure of Theorem 3 makes the main convergence claim appear unsupported even if the direct proof can be repaired.
  2. [Eq. (17) and proof of Theorem 4] The theorem statement writes a bound on +C||P*(W_K)−W⋄||², while the proof uses the Lyapunov function with Cγ²/2||P*(W_K)−W⋄||². Please align the notation so the reader can verify the constant dependence.
  3. [Definition 2 / Theorem 3] If the authors intend to keep Theorem 3, they must add an explicit assumption such as monotonicity of R(·) (or nonnegativity of Rc(·)) and state it in the theorem. The current text says only 'if R is moreover strictly monotone' in Definition 2, which is not part of the theorem's hypotheses.
  4. [Theorem 5 and surrounding text] The phrase 'a polynomial dependence on κ2 is unavoidable in general' overstates the result: Theorem 5 gives a lower bound for Residual Learning on a hard instance, not an information-theoretic lower bound for all algorithms. Please qualify the claim accordingly.
  5. [Appendix A, Eq. (64)] There are typos in the displayed chain, including 'E[∥|δZ∥2]' with a missing bracket and an apparent misplaced factor. These should be corrected in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation chain is self-contained; the questionable Rc≥0 step in Theorem 3 is a correctness gap, not a reduction of the result to its inputs.

full rationale

The paper's derivation is mathematically self-contained rather than circular. Theorem 1 constructs the penalty Rc as the antiderivative of R=G/F, so the gradient identity ∇<Σ,Rc>=Σ⊙R holds by definition; the theorem's quantitative statements are then proved directly in Appendix B, with no fitted parameters. The bilevel reformulations BP and IBP are algebraic restatements of the same optimization problem: P*(W)=W⋄+(W*−W)/γ makes the upper-level objective equal ||W−W*||²/(2γ²), so Theorem 2 is a definitional equivalence, and the nontrivial content is the algorithmic convergence in Lemmas 1–3 and Theorem 4, which contract a Lyapunov function under the biased updates (14)–(15). Theorem 5 is an explicit adversarial construction, not a lower bound reverse-engineered from the upper bound. Self-citations (Wu et al. 2024/2025, Kim et al. 2019) are used only for hardware motivation and for the practical assumption of a known symmetric point; none supplies a load-bearing mathematical premise. The most notable weakness—the proof of Theorem 3 asserts ⟨Σ,Rc(·)⟩≥0, which is not implied by Definition 1—is an unstated-assumption/correctness issue, not a circularity: the equivalence claim would be restored by explicitly assuming monotonicity of R or nonnegativity of Rc, and the rest of the analysis does not reduce to that assumption. No fitted input is later reported as a prediction, and no uniqueness theorem is imported from the authors' prior work to force the conclusion.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claims rely on standard smoothness/strong convexity and bounded-variance noise, plus hardware-specific domain assumptions: positive bounded Lipschitz response functions and a known symmetric point. The Theorem 3 proof additionally needs an unstated nonnegativity condition on the anti-derivative of the asymmetry ratio. The step sizes alpha, beta, gamma are proof-chosen and experiment-tuned, not fitted to a target result.

free parameters (3)
  • alpha (lower-level stepsize) = Theta(qmax log K / (gamma mu qmin^2 K))
    Algorithmic step size chosen in Lemma 3 to make the Lyapunov contraction work; not fitted to data, but the rate depends on this choice.
  • beta (upper-level stepsize) = Theta(alpha gamma mu qmin / qmax)
    Algorithmic step size chosen in Lemma 3; not fitted to data, but required for the descent coupling in Lemma 2.
  • gamma (mixing coefficient) = Omega(LS sigma sqrt(qmax) / (mu qmin^{3/2}))
    Hyper-parameter in the shifted problem and in Residual Learning updates. The proof requires a lower bound; in experiments it is swept by hand (Fig. 7), not fitted to reproduce the theorems.
assumptions (6)
  • domain assumption Response functions are positive, uniformly bounded, differentiable, and LS-Lipschitz (Definition 1).
    This models hardware operation away from saturation. It is stated in Definition 1 and used throughout.
  • domain assumption The symmetric point W_diamond is known (Assumption 1).
    Residual Learning requires shifting the problem around W_diamond. If W_diamond is unknown, the method cannot be implemented as described.
  • standard math Objective is L-smooth and mu-strongly convex (Assumption 2).
    Standard strong convexity and smoothness used for the descent lemmas and the equivalence theorems.
  • standard math Stochastic gradients are unbiased with bounded variance (Assumption 3).
    Standard SGD noise model; also used to define Sigma = E|grad f(W*; xi)| in the implicit penalty.
  • domain assumption For Theorem 1, q+'(W) < -c0 and q-'(W) > c0 on S.
    This derivative condition implies R is strictly increasing, making the Jacobian positive definite. It is stated in Theorem 1 but not in Definition 1.
  • ad hoc to paper For Theorem 3, <Sigma, Rc(W)> >= 0 for all W.
    The proof of Theorem 3 uses this inequality, but it is not stated as an assumption and does not follow from Definition 1. It fails for q+(w)=4+2 tanh(w), q-(w)=4-2 tanh(w).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimization under Persistent State-Dependent Bias: Gradient-based Method and Complexity Analysis." pith.science (2026). https://pith.science/paper/QCQ7GCRN

@misc{pith2026260726032,
  author       = {Pith},
  title        = {Pith review of: Optimization under Persistent State-Dependent Bias: Gradient-based Method and Complexity Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QCQ7GCRN}},
  note         = {Machine review of arXiv:2607.26032}
}
read the original abstract

This paper studies the convergence of stochastic gradient descent (SGD) when the implemented updates are subject to a persistent and state-dependent bias, in which the desired update is scaled by response functions component-wise. Our first contribution is to demonstrate that SGD in this setting implicitly optimizes a penalized problem whose minimizer does not coincide with the true minimizer. To mitigate this convergence failure, we reformulate the original task as an equivalent bilevel optimization problem and propose a gradient-based algorithm, termed Residual Learning. Theoretical analysis shows that Residual Learning finds a solution to the original, unbiased optimization problem despite the hardware imperfections. Beyond exact convergence, we quantify how the response functions affect convergence complexity via the hardware condition number and show that a polynomial dependence on it is unavoidable in general, via a construction of a hard instance. The theoretical results are supported by numerical simulations that demonstrate the effectiveness of the proposed algorithm.

Figures

Figures reproduced from arXiv: 2607.26032 by the authors.

Figure 1
Figure 1. Illustration of persistent state￾dependent bias in R 2 . Any desired update ∆Z, which is typically a de￾scent direction of the objective func￾tion, is scaled by state-dependent re￾sponse functions q+(·) and q−(·) component-wise. The bias is exacer￾bated when the hardware condition number increases. This paper investigates a distinct computational model involving persistent bias that arises in in-situ optimization on… view at source ↗
Figure 2
Figure 2. Illustration of the relationship between [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of response functions from Assumption [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustration of the shifted problem in 2-dimensional space ( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Numerical illustrations of the implicit penalty and the effect of the hardware condition number. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Test accuracy curves for the models with the [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Test accuracy of ResNet models after 100 epochs trained by [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: The weight distributions of ResNet models that are trained by [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

299 extracted references · 39 linked inside Pith

  1. [1]

    Transactions of the American Mathematical Society , volume =

    Federer, Herbert , title =. Transactions of the American Mathematical Society , volume =. 1959 , publisher =

    1959

  2. [2]

    arXiv preprint arXiv:1209.4340 , year=

    Moments and absolute moments of the normal distribution , author=. arXiv preprint arXiv:1209.4340 , year=

    arXiv

  3. [3]

    NIST Digital Library of Mathematical Functions

  4. [4]

    1948 , publisher=

    Handbook of mathematical functions with formulas, graphs, and mathematical tables , author=. 1948 , publisher=

    1948

  5. [5]

    2012 , publisher=

    Matrix Analysis , author=. 2012 , publisher=

    2012

  6. [6]

    2013 , publisher=

    Introductory Lectures on Convex Optimization: A Basic Course , author=. 2013 , publisher=

    2013

  7. [7]

    2018 , publisher=

    Reinforcement Learning: An Introduction , author=. 2018 , publisher=

    2018

  8. [8]

    2013 , publisher=

    Br. 2013 , publisher=

    2013

Show all 299 references
  1. [9]

    2004 , publisher=

    Kernel Methods for Pattern Analysis , author=. 2004 , publisher=

    2004

  2. [10]

    2006 , publisher=

    Nonlinear programming: theory and algorithms , author=. 2006 , publisher=

    2006

  3. [11]

    Artificial intelligence and statistics , pages=

    Communication-efficient learning of deep networks from decentralized data , author=. Artificial intelligence and statistics , pages=. 2017 , organization=

    2017

  4. [12]

    International Conference on Machine Learning , pages=

    Error feedback fixes signsgd and other gradient compression schemes , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  5. [13]

    International Conference on Learning Representations , year=

    Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training , author=. International Conference on Learning Representations , year=

  6. [14]

    IEEE Transactions on Signal Processing , volume=

    On maintaining linear convergence of distributed learning and optimization under limited communication , author=. IEEE Transactions on Signal Processing , volume=. 2020 , publisher=

    2020

  7. [15]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech

    Seide, Frank and Fu, Hao and Droppo, Jasha and Li, Gang and Yu, Dong , booktitle=. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech. 2014 , organization=

    2014

  8. [16]

    Advances in neural information processing systems , volume=

    Atomo: Communication-efficient learning via atomic sparsification , author=. Advances in neural information processing systems , volume=

  9. [17]

    Advances in neural information processing systems , volume=

    Terngrad: Ternary gradients to reduce communication in distributed deep learning , author=. Advances in neural information processing systems , volume=

  10. [18]

    arXiv preprint arXiv:1902.06101 , year=

    Local differential privacy in decentralized optimization , author=. arXiv preprint arXiv:1902.06101 , year=

    1902 arXiv

  11. [19]

    IEEE INFOCOM 2019-IEEE Conference on Computer Communications , pages=

    Compressed distributed gradient descent: Communication-efficient consensus over networks , author=. IEEE INFOCOM 2019-IEEE Conference on Computer Communications , pages=. 2019 , organization=

    2019

  12. [20]

    Qsparse-local-SGD: Distributed

    Basu, Debraj and Data, Deepesh and Karakus, Can and Diggavi, Suhas , journal=. Qsparse-local-SGD: Distributed

  13. [21]

    Advances in Neural Information Processing Systems , volume=

    Robust and communication-efficient collaborative learning , author=. Advances in Neural Information Processing Systems , volume=

  14. [22]

    IEEE Transactions on Smart Grid , volume=

    Real-time energy management in microgrids with reduced battery capacity requirements , author=. IEEE Transactions on Smart Grid , volume=. 2017 , publisher=

    2017

  15. [23]

    IEEE Journal on Selected Areas in Communications , volume=

    Dynamic energy management for smart-grid-powered coordinated multipoint systems , author=. IEEE Journal on Selected Areas in Communications , volume=. 2016 , publisher=

    2016

  16. [24]

    Proceedings of the IEEE international conference on computer vision , pages=

    Multi-view convolutional neural networks for 3d shape recognition , author=. Proceedings of the IEEE international conference on computer vision , pages=

  17. [25]

    IEEE Transactions on Wireless Communications , year=

    Multi-device task-oriented communication via maximal coding rate reduction , author=. IEEE Transactions on Wireless Communications , year=

  18. [26]

    IEEE Transactions on Circuits and Systems for Video Technology , year=

    Robust Multi-Agent Collaborative Perception via Spatio-Temporal Awareness , author=. IEEE Transactions on Circuits and Systems for Video Technology , year=

  19. [27]

    IEEE transactions on pattern analysis and machine intelligence , volume=

    Segmentation of multivariate mixed data via lossy data coding and compression , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2007 , publisher=

    2007

  20. [28]

    Sparsified

    Stich, Sebastian U and Cordonnier, Jean-Baptiste and Jaggi, Martin , journal=. Sparsified

  21. [29]

    NeurIPS , year=

    Gradient sparsification for communication-efficient distributed optimization , author=. NeurIPS , year=

  22. [30]

    Information and Inference: A Journal of the IMA , volume=

    Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor , author=. Information and Inference: A Journal of the IMA , volume=. 2022 , publisher=

    2022

  23. [31]

    Alistarh, Dan and Grubic, Demjan and Li, Jerry and Tomioka, Ryota and Vojnovic, Milan , journal=

  24. [32]

    Adaptive Step-Size Methods for Compressed

    Subramaniam, Adarsh M and Magesh, Akshayaa and Veeravalli, Venugopal V , journal=. Adaptive Step-Size Methods for Compressed. 2024 , publisher=

    2024

  25. [33]

    Error compensated distributed

    Qian, Xun and Richt. Error compensated distributed. NeurIPS , year=

  26. [34]

    IEEE Transactions on Parallel and Distributed Systems , volume=

    Accelerating federated learning via momentum gradient descent , author=. IEEE Transactions on Parallel and Distributed Systems , volume=. 2020 , publisher=

    2020

  27. [35]

    2009 , publisher=

    Learning multiple layers of features from tiny images , author=. 2009 , publisher=

    2009

  28. [36]

    Computer Vision and Pattern Recognition , pages=

    Imagenet: A large-scale hierarchical image database , author=. Computer Vision and Pattern Recognition , pages=. 2009 , organization=

    2009

  29. [37]

    International Conference on Learning Representations , year=

    Mixed Precision Training , author=. International Conference on Learning Representations , year=

  30. [38]

    1951 , publisher=

    A Stochastic Approximation Method , author=. 1951 , publisher=

    1951

  31. [39]

    Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithms , author=

  32. [40]

    Non-asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning , author=

  33. [41]

    2022 , publisher=

    Finite-Time Convergence Rates of Decentralized Stochastic Approximation with Applications in Multi-Agent and Multi-Task Learning , author=. 2022 , publisher=

    2022

  34. [42]

    arXiv preprint arXiv:2111.12665 , year=

    Finite-time Error Bounds For Distributed Linear Stochastic Approximation , author=. arXiv preprint arXiv:2111.12665 , year=

    arXiv

  35. [43]

    Linear Stochastic Approximation: How Far does Constant Step-size and Iterate Averaging Go? , author=

  36. [44]

    Finite Time Analysis of Linear Two-timescale Stochastic Approximation with

    Kaledin, Maxim and Moulines, Eric and Naumov, Alexey and Tadic, Vladislav and Wai, Hoi-To , booktitle=COLT, year=. Finite Time Analysis of Linear Two-timescale Stochastic Approximation with

  37. [45]

    arXiv preprint arXiv:2112.12770 , year=

    Optimal and Instance-dependent Guarantees for Markovian Linear Stochastic Approximation , author=. arXiv preprint arXiv:2112.12770 , year=

    arXiv

  38. [46]

    The Annals of Statistics , pages=

    Nonparametric Stochastic Approximation with Large Step-sizes , author=. The Annals of Statistics , pages=. 2016 , publisher=

    2016

  39. [47]

    2005 , volume =

    Learning the Kernel with Hyperkernels , author=. 2005 , volume =

    2005

  40. [48]

    Advances in Neural Information Processing Systems , volume=

    A single-timescale analysis for stochastic approximation with multiple coupled sequences , author=. Advances in Neural Information Processing Systems , volume=

  41. [49]

    IEEE Transactions on Signal Processing , year=

    Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness: Theories and Applications , author=. IEEE Transactions on Signal Processing , year=

  42. [50]

    Luo, Zhi-Quan and Pang, Jong-Shi and Ralph, Daniel , year=

  43. [51]

    Mathematical programming , volume=

    Gradient methods for minimizing composite functions , author=. Mathematical programming , volume=

  44. [52]

    Computational Optimization and Applications , volume=

    Inexact proximal stochastic gradient method for convex composite optimization , author=. Computational Optimization and Applications , volume=

  45. [53]

    2014 , publisher=

    Implicit Functions and Solution Mappings: A View from Variational Analysis , author=. 2014 , publisher=

    2014

  46. [54]

    2022 , address=

    An Implicit Gradient-Type Method for Linearly Constrained Bilevel Problems , author=. 2022 , address=

    2022

  47. [55]

    Foundations and Trends

    Learning with Limited Samples: Meta-Learning and Applications to Communication Systems , author=. Foundations and Trends

  48. [56]

    Foundations and Trends

    Bilevel methods for image reconstruction , author=. Foundations and Trends

  49. [57]

    Mathematics of Operations Research , volume=

    Lipschitz continuity of solutions of variational inequalities with a parametric polyhedral constraint , author=. Mathematics of Operations Research , volume=. 1995 , publisher=

    1995

  50. [58]

    Applied Mathematics & Optimization , volume=

    On Lipschitz continuity of projections onto polyhedral moving sets , author=. Applied Mathematics & Optimization , volume=. 2021 , publisher=

    2021

  51. [59]

    Mathematical Programming , volume=

    Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning , author=. Mathematical Programming , volume=. 2021 , publisher=

    2021

  52. [60]

    Mathematical Programming , volume=

    Tame functions are semismooth , author=. Mathematical Programming , volume=. 2009 , publisher=

    2009

  53. [61]

    Optimization Letters , volume=

    Convergence of a stochastic subgradient method with averaging for nonsmooth nonconvex constrained optimization , author=. Optimization Letters , volume=. 2020 , publisher=

    2020

  54. [62]

    Set-Valued and Variational Analysis , pages=

    Conservative and semismooth derivatives are equivalent for semialgebraic maps , author=. Set-Valued and Variational Analysis , pages=. 2021 , publisher=

    2021

  55. [63]

    Bilevel optimization , author=

  56. [64]

    ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally! , author=

  57. [65]

    A single-timescale method for stochastic bilevel optimization , author=

  58. [66]

    New necessary optimality conditions for bilevel programs by combining the

    Ye, Jane J and Zhu, Daoli , journal=. New necessary optimality conditions for bilevel programs by combining the. 2010 , publisher=

    2010

  59. [67]

    Gradient-based hyperparameter optimization through reversible learning , author=

  60. [68]

    Forward and reverse gradient-based hyperparameter optimization , author=

  61. [69]

    Model-agnostic meta-learning for fast adaptation of deep networks , author=

  62. [70]

    Liu, Hanxiao and Simonyan, Karen and Yang, Yiming , booktitle=ICLR, year=

  63. [71]

    Tarzanagh, Davoud Ataee and Li, Mingchen and Thrampoulidis, Christos and Oymak, Samet , booktitle=ICML, year=

  64. [72]

    arXiv preprint arXiv:1607.05447 , year=

    On differentiating parameterized argmin and argmax problems with application to bi-level optimization , author=. arXiv preprint arXiv:1607.05447 , year=

    arXiv

  65. [73]

    The Mathematical Gazette , volume=

    The generalised inverse , author=. The Mathematical Gazette , volume=. 1978 , publisher=

    1978

  66. [74]

    SIAM Journal on Optimization , volume=

    Stochastic model-based minimization of weakly convex functions , author=. SIAM Journal on Optimization , volume=. 2019 , publisher=

    2019

  67. [75]

    Mathematical Programming , volume=

    Efficiency of minimizing compositions of convex functions and smooth maps , author=. Mathematical Programming , volume=. 2019 , publisher=

    2019

  68. [76]

    Bilevel Programming for Hyperparameter Optimization and Meta-Learning , author =

  69. [77]

    Barata, Jo. The. Brazilian Journal of Physics , volume=. 2012 , publisher=

    2012

  70. [78]

    Foundations and trends

    Proximal algorithms , author=. Foundations and trends. 2014 , publisher=

    2014

  71. [79]

    Communication-efficient learning of deep networks from decentralized data , author=

  72. [80]

    The error-feedback framework: Better rates for

    Stich, Sebastian U and Karimireddy, Sai Praneeth , journal=. The error-feedback framework: Better rates for

  73. [81]

    Stich, Sebastian U , booktitle=ICLR, year=. Local

  74. [82]

    Parallel restarted

    Yu, Hao and Yang, Sen and Zhu, Shenghuo , booktitle=AAAI, year=. Parallel restarted

  75. [83]

    Is local

    Woodworth, Blake and Patel, Kumar Kshitij and Stich, Sebastian and Dai, Zhen and Bullins, Brian and Mcmahan, Brendan and Shamir, Ohad and Srebro, Nathan , booktitle=ICML, year=. Is local

  76. [84]

    Foundations and Trends

    Advances and open problems in federated learning , author=. Foundations and Trends

  77. [85]

    Operations Research , volume=

    Mathematical programs with optimization problems in the constraints , author=. Operations Research , volume=

  78. [86]

    Optimization , volume=

    Optimality conditions for bilevel programming problems , author=. Optimization , volume=

  79. [87]

    Journal of Global optimization , volume=

    Bilevel and multilevel programming: A bibliography review , author=. Journal of Global optimization , volume=

  80. [88]

    Annals of operations research , volume=

    An overview of bilevel optimization , author=. Annals of operations research , volume=

  81. [89]

    IEEE Transactions on Evolutionary Computation , volume=

    A review on bilevel optimization: from classical to evolutionary approaches and applications , author=. IEEE Transactions on Evolutionary Computation , volume=

  82. [90]

    On the iteration complexity of hypergradient computation , author=

  83. [91]

    Optimizing millions of hyperparameters by implicit differentiation , author=

  84. [92]

    Provably faster algorithms for bilevel optimization , author=

  85. [93]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

    Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

  86. [94]

    arXiv preprint arXiv:2203.16615 , year=

    A Fast and Convergent Proximal Algorithm for Regularized Nonconvex and Nonsmooth Bi-level Optimization , author=. arXiv preprint arXiv:2203.16615 , year=

    arXiv

  87. [95]

    arXiv preprint arXiv:2208.00732 , year=

    An Improved Unconstrained Approach for Bilevel Optimization , author=. arXiv preprint arXiv:2208.00732 , year=

    arXiv

  88. [96]

    Towards gradient-based bilevel optimization with non-convex followers and beyond , author=

  89. [97]

    Value Function Based Difference-of-Convex Algorithm for Bilevel Hyperparameter Selection Problems , author=

  90. [98]

    Optimization , volume=

    New necessary optimality conditions in optimistic bilevel programming , author=. Optimization , volume=. 2007 , publisher=

    2007

  91. [99]

    Mathematical Programming , volume=

    The bilevel programming problem: reformulations, constraint qualifications and optimality conditions , author=. Mathematical Programming , volume=. 2013 , publisher=

    2013

  92. [100]

    Discrete Optimization , volume=

    One-level reformulation of the bilevel knapsack problem using dynamic programming , author=. Discrete Optimization , volume=. 2013 , publisher=

    2013

  93. [101]

    Mathematics of Operations Research , volume=

    Error bounds, quadratic growth, and linear convergence of proximal methods , author=. Mathematics of Operations Research , volume=. 2018 , publisher=

    2018

  94. [102]

    Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning , author=

  95. [103]

    Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients , author=

  96. [104]

    arXiv preprint arXiv:2207.10751 , year=

    Federated Learning on Adaptively Weighted Nodes by Bilevel Optimization , author=. arXiv preprint arXiv:2207.10751 , year=

    arXiv

  97. [105]

    arXiv preprint arXiv:2205.01608 , year=

    Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction , author=. arXiv preprint arXiv:2205.01608 , year=

    arXiv

  98. [106]

    Decentralized Gossip-Based Stochastic Bilevel Optimization over Communication Networks , author=

  99. [107]

    arXiv preprint arXiv:2206.05670 , year=

    Decentralized Bilevel Optimization , author=. arXiv preprint arXiv:2206.05670 , year=

    arXiv

  100. [108]

    arXiv preprint arXiv:2206.15025 , year=

    Stochastic Bilevel Distributed Optimization over a Network , author=. arXiv preprint arXiv:2206.15025 , year=

    arXiv

  101. [109]

    Decentralized Bilevel Optimization for Personalized Client Learning , author=. Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing , year=

  102. [110]

    Mathematical Programming , volume=

    Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization , author=. Mathematical Programming , volume=. 2016 , publisher=

    2016

  103. [111]

    AutoBalance: Optimized Loss Functions for Imbalanced Data , author=

  104. [112]

    Label-Imbalanced and Group-Sensitive Classification under Overparameterization , author=

  105. [113]

    2004 , publisher=

    Convex optimization , author=. 2004 , publisher=

    2004

  106. [114]

    IEEE Transactions on signal processing , volume=

    K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation , author=. IEEE Transactions on signal processing , volume=. 2006 , publisher=

    2006

  107. [115]

    Journal of Global Optimization , volume=

    Global optimization of nonlinear bilevel programming problems , author=. Journal of Global Optimization , volume=. 2001 , publisher=

    2001

  108. [116]

    IEEE Transactions on Neural Networks , volume=

    A one-layer recurrent neural network for pseudoconvex optimization subject to linear equality constraints , author=. IEEE Transactions on Neural Networks , volume=. 2011 , publisher=

    2011

  109. [117]

    Mathematical programming , volume=

    Network design problem with congestion effects: A case of bilevel programming , author=. Mathematical programming , volume=. 1986 , publisher=

    1986

  110. [118]

    Transportation Research Part B: Methodological , volume=

    Two-stage stochastic bilevel programming over a transportation network , author=. Transportation Research Part B: Methodological , volume=. 2013 , publisher=

    2013

  111. [119]

    Differential properties of Sinkhorn approximation for learning with Wasserstein distance , author=

  112. [120]

    arXiv preprint arXiv:2202.12404 , year=

    Exploiting Problem Structure in Deep Declarative Networks: Two Case Studies , author=. arXiv preprint arXiv:2202.12404 , year=

    arXiv

  113. [121]

    Second-Order Sensitivity Analysis for Bilevel Optimization , author=

  114. [122]

    arXiv preprint arXiv:2205.14224 , year=

    Will Bilevel Optimizers Benefit from Loops , author=. arXiv preprint arXiv:2205.14224 , year=

    arXiv

  115. [123]

    arXiv preprint arXiv:2003.05437 , volume=

    Matrix concentration for products , author=. arXiv preprint arXiv:2003.05437 , volume=

    2003 arXiv

  116. [124]

    A Stochastic Linearized Augmented Lagrangian Method for Decentralized Bilevel Optimization , author=

  117. [125]

    Nonsmooth implicit differentiation for machine-learning and optimization , author=

  118. [126]

    Journal of Machine Learning Research , volume=

    Implicit differentiation for fast hyperparameter selection in non-smooth convex learning , author=. Journal of Machine Learning Research , volume=

  119. [127]

    arXiv preprint arXiv:2204.13299 , year=

    On the Convergence of Momentum-Based Algorithms for Federated Stochastic Bilevel Optimization Problems , author=. arXiv preprint arXiv:2204.13299 , year=

    arXiv

  120. [128]

    A framework for bilevel optimization that enables stochastic and global variance reduction algorithms , author=

  121. [129]

    Transportation Research Part B: Methodological , volume=

    Continuous equilibrium network design models , author=. Transportation Research Part B: Methodological , volume=

  122. [130]

    Management science , volume=

    A bilevel model of taxation and its application to optimal highway pricing , author=. Management science , volume=. 1998 , publisher=

    1998

  123. [131]

    Optimal transport for applied mathematicians , author=. Birk. 2015 , publisher=

    2015

  124. [132]

    Alternating Implicit Projected SGD and Its Efficient Variants for Equality-constrained Bilevel Optimization , author=

  125. [133]

    On implicit bias in overparameterized bilevel optimization , author=

  126. [134]

    Solving a class of non-convex min-max games using iterative first order methods , author=

  127. [135]

    Faster single-loop algorithms for minimax optimization without strong concavity , author=

  128. [136]

    Amortized Implicit Differentiation for Stochastic Bilevel Optimization , author=

  129. [137]

    Non-Convex Bilevel Games with Critical Point Selection Maps , author=

  130. [138]

    SIAM Journal on control and optimization , volume=

    An exact penalization viewpoint of constrained optimization , author=. SIAM Journal on control and optimization , volume=. 1991 , publisher=

    1991

  131. [139]

    BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach , author=

  132. [140]

    Set-Valued and Variational Analysis , pages=

    Variational analysis perspective on linear convergence of some first order methods for nonsmooth convex optimization problems , author=. Set-Valued and Variational Analysis , pages=. 2021 , publisher=

    2021

  133. [141]

    Mathematical Programming , pages=

    On Penalty-based Bilevel Gradient Descent Method , author=. Mathematical Programming , pages=

  134. [142]

    SIAM review , volume=

    On the perturbation of pseudo-inverses, projections and linear least squares problems , author=. SIAM review , volume=. 1977 , publisher=

    1977

  135. [143]

    Contextual transformation networks for online continual learning , author=

  136. [144]

    Revisiting and advancing fast adversarial training through the lens of bi-level optimization , author=

  137. [145]

    Journal of global optimization , volume=

    Bilevel programming in traffic planning: Models, methods and challenge , author=. Journal of global optimization , volume=. 1995 , publisher=

    1995

  138. [146]

    The Theory of Market Economy , Year =

    Heinrich Von Stackelberg , Publisher =. The Theory of Market Economy , Year =

  139. [147]

    2002 , publisher=

    Foundations of bilevel programming , author=. 2002 , publisher=

    2002

  140. [148]

    Provable representation learning for imitation learning via bi-level optimization , author=

  141. [149]

    Conference on Uncertainty in Artificial Intelligence , year=

    Learning intrinsic rewards as a bi-level optimization problem , author=. Conference on Uncertainty in Artificial Intelligence , year=

  142. [150]

    Coresets via Bilevel Optimization for Continual Learning and Streaming , author=

  143. [151]

    Mathematical programming , volume=

    The polynomial hierarchy and a simple model for competitive analysis , author=. Mathematical programming , volume=. 1985 , publisher=

    1985

  144. [152]

    SIAM Journal on scientific and Statistical Computing , volume=

    New branch-and-bound rules for linear bilevel programming , author=. SIAM Journal on scientific and Statistical Computing , volume=. 1992 , publisher=

    1992

  145. [153]

    Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity , author=

  146. [154]

    A value-function-based interior-point method for non-convex bi-level optimization , author=

  147. [155]

    1984 , publisher=

    Directional derivative of the marginal function in nonlinear programming , author=. 1984 , publisher=

    1984

  148. [156]

    Mathematical Programming , volume=

    On solving simple bilevel programs with a nonconvex lower level program , author=. Mathematical Programming , volume=. 2014 , publisher=

    2014

  149. [157]

    Applied and Computational Harmonic Analysis , volume=

    Loss landscapes and optimization in over-parameterized non-linear systems and neural networks , author=. Applied and Computational Harmonic Analysis , volume=. 2022 , publisher=

    2022

  150. [158]

    Global convergence of policy gradient methods for the linear quadratic regulator , author=

  151. [159]

    Foundations of Computational Mathematics , volume=

    A geometric analysis of phase retrieval , author=. Foundations of Computational Mathematics , volume=. 2018 , publisher=

    2018

  152. [160]

    Mathematical Programming , volume=

    Lower bounds for finding stationary points I , author=. Mathematical Programming , volume=. 2020 , publisher=

    2020

  153. [161]

    Nonlinear Analysis: Theory, Methods & Applications , volume=

    On the Karush--Kuhn--Tucker reformulation of the bilevel optimization problem , author=. Nonlinear Analysis: Theory, Methods & Applications , volume=. 2012 , publisher=

    2012

  154. [162]

    Optimization with Multivalued Mappings: Theory, Applications, and Algorithms , pages=

    Optimality conditions for bilevel programming problems , author=. Optimization with Multivalued Mappings: Theory, Applications, and Algorithms , pages=. 2006 , publisher=

    2006

  155. [163]

    Bilevel Optimization: Advances and Next Challenges , pages=

    Constraint qualifications and optimality conditions in bilevel optimization , author=. Bilevel Optimization: Advances and Next Challenges , pages=. 2020 , publisher=

    2020

  156. [164]

    Applicable Analysis , volume=

    On calmness conditions in convex bilevel programming , author=. Applicable Analysis , volume=. 2011 , publisher=

    2011

  157. [165]

    Hataya, Ryuichiro and Yamada, Makoto , booktitle=AISTATS, year=. Nystr

  158. [166]

    Hyperparameter optimization with approximate gradient , author=

  159. [167]

    Overparameterized nonlinear learning: Gradient descent takes the shortest path? , author=

  160. [168]

    Acceleration of

    Liu, Yanli and Feng, Fei and Yin, Wotao , booktitle=ICML, year=. Acceleration of

  161. [169]

    SIAM Journal on Control and Optimization , volume=

    Semismooth and semiconvex functions in constrained optimization , author=. SIAM Journal on Control and Optimization , volume=. 1977 , publisher=

    1977

  162. [170]

    arXiv preprint arXiv:2002.11962 , year=

    Can We Find Near-Approximately-Stationary Points of Nonsmooth Nonconvex Functions? , author=. arXiv preprint arXiv:2002.11962 , year=

    2002 arXiv

  163. [171]

    Closing the Gap: Tighter Analysis of Alternating Stochastic Gradient Methods for Bilevel Problems , author=

  164. [172]

    1973 , publisher=

    Generalized inverse of matrices and its applications , author=. 1973 , publisher=

    1973

  165. [173]

    2008 , publisher=

    Nonsmooth analysis and control theory , author=. 2008 , publisher=

    2008

  166. [174]

    Journal of Scientific Computing , volume=

    Moreau envelope augmented Lagrangian method for nonconvex optimization with linear constraints , author=. Journal of Scientific Computing , volume=. 2022 , publisher=

    2022

  167. [175]

    Advances in Neural Information Processing Systems , volume=

    Oracle complexity in nonsmooth nonconvex optimization , author=. Advances in Neural Information Processing Systems , volume=

  168. [176]

    Linear algebra and its applications , volume=

    Explicit solutions to the matrix inverse problem AX= B , author=. Linear algebra and its applications , volume=. 1999 , publisher=

    1999

  169. [177]

    Mathematical proceedings of the Cambridge philosophical society , volume=

    A generalized inverse for matrices , author=. Mathematical proceedings of the Cambridge philosophical society , volume=. 1955 , organization=

    1955

  170. [178]

    SIAM Journal on Optimization , volume=

    A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic , author=. SIAM Journal on Optimization , volume=. 2023 , publisher=

    2023

  171. [179]

    arXiv preprint arXiv:2201.04611 , year=

    A superlinearly convergent subgradient method for sharp semismooth problems , author=. arXiv preprint arXiv:2201.04611 , year=

    arXiv

  172. [180]

    Journal of optimization theory and applications , volume=

    On concepts of directional differentiability , author=. Journal of optimization theory and applications , volume=. 1990 , publisher=

    1990

  173. [181]

    1990 , publisher=

    Optimization and nonsmooth analysis , author=. 1990 , publisher=

    1990

  174. [182]

    arXiv preprint arXiv:1802.02246 , year=

    Approximation methods for bilevel programming , author=. arXiv preprint arXiv:1802.02246 , year=

    arXiv

  175. [183]

    Bilevel optimization: Convergence analysis and enhanced design , author=

  176. [184]

    A fully single loop algorithm for bilevel optimization without hessian inverse , author=

  177. [185]

    Wachsmuth, Gerd , journal=. On. 2013 , publisher=

    2013

  178. [186]

    arXiv preprint arXiv:2205.10054 , year=

    Towards Extremely Fast Bilevel Optimization with Self-governed Convergence Guarantees , author=. arXiv preprint arXiv:2205.10054 , year=

    arXiv

  179. [187]

    A near-optimal algorithm for stochastic bilevel optimization via double-momentum , author=

  180. [188]

    SIAM Journal on Optimization , volume=

    Constraint qualifications and necessary optimality conditions for optimization problems with variational inequality constraints , author=. SIAM Journal on Optimization , volume=. 2000 , publisher=

    2000

  181. [189]

    arXiv preprint arXiv:2303.03944 , year=

    On Momentum-Based Gradient Methods for Bilevel Optimization with Nonconvex Lower-Level , author=. arXiv preprint arXiv:2303.03944 , year=

    arXiv

  182. [190]

    Neural computation , volume=

    Fast exact multiplication by the Hessian , author=. Neural computation , volume=. 1994 , publisher=

    1994

  183. [191]

    Asian Conference on Machine Learning , year=

    Penalty method for inversion-free deep bilevel optimization , author=. Asian Conference on Machine Learning , year=

  184. [193]

    A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton , author=

  185. [194]

    Mathematical Programming , pages=

    Difference of convex algorithms for bilevel programs with applications in hyperparameter selection , author=. Mathematical Programming , pages=

  186. [195]

    A conditional gradient-based method for simple bilevel optimization with convex lower-level problem , author=

  187. [196]

    Optimization with Multivalued Mappings: Theory, Applications, and Algorithms , pages=

    Bilevel programming with convex lower level problems , author=. Optimization with Multivalued Mappings: Theory, Applications, and Algorithms , pages=

  188. [197]

    SIAM Journal on Optimization , volume=

    A first order method for solving convex bilevel optimization problems , author=. SIAM Journal on Optimization , volume=. 2017 , publisher=

    2017

  189. [198]

    Mathematical programming , volume=

    Complementarity formulations and existence of solutions of dynamic multi-rigid-body contact problems with coulomb friction , author=. Mathematical programming , volume=. 1996 , publisher=

    1996

  190. [199]

    arXiv preprint arXiv:2308.00788 , year=

    An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learning , author=. arXiv preprint arXiv:2308.00788 , year=

    arXiv

  191. [200]

    Linearly constrained bilevel optimization: A smoothed implicit gradient approach , author=

  192. [201]

    Convex-concave min-max stackelberg games , author=

  193. [202]

    A Generalized Alternating Method for Bilevel Optimization under the Polyak-

    Xiao, Quan and Lu, Songtao and Chen, Tianyi , booktitle=NIPS, year=. A Generalized Alternating Method for Bilevel Optimization under the Polyak-

  194. [203]

    A fully first-order method for stochastic bilevel optimization , author=

  195. [204]

    arXiv preprint arXiv:2203.01123 , year=

    A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima , author=. arXiv preprint arXiv:2203.01123 , year=

    arXiv

  196. [205]

    arXiv preprint arXiv:2306.14853 , year=

    Near-Optimal Fully First-Order Algorithms for Finding Stationary Points in Bilevel Optimization , author=. arXiv preprint arXiv:2306.14853 , year=

    arXiv

  197. [206]

    Automatic and harmless regularization with constrained and lexicographic optimization: A dynamic barrier approach , author=

  198. [207]

    arXiv preprint arXiv:2301.01716 , year=

    First-order penalty methods for bilevel optimization , author=. arXiv preprint arXiv:2301.01716 , year=

    arXiv

  199. [208]

    arXiv preprint arXiv:2305.13153 , year=

    Effective Bilevel Optimization via Minimax Reformulation , author=. arXiv preprint arXiv:2305.13153 , year=

    arXiv

  200. [209]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

    Value-Function-Based Sequential Minimization for Bi-Level Optimization , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

  201. [210]

    Mathematical Programming , volume=

    Stochastic first-order methods for convex and nonconvex functional constrained optimization , author=. Mathematical Programming , volume=

  202. [211]

    arXiv preprint arXiv:1908.01871 , year=

    Proximally constrained methods for weakly convex optimization with weakly convex constraints , author=. arXiv preprint arXiv:1908.01871 , year=

    1908 arXiv

  203. [212]

    Computational optimization and applications , volume=

    Complexity of an inexact proximal-point penalty method for constrained smooth non-convex optimization , author=. Computational optimization and applications , volume=

  204. [213]

    A single-loop gradient descent and perturbed ascent algorithm for nonconvex functional constrained optimization , author=

  205. [214]

    Lu, Songtao and Gao, Tian , booktitle=ICASSP, year=. Meta-

  206. [215]

    2009 , publisher=

    Variational analysis , author=. 2009 , publisher=

    2009

  207. [216]

    FairBatch: Batch Selection for Model Fairness , author=

  208. [217]

    Complexity of finding stationary points of nonconvex nonsmooth functions , author=

  209. [218]

    Efficient and modular implicit differentiation , author=

  210. [219]

    Implicit differentiation of lasso-type models for hyperparameter optimization , author=

  211. [220]

    arXiv preprint arXiv:2205.08011 , year=

    Level constrained first order methods for function constrained optimization , author=. arXiv preprint arXiv:2205.08011 , year=

    arXiv

  212. [221]

    Mathematical Programming , volume=

    A first order method for finding minimal norm-like solutions of convex optimization problems , author=. Mathematical Programming , volume=. 2014 , publisher=

    2014

  213. [222]

    SIAM Journal on Optimization , volume=

    A method with convergence rates for optimization problems with variational inequality constraints , author=. SIAM Journal on Optimization , volume=. 2021 , publisher=

    2021

  214. [223]

    On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximation , author=

  215. [224]

    Israel Journal of Mathematics , volume=

    A remark on regularization in Hilbert spaces , author=. Israel Journal of Mathematics , volume=. 1986 , publisher=

    1986

  216. [225]

    Advances in Neural Information Processing Systems , volume=

    A simple and efficient smoothing method for faster optimization and local exploration , author=. Advances in Neural Information Processing Systems , volume=

  217. [226]

    Advances in Neural Information Processing Systems , volume=

    On the convergence theory for hessian-free bilevel algorithms , author=. Advances in Neural Information Processing Systems , volume=

  218. [227]

    Journal of Convex Analysis , volume=

    An explicit descent method for bilevel convex optimization , author=. Journal of Convex Analysis , volume=

  219. [228]

    SIAM Journal on Optimization , volume=

    Proximal point algorithm controlled by a slowly vanishing term: applications to hierarchical minimization , author=. SIAM Journal on Optimization , volume=

  220. [229]

    arXiv preprint arXiv:1910.01727 , year=

    Generalized Inner Loop Meta-Learning , author=. arXiv preprint arXiv:1910.01727 , year=

    1910 arXiv

  221. [230]

    Computational Optimization and Applications , volume=

    Algorithms for stochastic optimization with function or expectation constraints , author=. Computational Optimization and Applications , volume=

  222. [231]

    Betty: An Automatic Differentiation Library for Multilevel Optimization , author=

  223. [232]

    arXiv preprint arXiv:2211.06934 , year =

    TorchOpt: An Efficient Library for Differentiable Optimization , author =. arXiv preprint arXiv:2211.06934 , year =

    arXiv

  224. [233]

    Adaptively truncating backpropagation through time to control gradient bias , author=

  225. [234]

    Truncated back-propagation for bilevel optimization , author=

  226. [235]

    Adversarial Training Should Be Cast as a Non-Zero-Sum Game , author=

  227. [236]

    Bilevel Coreset Selection in Continual Learning: A New Formulation and Algorithm , author=

  228. [237]

    Visualizing the loss landscape of neural nets , author=

  229. [238]

    arXiv preprint arXiv:2401.00108 , year=

    Stochastic Optimization under Hidden Convexity , author=. arXiv preprint arXiv:2401.00108 , year=

    arXiv

  230. [239]

    SIAM Journal on Optimization , volume=

    Global convergence of splitting methods for nonconvex composite optimization , author=. SIAM Journal on Optimization , volume=. 2015 , publisher=

    2015

  231. [240]

    Stackelberg games for adversarial prediction problems , author=. Proc. ACM SIGKDD international conference on Knowledge discovery and data mining , year=

  232. [241]

    Fast algorithms for stackelberg prediction game with least squares loss , author=

  233. [242]

    Solving Stackelberg Prediction Game with Least Squares Loss via Spherically Constrained Least Squares Reformulation , author=

  234. [243]

    Selected Papers Of Alan J Hoffman: With Commentary , pages=

    On approximate solutions of systems of linear inequalities , author=. Selected Papers Of Alan J Hoffman: With Commentary , pages=. 2003 , publisher=

    2003

  235. [244]

    ON THE GLOBAL CONVERGENCE OF TRAINING DEEP LINEAR RESNETS , author=

  236. [245]

    Optimal learning from verified training data , author=

  237. [246]

    Xu, Ziqing and Min, Hancheng and Tarmoun, Salma and Mallada, Enrique and Vidal, Rene , year=. A local

  238. [247]

    Linear Convergence of Gradient Descent for Finite Width Over-parametrized Linear Networks with General Initialization , author=

  239. [248]

    Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , year=

    Efficient bi-level optimization for recommendation denoising , author=. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , year=

  240. [249]

    arXiv preprint arXiv:2206.00147 , year=

    Unbiased implicit feedback via bi-level optimization , author=. arXiv preprint arXiv:2206.00147 , year=

    arXiv

  241. [250]

    arXiv preprint arXiv:2304.00459 , year=

    Fast Convergence of Random Reshuffling under Over-Parameterization and the Polyak- L ojasiewicz Condition , author=. arXiv preprint arXiv:2304.00459 , year=

    arXiv

  242. [251]

    Understanding Why Generalized Reweighting Does Not Improve Over

    Zhai, Runtian and Dan, Chen and Kolter, J Zico and Ravikumar, Pradeep Kumar , booktitle=ICLR, year=. Understanding Why Generalized Reweighting Does Not Improve Over

  243. [252]

    arXiv preprint arXiv:2310.17386 , year=

    A Challenge in Reweighting Data with Bilevel Optimization , author=. arXiv preprint arXiv:2310.17386 , year=

    arXiv

  244. [253]

    Siam Review , volume=

    Note on the generalized inverse of a matrix product , author=. Siam Review , volume=. 1966 , publisher=

    1966

  245. [254]

    Matrix Rank and the Inverse of a Full Rank Matrix , author =

  246. [255]

    Journal of the Society for Industrial and Applied Mathematics , volume=

    Representations for the generalized inverse of a partitioned matrix , author=. Journal of the Society for Industrial and Applied Mathematics , volume=

  247. [256]

    Energy Reports , volume=

    Bi-level and multi-objective optimization of renewable energy sources and storage planning to support existing overloaded electricity grids , author=. Energy Reports , volume=

  248. [257]

    IEEE transactions on Smart Grid , volume=

    Bilevel optimization framework for smart building-to-grid systems , author=. IEEE transactions on Smart Grid , volume=

  249. [258]

    International Journal of Electrical Power & Energy Systems , volume=

    A bi-level robust planning model for active distribution networks considering uncertainties of renewable energies , author=. International Journal of Electrical Power & Energy Systems , volume=

  250. [259]

    IEEE Transactions on Industry Applications , volume=

    Deep reinforcement learning based optimal schedule for a battery swapping station considering uncertainties , author=. IEEE Transactions on Industry Applications , volume=. 2020 , publisher=

    2020

  251. [260]

    Transportation Research Part B: Methodological , volume=

    Solution algorithm for the bi-level discrete network design problem , author=. Transportation Research Part B: Methodological , volume=

  252. [261]

    IEEE transactions on cybernetics , volume=

    A bilevel optimization approach for joint offloading decision and resource allocation in cooperative mobile edge computing , author=. IEEE transactions on cybernetics , volume=

  253. [262]

    Journal of Global Optimization , volume=

    Solving inverse optimal control problems via value functions to global optimality , author=. Journal of Global Optimization , volume=

  254. [263]

    International conference on mathematical optimization theory and operations research , pages=

    Computing locally optimal solutions of the bilevel optimization problem using the kkt approach , author=. International conference on mathematical optimization theory and operations research , pages=. 2019 , organization=

    2019

  255. [264]

    arXiv preprint arXiv:2202.03684 , year=

    Efficiently escaping saddle points in bilevel optimization , author=. arXiv preprint arXiv:2202.03684 , year=

    arXiv

  256. [265]

    SIAM Journal on Optimization , volume=

    Convergent semidefinite programming relaxations for global bilevel polynomial optimization problems , author=. SIAM Journal on Optimization , volume=

  257. [266]

    Journal of Global Optimization , volume=

    A global optimization method for solving convex quadratic bilevel programming problems , author=. Journal of Global Optimization , volume=

  258. [267]

    Applied Mathematics and Computation , volume=

    A globally convergent algorithm for a class of bilevel nonlinear programming problem , author=. Applied Mathematics and Computation , volume=

  259. [268]

    Journal of Optimization theory and applications , volume=

    Descent approaches for quadratic bilevel programming , author=. Journal of Optimization theory and applications , volume=

  260. [269]

    Applied Sciences , volume=

    Bi-level programming approach for the optimal allocation of energy storage systems in distribution networks , author=. Applied Sciences , volume=

  261. [270]

    IEEE transactions on cybernetics , volume=

    Optimized assistive human--robot interaction using reinforcement learning , author=. IEEE transactions on cybernetics , volume=

  262. [271]

    The International Journal of Robotics Research , volume=

    Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences , author=. The International Journal of Robotics Research , volume=

  263. [272]

    Journal of Global Optimization , volume=

    Global solution of bilevel programs with a nonconvex inner program , author=. Journal of Global Optimization , volume=

  264. [273]

    Efficient gradient approximation method for constrained bilevel optimization , author=

  265. [274]

    Principled Penalty-based Methods for Bilevel Reinforcement Learning and

    Shen, Han and Yang, Zhuoran and Chen, Tianyi , booktitle=ICML, year=. Principled Penalty-based Methods for Bilevel Reinforcement Learning and

  266. [275]

    Optimization Letters , volume=

    Restricted strong convexity and its applications to convergence analysis of gradient-type methods in convex optimization , author=. Optimization Letters , volume=

  267. [276]

    arXiv preprint arXiv:2408.16087 , year=

    Unlocking Global Optimality in Bilevel Optimization: A Pilot Study , author=. arXiv preprint arXiv:2408.16087 , year=

    arXiv

  268. [277]

    arXiv preprint arXiv:2410.02387 , year=

    BiSSL: Bilevel Optimization for Self-Supervised Pre-Training and Fine-Tuning , author=. arXiv preprint arXiv:2410.02387 , year=

  269. [278]

    SEAL: Safety-enhanced Aligned

    Shen, Han and Chen, Pin-Yu and Das, Payel and Chen, Tianyi , booktitle=ICLR, year=. SEAL: Safety-enhanced Aligned

  270. [279]

    arXiv preprint arXiv:2410.09758 , year=

    BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation , author=. arXiv preprint arXiv:2410.09758 , year=

    arXiv

  271. [280]

    Bi-level alignment for cross-domain crowd counting , author=

  272. [281]

    Neural Computation , volume=

    Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation , author=. Neural Computation , volume=

  273. [282]

    Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions , author=

  274. [283]

    Practical bayesian optimization of machine learning algorithms , author=

  275. [284]

    arXiv preprint arXiv:2103.09985 , year=

    A deep learning theory for neural networks grounded in physics , author=. arXiv preprint arXiv:2103.09985 , year=

    arXiv

  276. [285]

    A primal-dual-assisted penalty approach to bilevel optimization with coupled constraints , author=

  277. [286]

    arXiv preprint arXiv:2406.01992 , year=

    Overcoming Lower-Level Constraints in Bilevel Optimization: A Novel Approach with Regularized Gap Functions , author=. arXiv preprint arXiv:2406.01992 , year=

    arXiv

  278. [287]

    2024 , organization=

    On finding small hyper-gradients in bilevel optimization: Hardness results and improved analysis , author=. 2024 , organization=

    2024

  279. [288]

    Functional Bilevel Optimization for Machine Learning , author=

  280. [289]

    2018 , publisher=

    Lectures on convex optimization , author=. 2018 , publisher=

    2018

  281. [290]

    arXiv preprint arXiv:2405.18881 , year=

    Tuning-Free Alignment of Diffusion Models with Direct Noise Optimization , author=. arXiv preprint arXiv:2405.18881 , year=

    arXiv

  282. [291]

    Journal of Machine Learning Research , volume=

    Learning to warm-start fixed-point optimization algorithms , author=. Journal of Machine Learning Research , volume=

  283. [292]

    IEEE/ACM international symposium on microarchitecture , pages=

    Neural acceleration for general-purpose approximate programs , author=. IEEE/ACM international symposium on microarchitecture , pages=. 2012 , organization=

    2012

  284. [293]

    Over-parameterized Model Optimization with Polyak- \ L \ ojasiewicz Condition , author=

  285. [294]

    On the convergence of encoder-only shallow transformers , author=

  286. [295]

    On mesa-optimization in autoregressively trained transformers: Emergence and capability , author=

  287. [296]

    Implicit regularization of deep residual networks towards neural ODEs , author=

  288. [297]

    Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization , author=

  289. [298]

    On the Optimization Landscape of Low Rank Adaptation Methods for Large Language Models , author=

  290. [299]

    Better Theory for SGD in the Nonconvex World , author=

  291. [300]

    Sharp analysis of stochastic optimization under global kurdyka-lojasiewicz inequality , author=

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.