Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

A stochastic gradient method for trilevel optimization

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes TSG, presented as the first stochastic gradient method for unconstrained trilevel optimization, and proves that, under smoothness, strong convexity of the two inner problems, and a bias condition on inverted stochastic…

desk verdict First stochastic trilevel method with a real convergence theorem, but the theory does not cover the implemented variants; worth refereeing with major revision. read the letter →

arxiv 2505.06805 v1 pith:TEQBKSUX submitted 2025-05-11 math.OC cs.LGstat.ML

Tommaso Giovannelli , Griffin Dean Kent , Luis Nunes Vicente This is my paper · ORCID
classification math.OCcs.LGstat.ML MSC 90C2690C3090C15
keywords trileveloptimizationstochasticgradientdescentadjointimplicitdifferentiationnonconvexhyperparametertuningadversariallearningbilevel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Trilevel optimization nests three optimization problems; until now its stochastic variants had no convergence theory. This paper proposes TSG, a three-loop stochastic gradient method that alternates lower-, middle-, and upper-level updates, and proves that the expected squared gradient norm of the upper objective converges to zero at rate O(J/√I) for nonconvex upper problems when the two lower levels are strongly convex. The theorem is designed to absorb every practical source of error: inexact solves of the middle and lower problems, an inexactly computed adjoint gradient, and noisy estimates of gradients, Hessians, Jacobians, and third-order tensors. The authors also give two practical implementations, one using finite-difference Hessian-vector products and one using truncated Neumann series with automatic differentiation, and demonstrate the method on synthetic trilevel problems and on trilevel adversarial hyperparameter tuning. If the convergence proof holds, stochastic trilevel learning, such as hyperparameter tuning combined with adversarial robustness, becomes a problem class with guarantees rather than a heuristic exercise.

What carries the argument

The load-bearing object is the trilevel adjoint gradient, $$\nabla f = (\nabla_x f_1 - \$nabla^{2}$_{xz} f_3 (\$nabla^{2}$_{zz} f_3)^{-1} \nabla_z f_1) - \$nabla^{2}$_{xy} \bar{f} (\$nabla^{2}$_{yy} \bar{f})^{-1}(\nabla_y f_1 - \$nabla^{2}$_{yz} f_3 (\$nabla^{2}$_{zz} f_3)^{-1} \nabla_z f_1),$$ which reduces the nested problem to a single-level objective by implicit differentiation of the two optimality conditions. The method never computes this quantity exactly; each stochastic estimate carries bias from the approximate middle and lower solutions and from noisy or approximated second- and third-order derivatives. The proof controls that bias with a Lyapunov function $V_i = f(x_i) + \|y_i - y(x_i)\|^2 + \|z_i - z(x_i)\|^2 + \|z_i - z(x_i,y_i)\|^2$, whose expected decrease telescopes over the three loops into the $O(J/\sqrt{I})$ bound. The practical variants replace Hessian inverses either by conjugate-gradient solves with finite-difference Hessian-vector products (TSG-N-FD) or by truncated Neumann series evaluated through automatic differentiation (TSG-AD).

What would settle it

Measure, on a problem satisfying Assumptions 3.1–3.5, the actual bias $\| [\nabla^2_{zz} f_3]^{-1} - \mathbb{E}[[\nabla^2_{zz} f_3^{\xi}]^{-1} | \mathcal{F}_\xi] \|$ during TSG-N-FD or TSG-AD runs: if it does not decay as $O(\alpha_i \beta_i \gamma_i)$, Assumption 3.6 fails and Theorem 3.1 does not apply to the tested algorithms.

Watch

Extended reading notes

Core claim

The central claim is that the trilevel optimization problem can be solved by a stochastic gradient method with a convergence guarantee: under Assumptions 3.1–3.6, with step sizes $\alpha_i = 1/\sqrt{I}$, $\beta_i = (1/\sqrt{J})\alpha_i$, $\gamma_i = (1/(\sqrt{J}\sqrt{K}))\alpha_i$, and with iteration counts satisfying $\varsigma \le J$, $\varpi \le I$, and $K \ge \Xi(I,J) = O(J^3 I)$, the iterates satisfy $\frac{1}{I}\sum_{i=0}^{I-1} \mathbb{E}[\|\nabla f(x_i)\|^2] = O(J/\sqrt{I})$. A companion result improves this to $O(1/\sqrt{I})$ when step sizes are chosen using more problem-specific constants, at the price of requiring $K \ge O(J^4 I)$. The convergence statement is built around the inexact trilevel adjoint gradient, so bias from approximate inner solutions, approximate adjoint computation, and stochastic derivative estimation is covered by the same analysis. The paper also shows the method working on quadratic and quartic synthetic trilevel problems and on a trilevel formulation of adversarial hyperparameter tuning, where the full trilevel formulation is reported to be the most consistent performer across three tabular datasets compared with two bilevel ablations.

Load-bearing premise

The proof assumes that the inverses of the noisy Hessian estimates are biased by at most $O(\theta)$ with $\theta=\alpha\beta\gamma$, but the two practical algorithms whose experiments are reported never verify this bias bound.

Editorial extensions

If this is right

  • Stochastic trilevel learning problems, such as joint hyperparameter and adversarial tuning, now have an algorithm with a formal convergence rate rather than only heuristic support.
  • The step-size recipe $\alpha_i=1/\sqrt{I}$, $\beta_i=(1/\sqrt{J})\alpha_i$, $\gamma_i=(1/(\sqrt{J}\sqrt{K}))\alpha_i$ is directly implementable, with the lower level needing $K \ge O(J^3 I)$ inner iterations to keep the overall rate.
  • Because the analysis covers biased and inexact gradients, mini-batch sampling of Hessians and third-order tensors is theoretically permissible, not just sampling of gradients.
  • The tighter $O(1/\sqrt{I})$ rate shows the $O(J/\sqrt{I})$ bound is an artifact of simple step sizes, not a limitation intrinsic to trilevel problems.
  • On the three tabular datasets tested, keeping all three levels produced more consistent test error than removing either the upper or lower level, suggesting the full trilevel formulation is useful when both hyperparameter tuning and adversarial robustness matter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Assumption 3.6 were verified for the two practical implementations by choosing the Neumann truncation length and the finite-difference parameter as explicit functions of $\theta=\alpha\beta\gamma$, Theorem 3.1 would directly certify the algorithms whose numerical results are reported; as written, the guarantee covers an idealized estimator of the same type.
  • The requirement $K \ge O(J^3 I)$ suggests a scaling law for multi-level methods: adding upper levels multiplies the inner-level iteration budget polynomially, so in four-level problems the lowest-level cost would likely grow even faster. A four-level experiment could separate the paper's specific result from a general hierarchical principle.
  • The comparison between the trilevel formulation and its two bilevel ablations suggests a practical decision rule: include the third level only when both hyperparameter tuning and adversarial robustness matter, since the bilevel variants save runtime but were more dataset-sensitive. This rule is an inference from the reported data pattern, and image-classification tasks remain untested.
  • A testable extension is to replace the strong-convexity requirements on the middle and lower problems with penalty or regularization techniques, following recent bilevel practice; if the Lyapunov argument could be preserved without strong convexity, the method would cover nonconvex inner problems.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes TSG, a stochastic gradient method for unconstrained trilevel optimization, together with a convergence analysis. The algorithm updates the lower-level variables by stochastic gradient steps, the middle-level variables by an inexact bilevel stochastic gradient, and the upper-level variables by a noisy trilevel adjoint gradient. The main result, Theorem 3.1, states that under Assumptions 3.1–3.6 and with step sizes α_i=1/√I, β_i=(1/√J)α_i, γ_i=(1/(√J√K))α_i, the average squared gradient norm satisfies (1/I)ΣE[‖∇f(x_i)‖²]=O(J/√I), provided the iteration counts obey ς≤J, ϖ≤I, and Ξ(I,J)=O(J³I)≤K. The paper also presents two practical implementations, TSG-N-FD and TSG-AD, and reports numerical experiments on synthetic trilevel problems and on a trilevel adversarial hyperparameter tuning formulation.

Significance. If correct, Theorem 3.1 would provide the first convergence guarantee for a stochastic trilevel optimization method, extending the bilevel stochastic approximation framework. The main proof is detailed and appears internally consistent under the stated assumptions, and the paper supplies reproducible code. However, the advertised scope is currently wider than what the theorem actually covers: the practical algorithms used in the experiments are not shown to satisfy Assumption 3.6, and one of the synthetic test problems violates the strong convexity assumption. The result is therefore significant but needs substantial tightening before the main claims are fully supported.

major comments (4)
  1. [Assumption 3.6, Appendix F.2–F.3] Assumption 3.6 requires the inverted stochastic Hessian estimators [∇²_zz f_3^ξ]^{-1} and [∇²_yy f̄^ξ]^{-1} to have bias of order O(θ), with θ=αβγ. This assumption is load-bearing for Theorem 3.1, since Lemmas D.1, D.3, and D.5 all inherit it. However, the two implemented methods do not provide such estimators. TSG-AD approximates the inverse Hessians by a truncated Neumann series with a fixed accuracy level Q (Algorithms 5–6 and equations F.9, F.11); for fixed Q the truncation error is a constant independent of θ, and no schedule forcing Q to grow with I,J,K is specified. TSG-N-FD solves adjoint systems by CG with finite-difference Hessian-vector products and a fixed epsilon=0.1 (equation F.5), and neither the CG stopping tolerance nor the FD error is tied to θ. Consequently Theorem 3.1 does not cover the algorithms whose numerical performance is reported. The authors should either verify Assumption 3.6 for these implementations with explicit accuracy schedules, or state and prove a convergence theorem for the actually implemented estimators.
  2. [Section 3.2, Remark 3.1] Remark 3.1 claims a stronger O(1/√I) rate under more problem-dependent step sizes, referring to Theorem B.2.5 in the PhD thesis [26], which is listed as 'in preparation'. This result is advertised as part of the paper's convergence theory, but no proof is included or available for verification. Either provide the proof in the appendix or clearly mark the O(1/√I) rate as a conjecture/result to appear elsewhere.
  3. [Appendix E] Propositions E.1 and E.2 state Lipschitz continuity properties of z(x,y), y(x), ∇_y f̄, ∇²_xy f̄, ∇²_yy f̄, ∇f, and related Jacobians, but the proofs are deferred to Appendix B.5 of the PhD thesis [26]. These properties are used throughout the proof of Theorem 3.1 (e.g., equations E.1–E.13 in Lemmas B.2–B.4 and in the main proof). Since [26] is an in-preparation self-citation, the current manuscript is not self-contained on a load-bearing component of the central convergence result. The authors should include the proofs or cite a publicly available source.
  4. [Appendix F.4, equation (F.16)] The quartic synthetic problem uses f_3(x,y,z)=0.5‖zᵀH_zz z−zᵀH_zx x−zᵀH_zy y‖². The paper itself notes in Appendix F.4 that this lower-level problem has two optimal solutions, z=0 and z=H_zx x+H_zy y, and therefore f_3 is not μ_z-strongly convex in z. This directly violates Assumption 3.2, which is used in the convergence analysis. The statement that the initial points ensure convergence to one of the two solutions does not restore strong convexity. The quartic experiments should either use a strongly convex lower-level problem or include a separate local strong-convexity analysis for the relevant solution branch.
minor comments (4)
  1. [Table 2, Section 4.2] The numerical experiments use decaying step sizes α_i=ᾱ/i, β_j=β̄/j, γ_k=γ̄/k, whereas Theorem 3.1 is proved for constant step sizes α_i=1/√I, β_i=(1/√J)α_i, γ_i=(1/(√J√K))α_i. The paper should state explicitly that the experiments are illustrative and are not a direct verification of the theorem's step-size schedule.
  2. [Assumption 3.4] There is a typo in Assumption 3.4: 'idententically' should be 'identically'.
  3. [Figures 3–4] In Figure 3 the left-hand y-axis labels appear to contain overlapping tick values ('2, 0, 2'); the plot would be clearer with standard tick formatting. Similar minor formatting issues appear in other figures.
  4. [Section 4.3, Appendix F.5.1] The comparison with the trilevel formulation of [38] is only briefly reported in Appendix F.5.1; since this comparison is used to motivate the authors' formulation, a short statement in the main text summarizing the outcome would improve readability.

Circularity Check

2 steps flagged · score 4.0 of 10

Core O(J/√I) theorem is derived self-contained from Assumptions 3.1–3.6; circularity burden is limited to Lipschitz lemmas and the stronger-rate remark deferred to the authors' own in-preparation thesis.

  1. self citation load bearing [Appendix E, introductory paragraph before Proposition E.1]
    "All of their corresponding proofs are provided in Appendix B.5 of the PhD thesis [26]."

    Theorem 3.1's proof relies on Lemmas B.1–B.4, whose proofs in Appendix C invoke the Lipschitz continuity properties stated in Propositions E.1 and E.2 (equations (E.1)–(E.13)). Those propositions are load-bearing: they bound z(x), z(x,y), y(x), ∇f, ∇y f, and the Hessian terms that appear throughout the Lyapunov decomposition. The paper gives no proof for them, instead deferring all proofs to reference [26], which is the authors' own PhD thesis and is marked 'in preparation'. The derivation chain therefore contains a central step whose only support in this manuscript is an unchecked self-citation.

  2. self citation load bearing [Remark 3.1, Section 3.2]
    "Such a result also does not require lower-bounds on the UL or ML variables I and J, but requires K ≥ O(J4I). The formal statement of this result is given by Theorem B.2.5 in the PhD thesis [26]."

    The paper advertises a stronger convergence rate, (1/I)Σ E[||∇f(x_i)||^2] = O(1/√I), as a theorem-like result, but the formal statement and (presumably) its proof are located only in the authors' own PhD thesis [26], which is listed as 'in preparation'. Thus the advertised stronger claim is not proved in the paper at all; its support reduces to a self-citation. This is not the central O(J/√I) result, which is proved in Appendix C.5, so it raises the circularity score moderately rather than decisively.

full rationale

The central convergence theorem, Theorem 3.1, is not circular in the sense of assuming its own conclusion: it is a conditional statement proved from Assumptions 3.1–3.6, with the proof of the step-size schedule and the telescoping argument given in Appendix C.5. The step sizes are chosen explicitly, the constants ς, ϖ, and Ξ(I,J) are defined in the proof, and the final O(J/√I) rate follows from the stated inequalities. Assumption 3.6, which postulates that the inverted stochastic Hessian estimators have bias O(θ), is an input assumption rather than a fitted parameter, so the theorem is not a renamed version of the assumption. There is also no fitted-input-called-prediction pattern: the numerical experiments are benchmarked against f(x*) or against BLO variants, and no convergence quantity is fit and then reported as a prediction. The main circularity burden comes from self-citation. Propositions E.1 and E.2, which supply the Lipschitz properties of the response maps and of the trilevel gradient, are stated without proof and deferred to the authors' own PhD thesis [26], 'in preparation'. These properties are used in the proofs of Lemmas B.1–B.4 and hence in Theorem 3.1, so a load-bearing part of the derivation chain is not independently verifiable in the manuscript. Similarly, Remark 3.1's stronger O(1/√I) rate is deferred entirely to Theorem B.2.5 in the same thesis. These are genuine self-citations that carry part of the argument, but they do not make the central claim equivalent to its inputs: the Lipschitz lemmas are standard implicit-function results, and the main theorem has substantial independent proof content. The gap between Assumption 3.6 and the practical TSG-AD/TSG-N-FD implementations (fixed Neumann truncation Q and fixed FD step ε=0.1) is a correctness/scope concern, not a circularity concern, because the theorem is honest about being conditional on that assumption. Overall, a score of 4 reflects 'some self-citation; central claim still has independent content' rather than a higher score that would require the theorem itself to reduce by construction to a fit or to an unchecked self-citation chain.

Assumptions & free parameters 4 free parameters · 8 assumptions · 0 invented entities

The convergence theorem is conditional on a set of smoothness, strong convexity, unbiasedness, and bias assumptions. No new physical or mathematical entities are introduced. The practical algorithms introduce hand-chosen parameters that are not part of the theorem.

free parameters (4)
  • Per-algorithm step-size prefactors ᾱ, β̄, γ̄ = Various, e.g., 0.3, 0.2, 0.1 for TSG-H on the quadratic problem (Table 2)
    Selected by grid search over {0.1, 0.01, 0.001} for each algorithm and dataset; these choices affect the reported convergence curves and the comparisons.
  • Finite-difference parameter ε = 0.1
    Chosen by hand in Appendix F.2 for TSG-N-FD; affects the accuracy of Hessian-vector product approximations.
  • Truncation order Q for Neumann series = Unspecified
    TSG-AD Algorithms 5-6 depend on an accuracy level Q that is not given a value in the paper, so the implementation is not fully specified.
  • Penalty coefficient c in Ψ(δ) = 0.1
    Hand-chosen regularization strength in the adversarial tuning formulation (F.18); impacts the reported robustness results.
assumptions (8)
  • domain assumption Assumption 3.1: f1 once continuously differentiable, f2 twice, f3 thrice, with Lipschitz gradients, Hessians, and third-order tensors.
    Standard smoothness assumptions in stochastic optimization; they make the trilevel adjoint gradient well-defined and Lipschitz.
  • domain assumption Assumption 3.2: f3 is µz-strongly convex in z.
    Ensures a unique lower-level solution z(x,y); it is violated by the quartic test problem (F.16).
  • domain assumption Assumption 3.3: f̄ is µy-strongly convex in y.
    Ensures a unique middle-level solution y(x); restricts applicability, as the paper acknowledges in the conclusion.
  • domain assumption Assumption 3.4: stochastic derivatives are unbiased with bounded variance.
    Standard assumption for SGD; the variance bounds enter throughout the convergence proof.
  • domain assumption Assumption 3.5: inverted stochastic Hessians are uniformly bounded in norm.
    Standard in stochastic Newton methods; the constants bzz and byy appear in the proof.
  • ad hoc to paper Assumption 3.6: bias of inverted stochastic Hessians is O(θ) with θ=αβγ.
    This bridges ideal and practical Hessian inverses, but the paper does not prove it for the finite-difference or Neumann-series approximations used in TSG-N-FD and TSG-AD.
  • standard math Implicit function theorem applied to ∇_z f3=0 and ∇_y f̄=0.
    Used in Proposition A.1 to derive the trilevel adjoint gradient (2.2) and the Jacobians (A.3)-(A.8).
  • ad hoc to paper Lipschitz properties in Propositions E.1-E.2 are stated without proofs.
    The convergence proof depends on these Lipschitz bounds, but the proofs are deferred to the in-preparation PhD thesis [26].

how reviews work

0 comments
Cite this review

Pith. "Pith review of A stochastic gradient method for trilevel optimization." pith.science (2026). https://pith.science/paper/TEQBKSUX

@misc{pith2026250506805,
  author       = {Pith},
  title        = {Pith review of: A stochastic gradient method for trilevel optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TEQBKSUX}},
  note         = {Machine review of arXiv:2505.06805}
}
read the original abstract

With the success that the field of bilevel optimization has seen in recent years, similar methodologies have started being applied to solving more difficult applications that arise in trilevel optimization. At the helm of these applications are new machine learning formulations that have been proposed in the trilevel context and, as a result, efficient and theoretically sound stochastic methods are required. In this work, we propose the first-ever stochastic gradient descent method for solving unconstrained trilevel optimization problems and provide a convergence theory that covers all forms of inexactness of the trilevel adjoint gradient, such as the inexact solutions of the middle-level and lower-level problems, inexact computation of the trilevel adjoint formula, and noisy estimates of the gradients, Hessians, Jacobians, and tensors of third-order derivatives involved. We also demonstrate the promise of our approach by providing numerical results on both synthetic trilevel problems and trilevel formulations for hyperparameter adversarial tuning.

Figures

Figures reproduced from arXiv: 2505.06805 by the authors.

Figure 1
Figure 1. Quadratic problem, deterministic case. 0 2 4 6 8 10 UL Iterations 0 1 2 3 4 5 f(x i) 1e 1 UL, ML, and LL grad std devs: 0, 0, 0 ML and LL Hess std devs: 0, 0 TSG-H TSG-N-FD TSG-AD 0.0 0.5 1.0 1.5 2.0 Time (ms) 1e2 0 1 2 3 4 5 f(x i) 1e 1 UL, ML, and LL grad std devs: 0, 0, 0 ML and LL Hess std devs: 0, 0 TSG-H TSG-N-FD TSG-AD [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 3
Figure 3. Quadratic problem, stochastic case (low noise: two left plots; high noise: two right plots). [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Quartic problem, stochastic case (low noise: two left plots; high noise: two right plots). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Adversarial learning formula￾tion (F.18), red wine quality dataset. 0 25 50 75 100 125 150 175 200 UL Iterations 1 2 3 4 5 Test MSE 1e2 TSG-AD BSG-AD (without UL) BSG-AD (without LL) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Time (s) 1e1 1 2 3 4 5 Test MSE 1e2 TSG-A…
Figure 7
Figure 7. Figure 7: Adversarial learning formulation (F.18), California housing dataset. In this paper, we proposed the first stochastic first-order method for trilevel optimization along with a rigorous convergence theory for the non￾convex setting. The proposed theory also cov￾ers all f…
Figure 8
Figure 8. Figure 8: Breakdown of the algorithms, quadratic problem, deterministic case. [PITH_FULL_IMAGE:figures/full_fig_p046_8.png]
Figure 9
Figure 9. Figure 9: Breakdown of the algorithms, quartic problem, deterministic case. [PITH_FULL_IMAGE:figures/full_fig_p046_9.png]
Figure 10
Figure 10. Figure 10: Trilevel adversarial learning formulation proposed in [ [PITH_FULL_IMAGE:figures/full_fig_p047_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On computing Goldstein approximate second-order stationary points of structured nonsmooth nonconvex programs

    math.OC 2026-07 conditional novelty 8.0 of 10

    A randomized first-order algorithm computes Goldstein approximate second-order stationary points of L-smooth nonconvex functions with oracle complexity Õ(ΔL⁸n²/ε⁹ + ΔL⁶n³/ε⁷).

  2. First-order Constrained Trilevel Optimization Over Distributed Networks for Robust Coreset Selection

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A federated first-order algorithm for constrained trilevel optimization is applied to distributed robust coreset selection, with a claimed O(ε^{-3/2}) rate to ε-stationarity.

Reference graph

Works this paper leans on

45 extracted references · 33 canonical work pages · cited by 2 Pith papers

  1. [26]

    G. D. Kent. Stochastic Methods for Multi-Level and Multi-Objective Optimization. PhD thesis, Lehigh University, Department of Industrial and Systems Engineering, 2025, in preparation

    work page 2025

  2. [1]

    Arguello, E

    B. Arguello, E. S. Johnson, and J. L. Gearhart. A trilevel model for segmentation of the power transmission grid cyber network. IEEE Syst. J., 17:419–430, 2023

    work page 2023

  3. [2]

    J. F. Bard. An investigation of the linear three level programming problem. IEEE Trans. Syst. Man Cybern., SMC-14:711–717, 1984

    work page 1984

  4. [3]

    J. F. Bard and J. E. Falk. An explicit solution to the multi-level programming problem. Comput. Oper. Res., 9:77–100, 1982

    work page 1982

  5. [4]

    A. Beck. First-Order Methods in Optimization. SIAM-Society for Industrial and Applied Mathematics, Philadelphia, PA, 2017

    work page 2017

  6. [5]

    H. P. Benson. On the structure and properties of a linear multilevel programming problem. J. Optim. Theory Appl., 60:353–373, 1989

    work page 1989

  7. [6]

    C. Blair. The computational complexity of multi-level linear programs. Ann. Oper. Res., 34: 13–19, 1992

    work page 1992

  8. [7]

    Bollapragada, R

    R. Bollapragada, R. H. Byrd, and J. Nocedal. Exact and inexact subsampled Newton methods for optimization. IMA Journal of Numerical Analysis, 39:545–578, 04 2018

    work page 2018

Show all 45 references
  1. [8]

    Bottou, F

    L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 60:223–311, 2018

    2018

  2. [9]

    L. C. Cang and A. Petrusel. Krasnoselski-Mann iterations for hierarchical fixed point problems for a finite family of nonself mappings in Banach spaces. J. Optim. Theory Appl., 146:617–639, 2010

    2010

  3. [10]

    C. Chen, X. Chen, C. Ma, Z. Liu, and X Liu. Gradient-based bi-level optimization for deep learning: A survey, 2023

    2023

  4. [11]

    T. Chen, Y . Sun, and W. Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. In Advances in Neural Information Processing Systems, volume 34, pages 25294–25307. Curran Associates, Inc., 2021

    2021

  5. [12]

    T. Chen, Y . Sun, Q. Xiao, and W. Yin. A Single-Timescale Method for Stochastic Bilevel Optimization. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151, pages 2466 – 2488. PMLR, March 2022

    2022

  6. [13]

    S. K. Choe, W. Neiswanger, P. Xie, and E. Xing. Betty: An automatic differentiation library for multilevel optimization. arXiv e-prints, art. arXiv:2207.02849, July 2022

    2022 arXiv

  7. [14]

    Cortez, Antonio Luíz Cerdeira, Fernando Almeida, Telmo Matos, and José Reis

    P. Cortez, Antonio Luíz Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decis. Support Syst., 47: 547–553, 2009

    2009

  8. [15]

    A. M. Fathollahi-Fard, M. Hajiaghaei-Keshteli, and S. Mirjalili. Hybrid optimizers to solve a tri-level programming model for a tire closed-loop supply chain network design problem. Applied Soft Computing, 70:701–722, 2018

    2018

  9. [16]

    Franceschi, M

    L. Franceschi, M. Donini, P. Frasconi, and M. Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proc. 34th Int. Conf. Mach. Learn. (ICML) , volume 70 of Proceedings of Machine Learning Research, pages 1165–1173. PMLR, 06–11 Aug 2017

    2017

  10. [17]

    Ghadimi and M

    S. Ghadimi and M. Wang. Approximation methods for bilevel programming. arXiv e-prints, art. arXiv:1802.02246, February 2018

    2018 arXiv

  11. [18]

    Giovannelli, G

    T. Giovannelli, G. D. Kent, and L. N. Vicente. Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems. ISE Technical Report 21T-025, Lehigh University, December 2022

    2022

  12. [19]

    Giovannelli, G

    T. Giovannelli, G. D. Kent, and L. N. Vicente. Bilevel optimization with a multi-objective lower-level problem: risk-neutral and risk-averse formulations. Optim. Methods Softw., 39: 756–778, 2024. 10

    2024

  13. [20]

    M. Guo, Y . Yang, R. Xu, Z. Liu, and D. Lin. When NAS meets robustness: In search of robust architectures against adversarial attacks. arXiv e-prints, art. arXiv:1911.10695, November 2019

    1911 arXiv

  14. [21]

    Y . Guo, C. Guo, and J. Yang. A tri-level optimization model for power systems defense considering cyber-physical interdependence. IET Gener. Transm. Distrib., 17:1477–1490, 2023

    2023

  15. [22]

    K. Ji, J. Yang, and Y . Liang. Bilevel optimization: Convergence analysis and enhanced design. arXiv e-prints, art. arXiv:2010.07962, October 2020

    2010 arXiv

  16. [23]

    Y . Jiao, K. Yang, T. Wu, C. Jian, and J. Huang. Provably convergent federated trilevel learning. arXiv e-prints, art. arXiv:2312.11835, December 2023

    2023 arXiv

  17. [24]

    Y . Jiao, K. Yang, and C. Jian. Unlocking trilevel learning with level-wise zeroth order con- straints: Distributed algorithms and provable non-asymptotic convergence. arXiv e-prints, art. arXiv:2412.07138, December 2024

    2024 arXiv

  18. [25]

    X. Jin, J. Wang, J. Slocum, M. Yang, S. Dai, S. Yan, and J. Feng. RC-DARTS: Resource constrained differentiable architecture search. arXiv e-prints, art. arXiv:1912.12814, December 2019

    1912 arXiv

  19. [27]

    K. Lai, M. Illindala, and K. Subramaniam. A tri-level optimization model to mitigate coordinated attacks on electric power systems in a cyber-physical environment. Appl. Energy, 235:204–218, 2019

    2019

  20. [28]

    H. Liduka. Iterative algorithm for solving triple-hierarchical constrained optimization problem. J. Optim. Theory Appl., 148:580–592, 2011

    2011

  21. [29]

    H. Liu, K. Simonyan, and Y . Yang. DARTS: Differentiable architecture search. ArXiv, arXiv:1806.09055, June 2019

    2019 arXiv

  22. [30]

    R. Liu, J. Gao, J. Zhang, D. Meng, and Z. Lin. Investigating bi-Level optimization for learning and vision from a unified perspective: A survey and beyond. arXiv e-prints, art. arXiv:2101.11517, January 2021

    2021 arXiv

  23. [31]

    J. Lu, J. Han, Y . Hu, and G. Zhang. Multilevel decision-making: A survey.Information Sciences, 346-347:463–487, 2016

    2016

  24. [32]

    Madry, A

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv e-prints, art. arXiv:1706.06083, June 2017

    2017 arXiv

  25. [33]

    Nesterov.Lectures on Convex Optimization

    Y . Nesterov.Lectures on Convex Optimization. Springer Publishing Company, Incorporated, New York, 2nd edition, 2018

    2018

  26. [34]

    R. K. Pace and R. Barry. Sparse spatial autoregressions. Stat. Probab. Lett., 33:291–297, 1997

    1997

  27. [35]

    Rahdar, L

    M. Rahdar, L. Wang, and G. Hu. A tri-level optimization model for inventory control with uncertain demand and lead time. Int. J. Prod. Econ., 195:96–105, 2018

    2018

  28. [36]

    W. Rudin. Principles of Mathematical Analysis . McGraw-Hill Book Company, Inc., New York-Toronto-London, 1953

    1953

  29. [37]

    Saheya, C

    B. Saheya, C. T. Nguyen, and J.-S. Chen. Neural network based on systematically generated smoothing functions for absolute value equation. J. Appl. Math. Comput., 61:533–558, 2019

    2019

  30. [38]

    R. Sato, M. Tanaka, and A. Taked. A gradient method for multilevel optimization. In Adv. Neural Inf. Process. Syst., volume 34, pages 7522–7533. Curran Associates, Inc., 2021

    2021

  31. [39]

    Shafiei, V

    A. Shafiei, V . Kungurtsev, and J. Marecek. Trilevel and multilevel optimization using monotone operator theory. Math. Methods Oper. Res., 99:77–114, 2024

    2024

  32. [40]

    S. L. Tilahun, S. M. Kassa, and H. C. Ong. A new algorithm for multilevel optimization problems using evolutionary strategy, inspired by natural adaptation. In PRICAI 2012: Trends in Artificial Intelligence, pages 577–588. Springer Berlin Heidelberg, 2012. 11

    2012

  33. [41]

    Ue-Pyng and W

    W. Ue-Pyng and W. F. Bialas. The hybrid algorithm for solving the three-level linear program- ming problem. Comput. Oper. Res., 13:367–377, 1986

    1986

  34. [42]

    L. N. Vicente and P. H. Calamai. Bilevel and multilevel programming: A bibliography review. J. Global Optim., 5:291–306, 1994

    1994

  35. [43]

    Wu and A

    X. Wu and A. J. Conejo. An efficient tri-level optimization model for electric grid defense planning. IEEE Trans. Power Syst., 32:2984–2994, 2017

    2017

  36. [44]

    X. Xu, Z. Meng, and R. Shen. A tri-level programming model based on conditional value-at-risk for three-stage supply chain management. Comput. Ind. Eng., 66:470–475, 2013

    2013

  37. [45]

    N” in the algorithm name refers to the Newton-type system defined by the adjoint equation, while the “FD

    Y . Yao, T. Edmunds, D. Papageorgiou, and R. Alvarez. Trilevel optimization in power network defense. IEEE Trans. Syst. Man Cybern. C Appl. Rev., 37:712–718, 2007. 12 Technical Appendices A Derivation of the trilevel adjoint gradient This appendix contains the formal statement...

    2007

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.