An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation

Hok Shing Wong; Lea Bogensperger; Matthias J. Ehrhardt; Mohammad Sadegh Salehi; Thomas Pock

REVIEW 3 major objections 4 minor 1 cited by

An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper derives a fully computable a-posteriori error bound for the approximate hypergradient computed by primal-dual style differentiation, and an adaptive method that uses it to set tolerances and step sizes.

desk verdict Genuinely useful a-posteriori bound for inexact piggyback hypergradients, but Theorem 2's proof is missing two positive cross terms; repairable, but the current version should not be accepted as-is. read the letter →

arxiv 2412.06436 v3 pith:QU4LQL5R submitted 2024-12-09 math.OC cs.LG

Lea Bogensperger , Matthias J. Ehrhardt , Thomas Pock , Mohammad Sadegh Salehi , Hok Shing Wong This is my paper · ORCID

classification math.OCcs.LG MSC 90C3090C2665K1068T07

keywords bileveloptimizationhypergradienta-posteriorierrorboundprimal-dualmethodpiggybackdifferentiationadaptivestep-sizeinput-convexneuralnetworktotalvariation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a computable a-posteriori error bound for the hypergradient obtained by primal-dual style (piggyback) differentiation when the lower-level problem is solved only inexactly. It shows that the error of the approximate hypergradient can be controlled by residual-based terms involving the primal-dual residuals, tolerances for the lower-level and adjoint problems, and constants like strong convexity and Hessian Lipschitz constants. It then integrates this bound into an adaptive backtracking line-search method that dynamically adjusts tolerances and step sizes, proving convergence of the upper-level iterates. Experiments on learning TV discretizations and training input-convex neural networks for CT reconstruction support the approach.

What carries the argument

The machinery is the inexact piggyback method: the lower-level saddle-point problem (S) and its adjoint problem (ASI) are solved inexactly with PDHG, and the hypergradient is formed as z = ỹ ⊗ X̃ + Ȳ ⊗ x̃. The error analysis rests on Lemma 3, which gives residual-to-distance bounds for the adjoint problem using strong convexity, and Lemma 4, which establishes Lipschitz continuity of ∇²g with constant L_{∇²g*}(L_{∇g})³. Theorem 2 combines these to produce the fully computable bound (17), which is then used in the adaptive backtracking line search from the companion framework MAID to choose tolerances and step sizes.

What would settle it

Take a finite-dimensional quadratic lower-level problem where the saddle point and hypergradient have closed forms, compute the bound (17) with the true constants at several tolerance combinations, and verify it holds; a single violation would refute the theorem. Alternatively, on the ICNN-CT setup, compute the bound's right-hand side at the final iterate and compare with the actual gradient error estimated by a high-accuracy solve.

Watch

Extended reading notes

Core claim

The central discovery is Theorem 2: if the lower-level saddle-point solution is approximated within tolerances εx and εy, and the adjoint system is solved within residual-based tolerances δX and δY, then the approximate hypergradient z = ỹ ⊗ X̃ + Ȳ ⊗ x̃ satisfies an explicit bound, inequality (17), in terms of those tolerances and problem constants. The bound is a-posteriori and fully computable: it uses only residuals, tolerances, and known constants such as the strong convexity parameters and Lipschitz constants of gradients and Hessians, so it can serve as a stopping criterion. This gives, for primal-dual style bilevel differentiation, a principled way to choose how accurately the lower-level and adjoint problems must be solved to make progress in the upper-level optimization.

Load-bearing premise

The lower-level functions must be strongly convex with locally $C^{{2,1}}$ conjugates (Assumptions 1 and 3), and the smoothing-free TV experiment violates these, so the error bound and convergence result do not strictly apply there.

Editorial extensions

If this is right

The error bound enables adaptive stopping rules: tolerances for the lower-level and adjoint solves can be set from residuals, reducing computational cost while keeping the hypergradient accurate enough for descent.
The adaptive method, Algorithm 3 with the inexact sufficient-decrease condition of Lemma 5, guarantees convergence of the upper-level iterates: lim_{t→∞} ||∇L(K_t)|| = 0 under the stated assumptions.
In learning the discretization of total variation, the adaptive choice of step size yields smoother decrease of the upper-level loss and better reconstructions than fixed-step, fixed-tolerance piggyback, especially under a limited computational budget.
When training input-convex neural network regularizers for sparse-view CT, the bilevel-learned ICNN outperforms the adversarially trained ICNN with the same architecture, while the simpler Fields of Experts regularizer also gives strong results.
The framework is stated to extend directly to other linear inverse problems with data-driven, potentially non-smooth regularizers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A practical extension the paper leaves implicit: the bound (17) can serve as a per-iteration stopping criterion, so one could run the lower-level solver with loose tolerances early in training and tighten them only when the line-search condition fails, further reducing total cost.
The bound requires knowing or upper-bounding constants such as L_{∇²g*} and L_{∇²f}; in applications these are rarely known, so a usable implementation must estimate them, and the effectiveness of the adaptive method depends on those estimates being conservative enough.
The TV experiment violates the smoothness assumptions yet works well, suggesting the error bound and convergence theory may extend to non-smooth regularizers under weaker conditions; a direct test on a non-smooth problem with a known hypergradient would clarify whether the residual terms still control the error.
The convergence guarantee holds for deterministic upper-level optimization; combining the a-posteriori error control with variance-reduced or stochastic hypergradient estimates could make the method scalable to large data sets, a direction the paper lists as future work.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper's advertised a-posteriori bound for inexact piggyback hypergradients is a real contribution, but the proof of Theorem 2 as printed does not establish (17). The stress-test note is right: after substituting the triangle bounds, you pick up the cross terms C_X^1 ϵx ϵy + C_Y^2 ϵx ϵy, which are missing from the printed bound. Since (17) is the certificate that drives the adaptive tolerances, this is load-bearing. The good news is it is also easily repairable: add those two terms to the right-hand side, and the bound remains fully computable and the rest of the framework goes through. So the correct verdict is 'promising but needs a revision,' not 'wrong idea.'

What is actually new: an a-posteriori, computable error bound for the hypergradient computed via primal-dual style differentiation, extending [39] to this setting. Lemmas 3 and 4 are fine under the stated assumptions, and the adaptive backtracking routine is a reasonable way to turn the bound into step-size and tolerance control. The writing is clear and the attribution to [39] is honest: Theorem 3 is explicitly inherited, and the paper does not oversell it.

The softer spots are real but not fatal. The TV experiment violates the strong convexity and C^{2,1} assumptions, and the authors admit it; for a paper whose selling point is rigorous control, that is a gap between theory and demonstration. The experimental validation is narrow: one test image for CT, no error bars, no released code. That matters less for the math but limits the empirical claim. The convergence guarantee is a black-box import from the authors' earlier work, so the independent contribution is the bound, not the convergence theorem.

If I were the editor I would send this to peer review, but with the expectation of major revision. The referee should ask for the corrected bound, a discussion of whether the missing terms can be bounded or must be included, and stronger experiments (multiple images, error bars, or code). The paper deserves a serious referee because the underlying idea is likely correct and useful even if this version of the proof has a gap.

Referee Report

3 major / 4 minor

Summary. The paper studies bilevel optimization for learning linear operators in variational image reconstruction, where the lower-level problem is a convex saddle-point problem and the hypergradient is computed by a piggyback primal-dual differentiation scheme. The main theoretical contribution is an a-posteriori error bound (Theorem 2) for the inexactly computed hypergradient, expressed in terms of residuals, tolerances, and known constants, together with a transfer of the adaptive inexact descent framework of [39] to this setting (Theorem 3). The paper also reports experiments on learning total-variation discretizations and on training fields-of-experts and input-convex neural-network regularizers.

Significance. If Theorem 2 is correct, the paper fills a genuine gap: it provides a fully computable certificate for the accuracy of a piggyback-computed hypergradient, enabling principled tolerance control and adaptive step sizes in bilevel learning. The constants in (18) are explicit, the bound depends only on computable residuals and problem parameters, and the convergence result is imported from [39] rather than re-derived. The numerical results for the ICNN regularizer are encouraging: Table 1 reports 31.43 dB PSNR for the bilevel-trained ICNN versus 29.32 dB for ACR with the identical architecture. The main qualifications are that Eq. (17) must be corrected and that the experiments must either satisfy or explicitly disclaim the paper's regularity assumptions.

major comments (3)

[§3.1, Theorem 2, Eq. (17)] The final inequality (17) does not follow from the proof as written. Substituting (25)–(26) into the four-term triangle bound, together with ∥ˆX∥≤∥˜X∥+∥ΔX∥ and ∥ˆY∥≤∥˜Y∥+∥ΔY∥, yields, in addition to every term printed in (17), the cross terms C_X^1 ϵx ϵy + C_Y^2 ϵx ϵy. These terms are positive and are not bounded by the printed terms C_Y^1 (ϵx)^2 + C_X^2 (ϵy)^2 in general, because C_X^1 and C_Y^2 are independent constants that can dominate. The bound can be repaired by adding (C_X^1+C_Y^2)ϵx ϵy to the right-hand side; this preserves the a-posteriori and fully computable character, so the issue is correctable, but the central certificate as stated is not yet established.
[§4.1, Eq. (30) and surrounding text] The total-variation experiment is performed on the non-smooth problem (30), and the authors explicitly acknowledge that 'the non-smooth term does not fully satisfy the regularity assumptions imposed on g'. Since Lemma 3, Theorem 2, and the convergence transfer from [39] all require Assumptions 1 and 3 (strong convexity and local C^{2,1} regularity of the relevant functions), the results in Figures 1–3 do not provide an empirical check of the proved bound. The paper should either smooth the regularizer, for example with a Huber-type approximation, and rerun, or state clearly that this experiment is a heuristic illustration outside the theorem's hypotheses.
[§4.2, Eq. (33)] It is not clear that the ICNN lower-level problem satisfies Assumption 1. The term µg/2∥x∥² is quadratic only in x; the objective also contains δ_C(Vx,z) and γψ_w(Wz), and ψ_w has a linear tail for arguments larger than w. Thus there is no evident strong convexity in the joint variable (x,z), and the statement 'to ensure strong convexity of the primal function g' appears to require an additional argument or an additional quadratic penalty in z. As a consequence, the ICNN experiments, like the TV experiments, are outside the proved assumptions unless a joint strong-convexity argument is supplied.

minor comments (4)

[§3.1, Eq. (10b)] The notation 'δ2(Y,˜x.˜y)' contains a period instead of a comma; it should read 'δ2(Y,˜x,˜y)'.
[§3.1, Remark 3] The expression '∥∇1(ˆx)−∇1(˜x)∥' should read '∥∇ℓ1(ˆx)−∇ℓ1(˜x)∥'.
[§3.1, Algorithm 2, line 5] The stopping criterion for solving (ASI) should reference Lemma 3, i.e., Eq. (10) and the conditions (13), rather than (14)–(15), since (14)–(15) bound the distance to (ˆX,ˆY), not directly the residual to (¯X,¯Y).
[§3.2, Theorem 3] The statement of Theorem 3 should list the hypotheses needed from [39, Theorem 3.19]; in particular, Lemma 5 assumes that the upper-level loss ℓ is convex, but the theorem statement only mentions Assumptions 2 and 3.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: Theorem 2's a-posteriori bound is self-contained; convergence rests on a minor self-citation from [39].

full rationale

The central a-posteriori error bound in Theorem 2 is derived from first principles within the paper: the proof uses only Assumptions 1-3, Lemmas 1-4, and triangle inequalities applied to residuals and known constants (18a)-(18d). The bound (17) does not contain the target hypergradient ∇L(K) or any fitted parameter; it is a computable certificate based on measured residuals and problem constants. The adaptive step-size rule and convergence claim are imported from [39], a preprint by Salehi and Ehrhardt (two of the present authors) and two others; because that theorem is parameter-free with stated assumptions (Assumption 2 and 3) that do not include the target result, it qualifies as independent support under the review rules, making this at most a minor self-citation. Two non-circular concerns are separately noted: (i) Section 4.1 explicitly concedes that the total-variation regularizer 'does not fully satisfy the regularity assumptions imposed on g' yet proceeds without smoothing, so Theorem 2's assumptions are not strictly met in that experiment; (ii) the proof algebra for (17) may omit positive cross terms C^X_1 ϵx ϵy and C^Y_2 ϵx ϵy, which is a correctness risk rather than a circularity. Neither concern makes the derivation equivalent to its inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central error bound and convergence rely on several regularity assumptions. Assumptions 1 and 3 (strong convexity of g and f*, C^{2,1} regularity) are load-bearing: they guarantee uniqueness of the saddle point and Lipschitz control of Hessians in Theorem 2. The TV experiment breaks these assumptions and the paper proceeds anyway, so the theory does not strictly cover the flagship demonstration. The adaptive convergence theorem is taken from the authors' own prior work [39] as a black box. No free parameter is fitted to data to force the main result, but the algorithm's initial tolerances and step-size hyperparameters are hand-chosen and affect the reported budget comparisons.

free parameters (5)

Initial tolerances epsilon_0, delta_0 = epsilon_0=delta_0=1e-3 (TV); epsilon_0=20, delta_0=4 (ICNN); epsilon_0=delta_0=10 (FoE)
Hand-chosen initial accuracies for the lower-level and adjoint solvers; they control the computational budget and are not derived from data.
Strong convexity regularizer mu_g = 1e-8 (ICNN)
Added quadratic term to enforce strong convexity Assumption 1; chosen small to barely affect the reconstruction results.
Smoothing parameter w = 0.01
Width of the smooth activation psi_w in the ICNN and FoE regularizers; chosen by hand and affects the geometry of the regularizer.
Adaptive step-size hyperparameters rho, rhobar, nu, nubar = rho=0.5, rhobar=10/9, nu=0.5, nubar=1.05 (TV); rho=0.1, rhobar=1.1 (ICNN)
Taken from [39] or set by hand; they determine how aggressively step sizes and tolerances are increased or decreased during the line search.
TV regularization parameter lambda = not reported
The denoising experiment in Section 4.1 states lambda in R+ but never gives its value, leaving the exact lower-level problem underspecified.

assumptions (5)

domain assumption Assumption 1: f* and g are strongly convex
Ensures uniqueness of the saddle point and validity of the hypergradient formula; violated in the TV experiments.
domain assumption Assumption 3: f and g* are locally C^{2,1}
Needed for Lipschitz control of Hessians in Theorem 2; the TV regularizer in Section 4.1 is non-smooth.
domain assumption Assumption 2 and convexity of the upper-level loss in Lemma 5
The sufficient-decrease line search condition in Lemma 5 requires the upper-level loss to be convex, which is assumed but not verified for reconstruction losses.
standard math Alexandrov's theorem and Zhou's duality between strong convexity and gradient Lipschitzness
Used in Lemma 4 and Theorem 2 to relate Hessians of conjugate functions and to bound second-order differences.
standard math Theorem 3 from [39] (convergence of MAID)
The convergence of Algorithm 3 is imported as a black box from the authors' own prior work; it is not reproven in this paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation." pith.science (2026). https://pith.science/paper/QU4LQL5R

@misc{pith2026241206436,
  author       = {Pith},
  title        = {Pith review of: An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QU4LQL5R}},
  note         = {Machine review of arXiv:2412.06436}
}

read the original abstract

We consider a bilevel learning framework for learning linear operators. In this framework, the learnable parameters are optimized via a loss function that also depends on the minimizer of a convex optimization problem (denoted lower-level problem). We utilize an iterative algorithm called `piggyback' to compute the gradient of the loss and minimizer of the lower-level problem. Given that the lower-level problem is solved numerically, the loss function and thus its gradient can only be computed inexactly. To estimate the accuracy of the computed hypergradient, we derive an a-posteriori error bound, which provides guides for setting the tolerance for the lower-level problem, as well as the piggyback algorithm. To efficiently solve the upper-level optimization, we also propose an adaptive method for choosing a suitable step-size. To illustrate the proposed method, we consider a few learned regularizer problems, such as training an input-convex neural network.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Restarted contractive operators to learn at equilibrium
math.OC 2025-06 conditional novelty 6.0 of 10

ReTune proves that restarting a contractive truncated unrolled solver and backpropagating only through the final block yields gradients arbitrarily close to the deep-equilibrium hypergradient.

Reference graph

Works this paper leans on

58 extracted references · 50 canonical work pages · cited by 1 Pith paper

[39]

Salehi, M.S., Mukherjee, S., Roberts, L., Ehrhardt, M.J.: An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning (2025)

work page 2025
[1]

Physica D: nonlinear phenomena60(1-4), 259– 268 (1992)

Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena60(1-4), 259– 268 (1992)

work page 1992
[2]

SIAM Journal on Imaging Sciences3(3), 492–526 (2010)

Bredies, K., Kunisch, K., Pock, T.: Total gen- eralized variation. SIAM Journal on Imaging Sciences3(3), 492–526 (2010)

work page 2010
[3]

Communications on Pure and Applied Math- ematics: A Journal Issued by the Courant Institute of Mathematical Sciences57(11), 1413–1457 (2004)

Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Math- ematics: A Journal Issued by the Courant Institute of Mathematical Sciences57(11), 1413–1457 (2004)

work page 2004
[4]

SIAM review34(4), 561–580 (1992)

Hansen, P.C.: Analysis of discrete ill-posed problems by means of the l-curve. SIAM review34(4), 561–580 (1992)

work page 1992
[5]

Fundamentals of Algorithms

Hansen, P.C.: Discrete Inverse Problems: Insight and Algorithms. Fundamentals of Algorithms. Society for Industrial and Applied Mathematics, USA (2010)

work page 2010
[6]

Acta numerica27, 1–111 (2018)

Benning, M., Burger, M.: Modern regular- ization methods for inverse problems. Acta numerica27, 1–111 (2018)

work page 2018
[7]

Inverse Prob- lems36(12), 125014 (2020)

Bungert, L., Burger, M., Korolev, Y., Schön- lieb, C.-B.: Variational regularisation for inverse problems with imperfect forward oper- ators and general noise models. Inverse Prob- lems36(12), 125014 (2020)

work page 2020

Show all 58 references

[8]

IEEE Transactions on signal processing54(11), 4311–4322 (2006)

Aharon, M., Elad, M., Bruckstein, A.: K- svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing54(11), 4311–4322 (2006)

2006
[9]

IEEETransactionsonImageProcessing23(3), 1060–1072 (2014)

Chen, Y., Ranftl, R., Pock, T.: Insights into analysis operator learning: From patch- based sparse models to higher order mrfs. IEEETransactionsonImageProcessing23(3), 1060–1072 (2014)

2014
[10]

Inverse Problems36(6), 065005 (2020)

Li, H., Schwab, J., Antholzer, S., Haltmeier, M.: Nett: Solving inverse problems with deep neural networks. Inverse Problems36(6), 065005 (2020)

2020
[11]

Advances in neural information processing systems31(2018)

Lunz, S., Öktem, O., Schönlieb, C.-B.: Adversarial regularizers in inverse problems. Advances in neural information processing systems31(2018)

2018
[12]

arXiv preprint arXiv:2008.02839 (2020)

Mukherjee, S., Dittmer, S., Shumaylov, Z., Lunz, S., Öktem, O., Schönlieb, C.-B.: 15 Learned convex regularizers for inverse prob- lems. arXiv preprint arXiv:2008.02839 (2020)

2020 arXiv
[13]

IEEE transactions on medical imaging31(9), 1682–1697 (2012)

Xu, Q., Yu, H., Mou, X., Zhang, L., Hsieh, J., Wang, G.: Low-dose x-ray ct reconstruction via dictionary learning. IEEE transactions on medical imaging31(9), 1682–1697 (2012)

2012
[14]

Variational Methods: In Imaging and Geometric Control18(252), 2 (2017)

Calatroni, L., Cao, C., De Los Reyes, J.C., Schönlieb, C.-B., Valkonen, T.: Bilevel approaches for learning of variational imaging models. Variational Methods: In Imaging and Geometric Control18(252), 2 (2017)

2017
[15]

Foundations and Trends®in Signal Processing15(2-3), 121–289 (2022)

Crockett, C., Fessler, J.A.,et al.: Bilevel meth- ods for image reconstruction. Foundations and Trends®in Signal Processing15(2-3), 121–289 (2022)

2022
[16]

arXiv preprint arXiv:2305.18394 (2023)

Ehrhardt, M.J., Gazzola, S., Scott, S.J.: On optimal regularization parameters via bilevel learning. arXiv preprint arXiv:2305.18394 (2023)

2023 arXiv
[17]

SIAM Journal on Imaging Sciences 6(2), 938–983 (2013)

Kunisch, K., Pock, T.: A bilevel optimization approach for parameter learning in variational models. SIAM Journal on Imaging Sciences 6(2), 938–983 (2013)

2013
[18]

Jour- nal of Mathematical Imaging and Vision56, 175–194 (2016)

Ochs, P., Ranftl, R., Brox, T., Pock, T.: Tech- niques for gradient-based bilevel optimization with non-smooth lower level problems. Jour- nal of Mathematical Imaging and Vision56, 175–194 (2016)

2016
[19]

Neural computation12(8), 1889–1900 (2000)

Bengio, Y.: Gradient-based optimization of hyperparameters. Neural computation12(8), 1889–1900 (2000)

2000
[20]

IMA Journal of Applied Mathematics89(1), 254– 278 (2024)

Ehrhardt, M.J., Roberts, L.: Analyzing inex- act hypergradients for bilevel learning. IMA Journal of Applied Mathematics89(1), 254– 278 (2024)

2024
[21]

In: International Conference on Machine Learn- ing, pp

Franceschi, L., Frasconi, P., Salzo, S., Grazzi, R., Pontil, M.: Bilevel programming for hyper- parameter optimization and meta-learning. In: International Conference on Machine Learn- ing, pp. 1568–1577 (2018). PMLR

2018
[22]

In: International Conference on Machine Learning, pp

Grazzi, R., Franceschi, L., Pontil, M., Salzo, S.: On the iteration complexity of hypergradi- ent computation. In: International Conference on Machine Learning, pp. 3748–3758 (2020). PMLR

2020
[23]

In: International Conference on Machine Learning, pp

Pedregosa, F.: Hyperparameter optimization with approximate gradient. In: International Conference on Machine Learning, pp. 737–746 (2016). PMLR

2016
[24]

Advances in neural information processing systems34, 13537–13549 (2021)

Bolte, J., Le, T., Pauwels, E., Silveti-Falls, T.: Nonsmooth implicit differentiation for machine-learning and optimization. Advances in neural information processing systems34, 13537–13549 (2021)

2021
[25]

Advances in Neural Information Processing Systems35, 26404–26417 (2022)

Bolte, J., Pauwels, E., Vaiter, S.: Auto- matic differentiation of nonsmooth iterative algorithms. Advances in Neural Information Processing Systems35, 26404–26417 (2022)

2022
[26]

SIAM Journal on Imaging Sciences 16(4), 2202–2246 (2023)

Kofler, A., Altekrüger, F., Antarou Ba, F., Kolbitsch, C., Papoutsellis, E., Schote, D., Sirotenko, C., Zimmermann, F.F., Papafit- soros, K.: Learning regularization parameter- maps for variational image reconstruction using deep neural networks and algorithm unrolling. SIAM J...

2023
[27]

In: International Conference on Artificial Intelligence and Statistics, pp

Mehmood, S., Ochs, P.: Automatic differentia- tion of some first-order methods in parametric optimization. In: International Conference on Artificial Intelligence and Statistics, pp. 1584–1594 (2020). PMLR

2020
[28]

Bogensperger, L., Chambolle, A., Pock, T.: Convergence of a piggyback-style method for the differentiation of solutions of standard saddle-point problems. (2022)

2022
[29]

Chambolle, A., Pock, T.: Learning consistent discretizations of the total variation. (2021)

2021
[30]

In: Biegler, L.T., Heinkenschloss, M., Ghattas, O., Bloemen Waanders, B

Griewank, A., Faure, C.: Piggyback dif- ferentiation and optimization. In: Biegler, L.T., Heinkenschloss, M., Ghattas, O., Bloemen Waanders, B. (eds.) Large-Scale PDE-Constrained Optimization, pp. 148–164. Springer, Berlin, Heidelberg (2003) 16

2003
[31]

Suonperä, E., Valkonen, T.: Linearly con- vergent bilevel optimization with single-step inner methods (2023)

2023
[32]

Suonperä, E., Valkonen, T.: Single-loop meth- ods for bilevel parameter learning in inverse imaging (2025)

2025
[33]

In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K

Liu, B., Ye, M., Wright, S., Stone, P., liu: BOME! bilevel optimization made easy: A simple first-order approach. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022)

2022
[34]

In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J

Kwon, J., Kwon, D., Wright, S., Nowak, R.D.: Afullyfirst-ordermethodforstochasticbilevel optimization. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) Proceedings of the 40th International Conference on Machine Learning (2023)

2023
[35]

In: The Twelfth International Con- ference on Learning Representations (2024)

Kwon, J., Kwon, D., Wright, S., Nowak, R.D.: On penalty methods for nonconvex bilevel optimization and first-order stochastic approx- imation. In: The Twelfth International Con- ference on Learning Representations (2024)

2024
[36]

(eds.): Bilevel Opti- mization: Advances and Next Challenges

Dempe, S., Zemkoho, A. (eds.): Bilevel Opti- mization: Advances and Next Challenges. Springer Optimization and Its Applications, vol. 161. Springer, Cham (2020)

2020
[37]

arXiv preprint arXiv:1802.02246 (2018)

Ghadimi, S., Wang, M.: Approximation meth- ods for bilevel programming. arXiv preprint arXiv:1802.02246 (2018)

2018 arXiv
[38]

In: International Conference on Machine Learning, pp

Ji, K., Yang, J., Liang, Y.: Bilevel opti- mization: Convergence analysis and enhanced design. In: International Conference on Machine Learning, pp. 4882–4892 (2021). PMLR

2021
[40]

In: International Conference on Machine Learning, pp

Amos, B., Xu, L., Kolter, J.Z.: Input convex neural networks. In: International Conference on Machine Learning, pp. 146–155 (2017). PMLR

2017
[41]

Acta Numerica25, 161–319 (2016)

Chambolle, A., Pock, T.: An introduction to continuous optimization for imaging. Acta Numerica25, 161–319 (2016)

2016
[42]

Journal of math- ematical imaging and vision40, 120–145 (2011)

Chambolle, A., Pock, T.: A first-order primal- dual algorithm for convex problems with applications to imaging. Journal of math- ematical imaging and vision40, 120–145 (2011)

2011
[43]

arXiv preprint arXiv:1803.06573 (2018)

Zhou, X.: On the fenchel duality between strong convexity and lipschitz continuous gradient. arXiv preprint arXiv:1803.06573 (2018)

2018 arXiv
[44]

University of South Carolina, Columbia, South Carolina (1998)

Howard, R.: Alexandrov’s theorem on the second derivatives of convex functions via rademacher’s theorem on the first derivatives of lipschitz functions. University of South Carolina, Columbia, South Carolina (1998)

1998
[45]

Journal of mathematical imaging and vision63(5), 580–600 (2021)

Ehrhardt, M.J., Roberts, L.: Inexact derivative-free optimization for bilevel learn- ing. Journal of mathematical imaging and vision63(5), 580–600 (2021)

2021
[46]

Society for Industrial and Applied Math- ematics (SIAM), Philadelphia, PA (2017)

Beck, A.: First-Order Methods in Optimiza- tion. Society for Industrial and Applied Math- ematics (SIAM), Philadelphia, PA (2017)

2017
[47]

SIAM Journal on Imaging Sciences 3(4), 1015–1046 (2010)

Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM Journal on Imaging Sciences 3(4), 1015–1046 (2010)

2010
[48]

Computational Opti- mization and Applications76(2), 381–430 (2020)

Rasch, J., Chambolle, A.: Inexact first-order primal–dual algorithms. Computational Opti- mization and Applications76(2), 381–430 (2020)

2020
[49]

SIAM Journal on Imaging Sciences10(3), 1258–1290 (2017)

Condat, L.: Discrete total variation: New def- inition and minimization. SIAM Journal on Imaging Sciences10(3), 1258–1290 (2017)

2017
[50]

In: International Conference on Scale Space and Variational Methods in Computer Vision, 17 pp

Bogensperger, L., Chambolle, A., Effland, A., Pock, T.: Learned discretization schemes for the second-order total generalized variation. In: International Conference on Scale Space and Variational Methods in Computer Vision, 17 pp. 484–497 (2023). Springer

2023
[51]

IEEE transactions on pattern analysis and machine intelligence33(5), 898– 916 (2010)

Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence33(5), 898– 916 (2010)

2010
[52]

In: Proceedings of the IEEE International Con- ference on Computer Vision, pp

He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Con- ference on Computer Vision, pp. 1026–1034 (2015)

2015
[53]

Medical physics43(6Part35), 3759–3760 (2016)

McCollough, C.: Tu-fg-207a-04: overview of the low dose ct grand challenge. Medical physics43(6Part35), 3759–3760 (2016)

2016
[54]

arXiv preprint arXiv:2410.12441 (2024)

Wong, H.S., Ehrhardt, M.J., Mukherjee, S.: A primal-dual algorithm for image reconstruction with icnns. arXiv preprint arXiv:2410.12441 (2024)

2024
[55]

In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol

Roth, S., Black, M.J.: Fields of experts: A framework for learning image priors. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 2, pp. 860–867 (2005). IEEE

2005
[56]

IEEE transactions on medical imaging37(6), 1322–1332 (2018)

Adler, J., Öktem, O.: Learned primal-dual reconstruction. IEEE transactions on medical imaging37(6), 1322–1332 (2018)

2018
[57]

arXiv preprint arXiv:1711.05101 (2017)

Loshchilov, I.: Decoupled weight decay reg- ularization. arXiv preprint arXiv:1711.05101 (2017)

2017 arXiv
[58]

Advances in neural information processing systems32(2019) 18

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learn- ing library. Advances in neural information processing systems32(2019) 18

2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [39]

Salehi, M.S., Mukherjee, S., Roberts, L., Ehrhardt, M.J.: An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning (2025)

work page 2025

[2] [1]

Physica D: nonlinear phenomena60(1-4), 259– 268 (1992)

Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena60(1-4), 259– 268 (1992)

work page 1992

[3] [2]

SIAM Journal on Imaging Sciences3(3), 492–526 (2010)

Bredies, K., Kunisch, K., Pock, T.: Total gen- eralized variation. SIAM Journal on Imaging Sciences3(3), 492–526 (2010)

work page 2010

[4] [3]

Communications on Pure and Applied Math- ematics: A Journal Issued by the Courant Institute of Mathematical Sciences57(11), 1413–1457 (2004)

Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Math- ematics: A Journal Issued by the Courant Institute of Mathematical Sciences57(11), 1413–1457 (2004)

work page 2004

[5] [4]

SIAM review34(4), 561–580 (1992)

Hansen, P.C.: Analysis of discrete ill-posed problems by means of the l-curve. SIAM review34(4), 561–580 (1992)

work page 1992

[6] [5]

Fundamentals of Algorithms

Hansen, P.C.: Discrete Inverse Problems: Insight and Algorithms. Fundamentals of Algorithms. Society for Industrial and Applied Mathematics, USA (2010)

work page 2010

[7] [6]

Acta numerica27, 1–111 (2018)

Benning, M., Burger, M.: Modern regular- ization methods for inverse problems. Acta numerica27, 1–111 (2018)

work page 2018

[8] [7]

Inverse Prob- lems36(12), 125014 (2020)

Bungert, L., Burger, M., Korolev, Y., Schön- lieb, C.-B.: Variational regularisation for inverse problems with imperfect forward oper- ators and general noise models. Inverse Prob- lems36(12), 125014 (2020)

work page 2020