Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

Learning to Optimize by Differentiable Programming

T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This tutorial argues that differentiable programming turns first-order optimization algorithms into trainable, differentiable modules whose parameters can be learned end-to-end.

desk verdict A broad but shallow tutorial that rehashes standard material and overclaims 'learning' — the actual implementations are fixed first-order methods in PyTorch, and the one numerical validation is circular. read the letter →

arxiv 2601.16510 v4 pith:VPOGYORW submitted 2026-01-23 cs.MS cs.LGmath.OC

Liping Tao , Xindi Tong , Chee Wei Tan This is my paper
classification cs.MScs.LGmath.OC MSC 90C2590C4665K0568T07
keywords differentiableprogrammingfirst-orderoptimizationFenchel-RockafellardualityADMMprimal-dualhybridgradientconelearningtooptimizenonnegativeleastsquares
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that differentiable programming—building programs that are themselves differentiable—lets optimization algorithms be treated as trainable components rather than fixed solvers. By embedding classic first-order methods like ADMM and PDHG in automatic-differentiation frameworks, it claims, the algorithm's parameters and even its update structure can be learned from data, improving convergence and solution quality. Fenchel–Rockafellar duality provides the saddle-point structure that makes this possible: primal and dual variables are updated by differentiable steps, and the dual problem supplies a certificate of optimality. The tutorial demonstrates this recipe on the running nonnegative least-squares example and on case studies in linear programming, optimal power flow, neural-network verification, and Laplacian-regularized minimization. A sympathetic reader would take away that optimization algorithms are no longer black boxes but differentiable components that can adapt to problem structure and be tuned end-to-end.

What carries the argument

The central object is the primal–dual saddle-point reformulation of a convex cone program, typically obtained through Fenchel–Rockafellar duality. This reformulation turns the original constrained problem into an unconstrained min-max problem whose updates are proximal or gradient steps on primal and dual variables—ADMM, PDHG, or projected primal–dual gradient. Inside an automatic-differentiation framework, this iterative loop is a differentiable program: reverse-mode automatic differentiation propagates gradients backward through the iterates, so that free parameters such as step sizes, the ADMM penalty ρ, and the dual variables themselves can be learned or adjusted end-to-end. The Fenchel

What would settle it

Solve an LR-NNLS instance with a non-trivial active set, attempt to determine the scalar constant from complementary slackness and feasibility alone (without peeking at the primal solution), and compare the reconstructed x to a reference solver's output; if the constant is underdetermined, the reconstruction will fail.

Watch

Extended reading notes

Core claim

The central claim is that a first-order optimization algorithm can be re-expressed as a differentiable program, and that enclosing it in an automatic-differentiation framework creates a mechanism for learning its design. The theoretical engine is duality: Fenchel–Rockafellar duality converts a convex constrained problem into a min-max saddle-point form whose iterates are proximal or gradient steps on primal and dual variables. Because each step is differentiable, reverse-mode automatic differentiation can propagate gradients backward through the entire iterative loop, allowing step sizes, penalty parameters, and dual variables to be adjusted by training. The authors establish this through de

Load-bearing premise

The Laplacian-regularized case study assumes that the scalar constant in the dual-to-primal recovery formula is uniquely fixed by primal feasibility and complementary slackness, but no derivation is given and the only numerical check fits that constant using the known answer.

Editorial extensions

If this is right

  • The same solver code can double as a differentiable layer inside a neural network, so downstream learning objectives can directly tune optimization hyperparameters without manual gradient derivations.
  • Duality provides a certificate of solution quality during training, enabling principled stopping criteria based on duality gap rather than iteration count.
  • The framework supports parallel and GPU-accelerated execution of decomposed dual subproblems, as illustrated in the distributed nonnegative least-squares example, potentially scaling to very large cone programs.
  • The recipe extends across problem families—LPs, QPs, SDP relaxations—meaning a single differentiable programming approach can replace bespoke solver wrappers in end-to-end pipelines.
  • Training signals can flow through both primal and dual variables, allowing a dual solution to be refined iteratively until it is certified, which is useful when high-accuracy primal solutions are needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's demonstrations implement standard algorithms (ADMM, PDHG, projected gradient) and do not yet train any of their parameters from data; the central 'learning to design' claim would be tested by experiments showing learned step sizes or penalties outperforming fixed ones on unseen problem instances.
  • The dual-to-primal recovery formula in the Laplacian-regularized case study is validated by fitting the scalar constant to the known solution; a non-circular check would compute that constant from complementary slackness and active-set information alone.
  • If the recovery formula is correct, a practical method could solve Laplacian-regularized problems entirely in the dual space, avoiding costly pseudoinverse factorizations; this is worth testing on large graphs where the pseudoinverse is infeasible.
  • The tutorial's framework suggests that automatic differentiation through iterative solvers could serve as a generic mechanism for algorithm discovery, but supporting that broader claim will require evidence of learned algorithms, not just reimplementations of classical ones.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This tutorial argues that differentiable programming frameworks (PyTorch, JAX, TensorFlow) can be used not only to execute first-order optimization algorithms but also to "learn how to design" them. It reviews automatic differentiation, cone programming, Lagrangian and Fenchel duality, ADMM, PDHG, and presents PyTorch implementations for NNLS and for case studies spanning the Stigler diet problem, neural network verification, optimal power flow, and Laplacian-regularized NNLS. The paper's stated thesis is that embedding first-order methods in automatic-differentiation frameworks allows end-to-end training that improves convergence and solution quality, and that the case studies demonstrate these gains. As submitted, however, the manuscript does not actually learn any algorithm: the only "learning" routine treats fixed Lagrangian variables as PyTorch parameters and runs Adam, while the case studies execute hand-specified iterations. In addition, a core Fenchel-conjugate derivation is mathematically incorrect, and the one numerical validation of a duality-based recovery formula is circular.

Significance. If the claims were established, this would be a useful tutorial bridging differentiable programming and convex optimization, with reproducible PyTorch code. The paper has genuine strengths: a broad survey of differentiable-programming frameworks and optimization packages, a correct Lagrangian derivation of the NNLS dual, and concrete ADMM/PDHG implementations with accompanying source code. However, the central claim about learning to design algorithms is not demonstrated, a key conjugate computation is invalid, and the only recovery validation is circular. The current value of the manuscript is therefore primarily expository rather than as evidence for the learning-to-optimize thesis.

major comments (5)
  1. [§3.3.2, Eq. (36)] The computation of the Fenchel conjugate g*(y) drops the term -1/2||Ax-(y+b)||^2 inside a supremum over x≥0. This is valid only if inf_{x≥0}||Ax-(y+b)||^2=0 for every y, i.e., y+b∈range(A) with attainment in the nonnegative orthant. No such range condition is stated. Consequently, the displayed dual max_y -g*(y) is not generally equal to the Lagrangian dual (NNLS-Lagrangian-Dual); the essential constraint A^T y≥0 is missing. This error propagates to the Fenchel saddle-point representation and to the PDHG update in Example 3.5. The conjugate computation and the resulting dual must be corrected.
  2. [§4.4, Eq. (92)] The representation x* = L†(μ* - A^T λ*) + c·1 is asserted, with c "uniquely determined by primal feasibility and complementary slackness," but no derivation is given. The numerical validation fits c from the known CVXPY solution x* as c=(1/n)1^T(x* - z); z is not even defined in this section. The reported reconstruction error of 10^-6 therefore only shows that x* - L†(μ* - A^T λ*) is nearly constant, not that c can be recovered from the KKT conditions alone. The uniqueness/recoverability claim needs a proof and a validation that does not use the true primal solution.
  3. [Abstract and §3.5–§4] The paper's central thesis is that differentiable programming can "learn how to design" optimization algorithms and that end-to-end training improves convergence and solution quality. The only training routine, learn_nnls_dual_by_backprop (§3.5), initializes λ, μ, z and runs Adam on the fixed Lagrangian of one NNLS instance; no update rule, step size, preconditioner, or proximal operator is learned. The Section 4 case studies run fixed ADMM/PDHG/projected-gradient iterations with hand-chosen parameters, e.g., Eqs. (83)–(86) in §4.3. There is no training set, no held-out evaluation, and no comparison against an untrained baseline. The stated thesis is therefore not demonstrated. The paper should either be reframed as a tutorial on implementing known algorithms or include genuine learned-algorithm experiments.
  4. [§3.4.1, Eq. (42)] In Example 3.3 the Lagrangian is ℒ = f(x) - μ^T x with μ≥0, so ∇_μ ℒ = -x. The dual problem is a maximization, so the projected dual update should be μ^{k+1} = Π(μ^k + σ x^{k+1}). Equation (42) instead uses μ^{k+1} = Π(μ^k - σ x^{k+1}), which is a descent step for the dual and is incorrect as written. This sign error affects the PDG algorithm described in the tutorial.
  5. [§4.1 and §4.2] The Stigler Diet and Neural Network Verification case studies contain only problem formulations and references to external tools; no differentiable-programming implementation, convergence plot, or numerical result is reported for either. Table 2 nevertheless claims that these case studies illustrate the benefits of the framework. At minimum, the authors should either provide actual experiments or state explicitly that these are purely illustrative formulations with no empirical evaluation.
minor comments (5)
  1. [Throughout] Fenchel–Rockafellar is consistently misspelled as "Fenchel–Rockafeller." Also, "computation graphs, also as known Kantorovich graphs" is ungrammatical.
  2. [§3.5 code block] The function recover_x_from_mu_active_set is called with parameters tau and max_refine that are not defined in the displayed code or the surrounding text. The reader cannot reproduce the primal recovery step without additional detail.
  3. [§4.4] In the validation sentence, z is used in c=(1/n)1^T(x* - z) but z is never defined. It should be z = L†(μ* - A^T λ*).
  4. [References] There are duplicate references for PyTorch ([96] and [136]) and for CVX ([77] and [78]). The reference list should be consolidated.
  5. [§3.3.2, Eq. (33)] The line inf_{x≥0}{y^T A x} is not well defined for general y: if any component of A^T y is negative, the infimum is -∞. This ambiguity is part of the problem that makes Eq. (36) invalid and should be clarified in the revision.

Circularity Check

1 steps flagged · score 6.0 of 10

Dual-to-primal recovery in §4.4 is validated with a constant fitted to the target solution; the rest of the tutorial's mathematics is standard and not circular.

  1. fitted input called prediction [Section 4.4, after Eq. (92)]
    "where μ⋆ and λ⋆ denote the optimal dual solutions, and the scalar c is uniquely determined by primal feasibility and the complementary slackness conditions. To numerically validate (92), we solve an LR-NNLS instance with(m, n) = (50, 30) using CVXPY and reconstruct x⋆ from the dual solutions via a least-squares estimate of c, namely c = 1/n 1⊤(x⋆ − z). The resulting reconstruction error is on the order of 10−6, confirming the theoretical consistency."

    Eq. (92) claims x* = L†(μ*−Aᵀλ*) + c1 with c determined by KKT conditions. The numerical validation instead computes c = (1/n)1ᵀ(x*−z), i.e., c is fitted from the CVXPY solution x* that the formula is supposed to recover from dual variables alone. The 10⁻⁶ error therefore only shows that x*−L†(μ*−Aᵀλ*) is nearly constant; it does not test whether primal feasibility and complementary slackness determine c. The 'reconstruction' is a fit to the target, so the reported validation is forced by construction.

full rationale

The core tutorial mathematics—computation-graph backpropagation, Lagrangian/Fenchel duals of NNLS, ADMM and PDHG updates, and the OPF/NNV formulations—is standard textbook material with external grounding and is not circular. The only concrete circular step is the §4.4 validation: the constant c in the closed-form primal recovery is estimated from the true CVXPY solution, turning the claimed KKT-based recovery into a fitted reconstruction. Self-citations ([155], [156], [173]) are present but not load-bearing: they are cited as examples/background, and the paper's derivations do not rest on them. Separately, the headline claim that the paper 'learns how to design' optimization algorithms is not demonstrated—the only 'learning' routine (learn_nnls_dual_by_backprop) runs Adam on the Lagrangian of one fixed NNLS instance and no update rule or step-size policy is learned—but this is a claim-evidence gap, not a circular derivation, so it does not change the circularity score beyond the one fitted validation.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its main free parameter is the null-space constant c in the LR-NNLS validation, which is fitted to the known solution. The key axioms are standard strong-duality, graph-connectivity, and modeling assumptions, plus an unflagged range assumption in the Fenchel conjugate derivation. The invented-entities list is empty because the paper reuses existing concepts (dual variables, Laplacian pseudoinverse, etc.).

free parameters (2)
  • c (LR-NNLS null-space constant) = c = (1/n) 1^T (x* - z), computed from the CVXPY solution
    In §4.4, Eq. (92) introduces x* = L†(μ* - A^T λ*) + c1. The text claims c is uniquely determined by KKT conditions, but the numerical validation estimates c from the known true solution x* using least squares. This makes the reported 10^-6 reconstruction error a fit, not a prediction.
  • ADMM/PDHG hyperparameters (ρ, ρ_c, τ, σ, θ, learning rate, iteration counts) = ρ=1.0, ρ_c=1.0, lr=1e-2, iters=500–3000, etc.
    These are chosen by hand in the code blocks without sensitivity analysis. The convergence plots and claimed performance depend on these unstated default choices, and no tuning procedure is described.
assumptions (5)
  • domain assumption Slater's condition holds for NNLS and the cone programs considered, so strong duality holds and the primal optimum is recoverable from the dual.
    Invoked in §3.2.3 and §3.5.1 to justify solving the dual and recovering primal solutions. This is standard for feasible convex cone programs but is not verified for the specific instances.
  • domain assumption The graph Laplacian L is connected, so its null space is span{1} and the solution form x* = -L† g + c1 is valid.
    Used in §4.4, Eq. (89). Connectivity of the graph is not stated, and the derivation of the solution decomposition depends on it.
  • domain assumption The dual of the nonconvex neural network verification problem provides valid certificates with no duality gap.
    In §4.2, the Lagrangian dual (NNV-Dual) is used to certify properties of ReLU networks. NNV is nonconvex, so the dual may be weak; the paper does not discuss regularity conditions under which the lower bound is tight.
  • ad hoc to paper The OPF case study is well modeled by the resistive-network surrogate with nodal power p_i(v) = v_i(Gv)_i and a quadratic loss v^T G v.
    In §4.3, the paper replaces AC optimal power flow with a convex Laplacian-conductance model and explicitly 'temporarily disregards' voltage bounds (Constraint 77). This is a modeling choice that is not validated against standard OPF benchmarks.
  • ad hoc to paper In Example 3.2, inf_{x≥0} 1/2 ||Ax - (y+b)||² = 0 for every y, i.e., y+b ∈ range(A) is actually attained over the nonnegative orthant.
    Eq. (36) computes the Fenchel conjugate g*(y) by setting the infimum over x≥0 to zero, which assumes a range condition that is not stated and does not hold in general. This invalidates the displayed Fenchel dual max_y -g*(y).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Optimize by Differentiable Programming." pith.science (2026). https://pith.science/paper/VPOGYORW

@misc{pith2026260116510,
  author       = {Pith},
  title        = {Pith review of: Learning to Optimize by Differentiable Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VPOGYORW}},
  note         = {Machine review of arXiv:2601.16510}
}
read the original abstract

Solving massive-scale optimization problems requires scalable first-order methods with low per-iteration cost. This tutorial highlights a shift in optimization: using differentiable programming not only to execute algorithms but to learn how to design them. Modern frameworks such as PyTorch, TensorFlow, and JAX enable this paradigm through efficient automatic differentiation. Embedding first-order methods within these systems allows end-to-end training that improves convergence and solution quality. Guided by Fenchel-Rockafellar duality, the tutorial demonstrates how duality-informed iterative schemes such as the alternating direction method of multipliers, and the primal-dual hybrid gradient can be learned and adapted through representative case studies.

Figures

Figures reproduced from arXiv: 2601.16510 by the authors.

Figure 1
Figure 1. Learning to Optimize via Differentiable Programming by Combining Duality and First-order Methods. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Backpropagation in PyTorch via the Chain Rule: The forward pass computes [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Convergence comparison of PDG, ADMM, and PDHG on the NNLS problem [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: ADMM with PyTorch. The primal problem is reformulated into a decomposed dual problem, whose [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Learning with PyTorch. The dual problem is reformulated as minimizing its Lagrangian, treated as a [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Convergence of the learning-based method on the NNLS problem [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Multi-GPU ADMM implementation in PyTorch. The dual problem is decomposed into multiple sub [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Controlled Periodic Synchronization for Efficient Data-Parallel Training

    cs.DC 2026-07 conditional novelty 4.0 of 10

    Periodic gradient+parameter synchronization with SlowMo beats DDP by 2.44 pp (K=4) on a WAN while cutting average wall-clock time by 13.8%, but only under a fixed LR=0.1 protocol.

  2. Learning-Based Spectrum Cartography in Low Earth Orbit Satellite Networks: An Overview

    cs.NI 2026-05 unverdicted novelty 3.0 of 10

    The paper overviews attention-based learning methods for spectrum cartography in LEO satellite networks to enable adaptive fusion of heterogeneous measurements for inference and resource allocation.

Reference graph

Works this paper leans on

188 extracted references · 20 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe- mawat, Geoffrey Irving, Michael Isard, et al. 2016. TensorFlow: A system for large-scale machine learning. In12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265–283

    2016

  2. [2]

    Martín Abadi and Gordon D Plotkin. 2019. A simple differentiable programming language.Proceedings of the ACM on Programming Languages4, POPL (2019), 1–28. 8Source code is available athttps://github.com/convexsoft/diffprog. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Learning to Optimize by Differentiable Programming 111:31

    2019

  3. [3]

    Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. 2019. Differen- tiable convex optimization layers.Advances in Neural Information Processing Systems32 (2019)

    2019

  4. [4]

    Akshay Agrawal and Stephen Boyd. 2020. Differentiating through log-log convex programs. arXiv preprint arXiv:2004.12553 (2020)

    arXiv 2020

  5. [5]

    Akshay Agrawal, Stephen Boyd, Deepak Narayanan, Fiodar Kazhamiaka, and Matei Zaharia. 2022. Allocation of fungible resources via a fast, scalable price discovery method.Mathematical Programming Computation14, 3 (2022), 593–622

    2022

  6. [6]

    Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. 2018. A rewriting system for convex optimization problems.Journal of Control and Decision5, 1 (2018), 42–60

    2018

  7. [7]

    Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. 2023. Optimus: Optimization modeling using mip solvers and large language models.arXiv preprint arXiv:2310.06116(2023)

    arXiv 2023

  8. [8]

    Aws Albarghouthi et al. 2021. Introduction to neural network verification.Foundations and Trends® in Programming Languages 7, 1–2 (2021), 1–157

    2021

Show all 188 references
  1. [9]

    Rita Ammanouil, André Ferrari, and Cédric Richard. 2015. A graph Laplacian regularization for hyperspectral data unmixing.In 2015IEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing(ICASSP) .IEEE,1637–1641

    2015

  2. [10]

    Brandon Amos. 2019. Differentiable optimization-based modeling for machine learning. (2019)

    2019

  3. [11]

    Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and J Zico Kolter. 2018. Differentiable mpc for end-to-end planning and control.Advances in Neural Information Processing Systems31 (2018)

    2018

  4. [12]

    Brandon Amos and J Zico Kolter. 2017. Optnet: Differentiable optimization as a layer in neural networks. InInter- national Conference on Machine Learning. PMLR, 136–145

    2017

  5. [13]

    Erling D Andersen and Knud D Andersen. 2000. The MOSEK interior point optimizer for linear programming: an implementation of the homogeneous algorithm. InHigh Performance Optimization. Springer, 197–232

    2000

  6. [14]

    MarcinAndrychowicz,MishaDenil,SergioGomez,MatthewWHoffman,DavidPfau,TomSchaul,BrendanShilling- ford, and Nando De Freitas. 2016. Learning to learn by gradient descent by gradient descent.Advances in Neural Information Processing Systems29 (2016)

    2016

  7. [16]

    David Applegate, Mateo Díaz, Oliver Hinder, Haihao Lu, Miles Lubin, Brendan O’Donoghue, and Warren Schudy

  8. [17]

    AnewinsightonaugmentedLagrangianmethodwithapplications in machine learning.Journal of Scientific Computing99, 2 (2024), 53

    JianchaoBai,LinyuanJia,andZhengPeng.2024. AnewinsightonaugmentedLagrangianmethodwithapplications in machine learning.Journal of Scientific Computing99, 2 (2024), 53

    2024

  9. [18]

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2019. Deep equilibrium models.Advances in Neural Information Processing Systems32 (2019)

    2019

  10. [19]

    Friedrich L Bauer. 1974. Computational graphs and rounding error.SIAM J. Numer. Anal.11, 1 (1974), 87–96

    1974

  11. [20]

    Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018. Automatic differentiation in machine learning: A survey.Journal of Machine Learning Research18, 153 (2018), 1–43

    2018

  12. [21]

    2017.First-order methods in optimization

    Amir Beck. 2017.First-order methods in optimization. SIAM

    2017

  13. [22]

    Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. 2021. Machine learning for combinatorial optimization: a methodological tour d’horizon.European Journal of Operational Research290, 2 (2021), 405–421

    2021

  14. [23]

    Bentkamp, R

    A. Bentkamp, R. Fernández Mir, and J. Avigad. 2023. Verified reductions for optimization. InTools and Algorithms for the Construction and Analysis of Systems, Vol. 13994. Springer, 74–92

    2023

  15. [24]

    Quentin Bertrand, Quentin Klopfenstein, Mathurin Massias, Mathieu Blondel, Samuel Vaiter, Alexandre Gramfort, and Joseph Salmon. 2022. Implicit differentiation for fast hyperparameter selection in non-smooth convex learning. Journal of Machine Learning Research23, 149 (2022), 1–43

    2022

  16. [25]

    2014.Constrained optimization and Lagrange multiplier methods

    Dimitri P Bertsekas. 2014.Constrained optimization and Lagrange multiplier methods. Academic press

    2014

  17. [26]

    Mathieu Blondel and Vincent Roulet. 2024. The elements of differentiable programming. arXiv preprint arXiv:2403.14606 (2024)

    2024 arXiv

  18. [27]

    Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. 2020. Fast differentiable sorting and ranking. In International Conference on Machine Learning. PMLR, 950–959

    2020

  19. [28]

    Jonathan Borwein and Adrian Lewis. 2006. Convex Analysis and Nonlinear Optimization: Theoryand Examples. Springer

    2006

  20. [29]

    Nacime Bouziani, David A Ham, and Ado Farsi. 2024. Differentiable programming across the PDE and machine learning barrier.arXiv preprint arXiv:2409.06085(2024)

    2024 arXiv

  21. [30]

    Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers.Foundations and Trends® in Machine learning 3, 1 (2011), 1–122. J. ACM, Vol. 37, No. 4, Art...

    2011

  22. [31]

    2004.Convex optimization

    Stephen P Boyd and Lieven Vandenberghe. 2004.Convex optimization. Cambridge university press

    2004

  23. [32]

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. 2018. JAX: composable transformations of Python+ NumPy programs. (2018)

    2018

  24. [33]

    Robin A Brown, Edward Schmerling, Navid Azizan, and Marco Pavone. 2022. A unified view of SDP-based neural network verification through completely positive programming. InInternational Conference on Artificial Intelligence and Statistics. PMLR, 9334–9355

    2022

  25. [34]

    Deng Cai, Xiaofei He, Jiawei Han, and Thomas S Huang. 2010. Graph regularized nonnegative matrix factorization for data representation.IEEE Transactions on Pattern Analysis and Machine Intelligence33, 8 (2010), 1548–1560

    2010

  26. [35]

    Raafi Careem, G Johar, and Ali Khatibi. 2024. Deep neural networks optimization for resource-constrained envi- ronments: Techniques and models.Indonesian Journal of Electrical Engineering and Computer Science33, 3 (2024), 1843–1854

    2024

  27. [36]

    Antonin Chambolle and Thomas Pock. 2011. A first-order primal-dual algorithm for convex problems with applica- tions to imaging.Journal of Mathematical Imaging and Vision40, 1 (2011), 120–145

    2011

  28. [37]

    Tsung-Hui Chang, Mingyi Hong, Wei-Cheng Liao, and Xiangfeng Wang. 2016. Asynchronous distributed ADMM for large-scale optimization—Part I: Algorithm and convergence analysis.IEEE Transactions on Signal Processing64, 12 (2016), 3118–3130

    2016

  29. [38]

    Fei Chen, Gene Cheung, and Xue Zhang. 2024. Manifold graph signal restoration using gradient graph Laplacian regularizer. IEEE Transactions on Signal Processing72 (2024), 744–761

    2024

  30. [39]

    Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. 2022. Learning to optimize: A primer and a benchmark.Journal of Machine Learning Research23, 189 (2022), 1–59

    2022

  31. [40]

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In13th USENIX Symposium on Op...

    2018

  32. [41]

    Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cuDNN: Efficient primitives for deep learning.arXiv preprint arXiv:1410.0759(2014)

    2014 arXiv

  33. [42]

    Hong-Ming Chiu, Hao Chen, Huan Zhang, and Richard Y Zhang. 2025. SDP-CROWN: Efficient bound propagation for neural network verification with tightness of semidefinite programming.arXiv preprint arXiv:2506.06665(2025)

    2025 arXiv

  34. [43]

    Ontheconvergenceanalysisofthedecentralizedprojectedgradientdescent method

    WoocheolChoiandJimyeongKim.2025. Ontheconvergenceanalysisofthedecentralizedprojectedgradientdescent method. SIAM Journal on Optimization35, 3 (2025), 1673–1702

    2025

  35. [44]

    1990.Optimization and nonsmooth analysis

    Frank H Clarke. 1990.Optimization and nonsmooth analysis. SIAM

    1990

  36. [45]

    Gonzalo E Constante-Flores, Hao Chen, and Can Li. 2025. Enforcing hard linear constraints in deep learning models with decision rules.arXiv preprint arXiv:2505.13858(2025)

    2025

  37. [46]

    Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 (2016)

    2016 arXiv

  38. [47]

    Ron S Dembo, Stanley C Eisenstat, and Trond Steihaug. 1982. Inexact newton methods.SIAM J. Numer. Anal.19, 2 (1982), 400–408

    1982

  39. [48]

    Jelena Diakonikolas, Chenghui Li, Swati Padmanabhan, and Chaobing Song. 2022. A fast scale-invariant algorithm for non-negative least squares with non-negative data.Advances in Neural Information Processing Systems35 (2022), 6264–6277

    2022

  40. [49]

    Steven Diamond and Stephen Boyd. 2016. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research17, 83 (2016), 1–5

    2016

  41. [50]

    Jean C Digitale, Jeffrey N Martin, and Medellena Maria Glymour. 2022. Tutorial on directed acyclic graphs.Journal of Clinical Epidemiology142 (2022), 264–267

    2022

  42. [51]

    Joshua V Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif A Saurous. 2017. Tensorflow distributions.arXiv preprint arXiv:1711.10604(2017)

    2017 arXiv

  43. [52]

    Alexander Domahidi, Eric Chu, and Stephen Boyd. 2013. ECOS: An SOCP solver for embedded systems. In2013 European control conference (ECC). IEEE, 3071–3076

    2013

  44. [53]

    Hermann W Dommel and William F Tinney. 2007. Optimal power flow solutions. IEEE Transactions on Power Apparatus and Systems10 (2007), 1866–1876

    2007

  45. [54]

    Simon S Du and Wei Hu. 2019. Linear convergence of the primal-dual gradient method for convex-concave saddle point problems without strong convexity. InThe 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 196–205

    2019

  46. [55]

    Mirjam Dür and Franz Rendl. 2021. Conic optimization: A survey with special focus on copositive optimization and binary quadratic problems. (2021). J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Learning to Optimize by Differentiable Programming 111:33

    2021

  47. [56]

    Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O’Donoghue, Jonathan Ue- sato, and Pushmeet Kohli. 2018. Training verified learners with learned verifiers.arXiv preprint arXiv:1805.10265 (2018)

    2018 arXiv

  48. [57]

    First-ordermethodsforconvexoptimization

    PavelDvurechensky,ShimritShtern,andMathiasStaudigl.2021. First-ordermethodsforconvexoptimization. EURO Journal on Computational Optimization9 (2021), 100015

    2021

  49. [58]

    Zeinab Ebrahimi, Gustavo Batista, and Mohammad Deghat. 2024. AA-DLADMM: An accelerated ADMM-based framework for training deep neural networks.arXiv preprint arXiv:2401.03619(2024)

    2024 arXiv

  50. [59]

    DualitybetweenLagrangianandLegendrianinvariants

    TobiasEkholmandYankıLekili.2023. DualitybetweenLagrangianandLegendrianinvariants. Geometry & Topology 27, 6 (2023), 2049–2179

    2023

  51. [60]

    Ernie Esser, Yifei Lou, and Jack Xin. 2013. A method for finding structured sparse solutions to nonnegative least squares problems with applications.SIAM Journal on Imaging Sciences6, 4 (2013), 2010–2046

    2013

  52. [61]

    Ernie Esser, Xiaoqun Zhang, and Tony F Chan. 2010. A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science.SIAM Journal on Imaging Sciences3, 4 (2010), 1015–1046

    2010

  53. [62]

    Werner Fenchel. 2013. On conjugate convex functions. InTraces and Emergence of Nonlinear Programming. Springer, 125–129

    2013

  54. [63]

    FerdinandoFioretto,PascalVanHentenryck,TerrenceWKMak,CuongTran,FedericoBaldo,andMicheleLombardi

  55. [64]

    Robert Fourer, Chandrakant Maheshwari, Arnold Neumaier, Dominique Orban, and Hermann Schichl. 2010. Con- vexity and concavity detection in computational graphs: Tree walks for convexity assessment.INFORMS Journal on Computing 22, 1 (2010), 26–43

    2010

  56. [65]

    Marguerite Frank, Philip Wolfe, et al. 1956. An algorithm for quadratic programming.Naval Research Logistics Quarterly 3, 1-2 (1956), 95–110

    1956

  57. [66]

    Roy Frostig, Matthew James Johnson, and Chris Leary. 2019. Compiling machine learning programs via high-level tracing. InSysML Conference 2018. Stanford, United States

    2019

  58. [67]

    Bin Gao, Tie-Yan Liu, Wei Wei, Taifeng Wang, and Hang Li. 2011. Semi-supervised ranking on very large graphs with rich metadata. InProceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 96–104

    2011

  59. [68]

    Yuan Gao and Christian Kroer. 2020. First-order methods for large-scale market equilibrium computation.Advances in Neural Information Processing Systems33 (2020), 21738–21750

    2020

  60. [69]

    Susan Garner Garille and Saul I Gass. 2001. Stigler’s diet problem revisited.Operations Research49, 1 (2001), 1–13

    2001

  61. [70]

    Maximilian Gelbrecht, Alistair White, Sebastian Bathiany, and Niklas Boers. 2023. Differentiable programming for Earth system modeling.Geoscientific Model Development16, 11 (2023), 3123–3135

    2023

  62. [71]

    Google. 2024. OR-Tools: Google optimization tools.https://github.com/google/or-tools.git

    2024

  63. [72]

    Google. 2026. Google OR-Tools.https://developers.google.com/optimization

    2026

  64. [73]

    Stephen Gould. 2025. Lecture notes on differentiable optimisation in deep learning

    2025

  65. [74]

    Stephen Gould, Richard Hartley, and Dylan Campbell. 2019. Deep declarative networks: A new hope.arXiv e-prints (2019), arXiv–1909

    2019

  66. [75]

    Deepdeclarativenetworks

    StephenGould,RichardHartley,andDylanCampbell.2021. Deepdeclarativenetworks. IEEE Transactions on Pattern Analysis and Machine Intelligence44, 8 (2021), 3988–4004

    2021

  67. [76]

    Graphimplementationsfornonsmoothconvexprograms

    MichaelGrantandStephenBoyd.2008. Graphimplementationsfornonsmoothconvexprograms. In RecentAdvances in Learning and Control. Springer-Verlag Limited, 95–110

    2008

  68. [77]

    Michael Grant and Stephen Boyd. 2020. CVX: Matlab software for disciplined vonvex programming, version 2.2. https://cvxr.com/cvx/

    2020

  69. [78]

    Michael Grant, Stephen Boyd, and Yinyu Ye. 2008. CVX: Matlab software for disciplined convex programming

    2008

  70. [79]

    Harshit Gupta, Kyong Hwan Jin, Ha Q Nguyen, Michael T McCann, and Michael Unser. 2018. CNN-based projected gradient descent for consistent CT image reconstruction.IEEE Transactions on Medical Imaging37, 6 (2018), 1440– 1453

    2018

  71. [80]

    Mohamad Azrin Syafiq Halim and Yeak Su Hoe. 2024. Introduction to automatic differentiation and neural differen- tiation equation.Proceedings of the Scientific Mathematics Journal24 (2024), 81–89

    2024

  72. [81]

    Nguyen TV Hang, Boris S Mordukhovich, and M Ebrahim Sarabi. 2020. Second-order variational analysis in second- order cone programming.Mathematical Programming180, 1 (2020), 75–116

    2020

  73. [82]

    Moritz Hardt, Ben Recht, and Yoram Singer. 2016. Train faster, generalize better: Stability of stochastic gradient descent. InInternational Conference on Machine Learning. PMLR, 1225–1234

    2016

  74. [83]

    EE227C:Convexoptimizationandapproximation

    MoritzHardtandMaxSimchowitz.2018. EE227C:Convexoptimizationandapproximation. UniversityofCalifornia, Berkeley. https://ee227c.github.io/ J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:34 Tao, Tong and Tan

    2018

  75. [84]

    Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. 2020. Array programming with NumPy.Nature 585, 7825 (2020), 357–362

    2020

  76. [85]

    Ang He, Heng Pan, Yueyue Dai, Xueming Si, Chau Yuen, and Yan Zhang. 2024. ADMM for mobile edge intelligence: A survey.IEEE Communications Surveys & Tutorials(2024)

    2024

  77. [86]

    Robin J Hogan. 2014. Fast reverse-mode automatic differentiation using expression templates in C++.ACM Trans. Math. Software40, 4 (2014), 1–16

    2014

  78. [87]

    Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. 2020. Difftaichi: Differentiable programming for physical simulation.International Conference on Learning Representations (2020)

    2020

  79. [88]

    Yanyan Hu, Yuchen Jin, Xuqing Wu, and Jiefu Chen. 2021. A theory-guided deep neural network for time domain electromagnetic simulation and inversion using a differentiable programming platform.IEEE Transactions on Anten- nas and Propagation70, 1 (2021), 767–772

    2021

  80. [89]

    2023.What is CPLEX? https://www.ibm.com/docs/en/icos/22.1.1?topic=mc-what-is-cplex

    IBM. 2023.What is CPLEX? https://www.ibm.com/docs/en/icos/22.1.1?topic=mc-what-is-cplex

    2023

  81. [90]

    Adiffer- entiable programming system to bridge machine learning and scientific computing.arXiv preprint arXiv:1907.07587 (2019)

    MikeInnes,AlanEdelman,KenoFischer,ChrisRackauckas,ElliotSaba,ViralBShah,andWillTebbutt.2019. Adiffer- entiable programming system to bridge machine learning and scientific computing.arXiv preprint arXiv:1907.07587 (2019)

    2019 arXiv

  82. [91]

    E Joannopoulos, F Dubeau, JP Dussault, and C Pomar. 2015. Diet problems.Handbook of Operations Research in Agriculture and the Agri-Food Industry(2015), 397–417

    2015

  83. [92]

    2025.PyTorch foundations and applications: Definitive reference for developers and engineers

    Richard Johnson. 2025.PyTorch foundations and applications: Definitive reference for developers and engineers. HiTeX Press

    2025

  84. [93]

    Leonid Vitalievich Kantorovich. 1957. On a system of mathematical symbols, convenient for electronic computer operations. Doklady Akademii Nauk SSSR113, 4 (1957), 738–741

    1957

  85. [94]

    2008.Linear programming

    Howard Karloff. 2008.Linear programming. Springer Science & Business Media

    2008

  86. [95]

    Differentiableconvexoptimizationlayersinneuralarchitectures:Foundationsandperspectives

    CalderKatyal.2024. Differentiableconvexoptimizationlayersinneuralarchitectures:Foundationsandperspectives. arXiv preprint arXiv:2412.20679(2024)

    2024 arXiv

  87. [96]

    Nikhil Ketkar and Jojo Moolayil. 2021. Introduction to pytorch. InDeep Learning with Python: Learn Best Practices of Deep Learning Models with PyTorch. Springer, 27–91

    2021

  88. [97]

    Minsoo Kim and Hongseok Kim. 2024. Unsupervised deep lagrange dual with equation embedding for ac optimal power flow.IEEE Transactions on Power Systems40, 1 (2024), 1078–1090

    2024

  89. [98]

    Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization.International Conference on Learning Representations(2015)

    2015

  90. [99]

    Thesingularvaluedecomposition:Itscomputationandsomeapplications

    VirginiaKlemaandAlanLaub.1980. Thesingularvaluedecomposition:Itscomputationandsomeapplications. IEEE Trans. Automat. Control25, 2 (1980), 164–176

    1980

  91. [100]

    Nikos Komodakis and Jean-Christophe Pesquet. 2015. Playing with duality: An overview of recent primal? dual approaches for solving large-scale optimization problems.IEEE Signal Processing Magazine32, 6 (2015), 31–54

    2015

  92. [101]

    James Kotary and Ferdinando Fioretto. 2024. Learning constrained optimization with deep augmented lagrangian methods. arXiv preprint arXiv:2403.03454(2024)

    2024 arXiv

  93. [102]

    James Kotary, Ferdinando Fioretto, and Pascal Van Hentenryck. 2022. Fast approximations for job shop scheduling: A lagrangian dual deep learning method. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 7239–7246

    2022

  94. [103]

    James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, and Bryan Wilder. 2021. End-to-end constrained opti- mization learning: A survey.arXiv preprint arXiv:2103.16378(2021)

    2021 arXiv

  95. [104]

    Recentadvancesinquadraticprogram- ming algorithms for nonlinear model predictive control.Vietnam Journal of Mathematics46, 4 (2018), 863–882

    DimitrisKouzoupis,GianlucaFrison,AndreaZanelli,andMoritzDiehl.2018. Recentadvancesinquadraticprogram- ming algorithms for nonlinear model predictive control.Vietnam Journal of Mathematics46, 4 (2018), 863–882

    2018

  96. [105]

    Dmitry Kovalev, Alexander Gasnikov, and Peter Richtárik. 2022. Accelerated primal-dual gradient method for smooth and convex-concave saddle-point problems with bilinear coupling. Advances in Neural Information Pro- cessing Systems35 (2022), 21725–21737

    2022

  97. [106]

    2023.∇-Prox: Differentiable proximal algorithm modeling for large-scale optimization.ACM Transactions on Graphics42, 4 (2023), 1–19

    Zeqiang Lai, Kaixuan Wei, Ying Fu, Philipp Härtel, and Felix Heide. 2023.∇-Prox: Differentiable proximal algorithm modeling for large-scale optimization.ACM Transactions on Graphics42, 4 (2023), 1–19

    2023

  98. [107]

    2009.Moments, positive polynomials and their applications

    Jean Bernard Lasserre. 2009.Moments, positive polynomials and their applications. Vol. 1. World Scientific

    2009

  99. [108]

    LeanOpt. 2024. LeanOpt: Optimization Learning Platform.https://leanopt.github.io/

    2024

  100. [109]

    Huan Li, Cong Fang, and Zhouchen Lin. 2020. Accelerated first-order optimization algorithms for machine learning. Proc. IEEE108, 11 (2020), 2067–2082

    2020

  101. [110]

    Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable pro- gramming for image processing and deep learning in Halide.ACM Transactions on Graphics37, 4 (2018), 1–13. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August ...

    2018

  102. [111]

    Heng Liang and Changhong Zhao. 2023. DeepOPF-U: A unified deep neural network to solve AC optimal power flow in multiple networks.arXiv preprint arXiv:2309.12849(2023)

    2023 arXiv

  103. [112]

    Hai-Jun Liao, Jin-Guo Liu, Lei Wang, and Tao Xiang. 2019. Differentiable programming tensor networks.Physical Review X9, 3 (2019), 031041

    2019

  104. [113]

    Erik Lindholm, John Nickolls, Stuart Oberman, and John Montrym. 2008. NVIDIA Tesla: A unified graphics and computing architecture.IEEE Micro28, 2 (2008), 39–55

    2008

  105. [114]

    OnthelimitedmemoryBFGSmethodforlargescaleoptimization

    DongCLiuandJorgeNocedal.1989. OnthelimitedmemoryBFGSmethodforlargescaleoptimization. Mathematical programming 45, 1 (1989), 503–528

    1989

  106. [115]

    Jun Liu and Jieping Ye. 2009. Efficient Euclidean projections in linear time. InProceedings of the 26th Annual Inter- national Conference on Machine Learning. 657–664

    2009

  107. [116]

    Applicationsofsecond-ordercone programming

    MiguelSousaLobo,LievenVandenberghe,StephenBoyd,andHervéLebret.1998. Applicationsofsecond-ordercone programming. Linear Algebra and Its Applications284, 1-3 (1998), 193–228

    1998

  108. [117]

    Haihao Lu, Zedong Peng, and Jinwen Yang. 2024. Mpax: Mathematical programming in jax. arXiv preprint arXiv:2412.09734 (2024)

    2024

  109. [118]

    Zhi-Quan Luo, Wing-Kin Ma, Anthony Man-Cho So, Yinyu Ye, and Shuzhong Zhang. 2010. Semidefinite relaxation of quadratic optimization problems.IEEE Signal Processing Magazine27, 3 (2010), 20–34

    2010

  110. [119]

    Jayanta Mandi, James Kotary, Senne Berden, Maxime Mulamba, Victor Bucarey, Tias Guns, and Ferdinando Fioretto

  111. [120]

    Charles C Margossian. 2019. A review of automatic differentiation and its efficient implementation.Wiley Interdis- ciplinary Reviews: Data Mining and Knowledge Discovery9, 4 (2019), e1305

    2019

  112. [121]

    Jared Miller, Yang Zheng, Mario Sznaier, and Antonis Papachristodoulou. 2022. Decomposed structured subsets for semidefinite and sum-of-squares optimization.Automatica 137 (2022), 110125

    2022

  113. [122]

    William S Moses, Valentin Churavy, Ludger Paehler, Jan Hückelheim, Sri Hari Krishna Narayanan, Michel Schanen, and Johannes Doerfert. 2021. Reverse-mode automatic differentiation and optimization of GPU kernels via Enzyme. In Proceedings of the International Conference for Hig...

    2021

  114. [123]

    Yurii Nesterov. 2009. Primal-dual subgradient methods for convex problems.Mathematical Programming 120, 1 (2009), 221–259

    2009

  115. [124]

    Yu Nesterov. 2011. Towards non-symmetric conic optimization.Optimization Methods and Software27, 4–5 (2011), 893–917

    2011

  116. [125]

    Yu Nesterov. 2013. Gradient methods for minimizing composite functions.Mathematical Programming140, 1 (2013), 125–161

    2013

  117. [126]

    Vishnoi NK. 2012. Lx= b Laplacian solvers and their algorithmic applications.Foundations and Trends® Theoretical Computer Science8, 1-2 (2012), 1–141

    2012

  118. [127]

    Deniz Oktay, Nick McGreivy, Joshua Aduol, Alex Beatson, and Ryan P Adams. 2020. Randomized automatic differ- entiation. arXiv preprint arXiv:2007.10412(2020)

    2020 arXiv

  119. [128]

    O’Donoghue, E

    B. O’Donoghue, E. Chu, N. Parikh, and S. Boyd. 2016. Conic optimization via operator splitting and homogeneous self-dual embedding.Journal of Optimization Theory and Applications169 (2016), 1042–1068

    2016

  120. [129]

    Aleksandra Pachalieva, Daniel O’Malley, Dylan Robert Harp, and Hari Viswanathan. 2022. Physics-informed ma- chine learning with differentiable programming for heterogeneous underground reservoir pressure management. Scientific Reports12, 1 (2022), 18734

    2022

  121. [130]

    Jianming Pan, Zeqi Ye, Xiao Yang, Xu Yang, Weiqing Liu, Lewen Wang, and Jiang Bian. 2024. BPQP: A differentiable convexoptimizationframeworkforefficientend-to-endlearning. Advances in Neural Information Processing Systems 37 (2024), 77468–77493

    2024

  122. [131]

    Bo Pang, Erik Nijkamp, and Ying Nian Wu. 2020. Deep learning with tensorflow: A review.Journal of Educational and Behavioral Statistics45, 2 (2020), 227–248

    2020

  123. [132]

    McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami

    Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 2016. Distillation as a defense to adversarial perturbations against deep neural networks. InIEEE Symposium on Security and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016. IEEE Computer Societ...

    2016

  124. [133]

    Neal Parikh, Stephen Boyd, et al. 2014. Proximal algorithms.Foundations and Trends® in Optimization1, 3 (2014), 127–239

    2014

  125. [134]

    Seonho Park and Pascal Van Hentenryck. 2023. Self-supervised primal-dual learning for constrained optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 4052–4060

    2023

  126. [135]

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017). J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:36 Tao, ...

    2017

  127. [136]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing System...

    2019

  128. [137]

    Fabian Pedregosa. 2016. Hyperparameter optimization with approximate gradient. InInternational Conference on Machine Learning. PMLR, 737–746

    2016

  129. [138]

    Marin Vlastelica Pogančić, Anselm Paulus, Vit Musil, Georg Martius, and Michal Rolinek. 2019. Differentiation of blackbox combinatorial solvers. InInternational Conference on Learning Representations

    2019

  130. [139]

    Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. 2018. Semidefinite relaxations for certifying robustness to adversarial examples.Advances in Neural Information Processing Systems31 (2018)

    2018

  131. [140]

    Ben Recht. 2024. The art of deconstraining: A guide to convex optimization in PyTorch.https://www.argmin.net/p/ the-art-of-deconstraining

    2024

  132. [141]

    Ben Recht. 2024. Cone programming. Available athttps://www.argmin.net/p/cone-programming

    2024

  133. [142]

    Ralph Tyrell Rockafellar. 2015. Convex analysis. (2015)

    2015

  134. [143]

    Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms.arXiv preprint arXiv:1609.04747 (2016)

    2016 arXiv

  135. [144]

    Facundo Sapienza, Jordi Bolibar, Frank Schäfer, Brian Groenke, Avik Pal, Victor Boussange, Patrick Heimbach, Giles Hooker, Fernando Pérez, Per-Olof Persson, et al. 2024. Differentiable programming for differential equations: A review. arXiv preprint arXiv:2406.09699(2024)

    2024

  136. [145]

    2024.Deep learning with JAX

    Grigory Sapunov. 2024.Deep learning with JAX. Simon and Schuster

    2024

  137. [146]

    Maximilian Schaller and Stephen Boyd. 2025. Code generation for solving and differentiating through convex opti- mization problems.arXiv preprint arXiv:2504.14099(2025)

    2025 arXiv

  138. [147]

    Joshua Schneider, David Basin, Frederik Brix, Srđan Krstić, and Dmitriy Traytel. 2021. Scalable online first-order monitoring. International Journal on Software Tools for Technology Transfer23, 2 (2021), 185–208

    2021

  139. [148]

    Samuel Schoenholz and Ekin Dogus Cubuk. 2020. Jax md: A framework for differentiable physics.Advances in Neural Information Processing Systems33 (2020), 11428–11441

    2020

  140. [149]

    John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. 2015. Gradient estimation using stochastic computation graphs.Advances in Neural Information Processing Systems28 (2015)

    2015

  141. [150]

    C H SCOTT and TR Jefferson. 1989. Conjugate duals for power functions.International Journal of Systems Science 20, 11 (1989), 2153–2160

    1989

  142. [151]

    2018.Machine learning with TensorFlow

    Nishant Shukla and Kenneth Fricklas. 2018.Machine learning with TensorFlow. Vol. 7. Manning Greenwich

    2018

  143. [152]

    Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alberto Bemporad, and Stephen Boyd. 2020. OSQP: An operator splitting solver for quadratic programs.Mathematical Programming Computation12, 4 (2020), 637–672

    2020

  144. [153]

    2022.Introduction to linear algebra

    Gilbert Strang. 2022.Introduction to linear algebra. SIAM

    2022

  145. [154]

    2012.Linear programming and its applications

    James K Strayer. 2012.Linear programming and its applications. Springer Science & Business Media

    2012

  146. [155]

    Chee Wei Tan, Desmond WH Cai, and Xin Lou. 2014. Resistive network optimal power flow: Uniqueness and algorithms. IEEE Transactions on Power Systems30, 1 (2014), 263–273

    2014

  147. [156]

    Chee Wei Tan and Siya Chen. 2026. DIFFRACT: Neuralized utility maximization for wireless networks by differen- tiable programming. InIEEE INFOCOM 2026-IEEE Conference on Computer Communications. IEEE, 1–10

    2026

  148. [157]

    Bo Tang and Elias B Khalil. 2024. PyEPO: A PyTorch-based end-to-end predict-then-optimize library for linear and integer programming.Mathematical Programming Computation16, 3 (2024), 297–335

    2024

  149. [158]

    Ke Tang and Xin Yao. 2024. Learn to optimize-A brief overview.National Science Review(2024), nwae132

    2024

  150. [159]

    Mathieu Tanneau and Pascal Van Hentenryck. 2024. Dual lagrangian learning for conic optimization.Advances in Neural Information Processing Systems37 (2024), 55538–55561

    2024

  151. [160]

    Teel, Jorge I

    Andrew R. Teel, Jorge I. Poveda, and Justin H. Le. 2019. First-order optimization algorithms with resets and hamil- tonian flows. In58th IEEE Conference on Decision and Control, CDC 2019, Nice, France, December 11-13, 2019. IEEE, 5838–5843

    2019

  152. [161]

    Cuong Tran, Ferdinando Fioretto, and Pascal Van Hentenryck. 2021. Differentially private and fair deep learning: A lagrangian dual approach. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 9932–9939

    2021

  153. [162]

    Jonathan Tuck, David Hallac, and Stephen Boyd. 2019. Distributed majorization-minimization for Laplacian regu- larized problems.IEEE/CAA Journal of Automatica Sinica6, 1 (2019), 45–52

    2019

  154. [163]

    TheNumPyarray:Astructureforefficientnumerical computation

    StefanVanDerWalt,SChrisColbert,andGaelVaroquaux.2011. TheNumPyarray:Astructureforefficientnumerical computation. Computing in Science & Engineering13, 2 (2011), 22–30

    2011

  155. [164]

    Pascal Van Hentenryck and Kevin Dalmeijer. 2024. AI4OPT: AI institute for advances in optimization.AI Magazine 45, 1 (2024), 42–47

    2024

  156. [165]

    Lieven Vandenberghe and Stephen Boyd. 1996. Semidefinite programming.SIAM Rev.38, 1 (1996), 49–95

    1996

  157. [166]

    Tim Vieira. 2017. Backprop is not just the chain rule. Blog post, Graduate Descent.https://timvieira.github.io/blog/ post/2017/08/18/backprop-is-not-just-the-chain-rule/ J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Learning to Optimize by Differentiable ...

    2017

  158. [167]

    Chaoming Wang, Sichao He, Shouwei Luo, Yuxiang Huan, and Si Wu. 2025. Integrating physical units into high- performance AI-driven scientific computing.Nature Communications16, 1 (2025), 3609

    2025

  159. [168]

    Fei Wang, James Decker, Xilun Wu, Gregory Essertel, and Tiark Rompf. 2018. Backpropagation with callbacks: Foundations for efficient and expressive differentiable programming. Advances in Neural Information Processing Systems 31 (2018)

    2018

  160. [169]

    Demystifyingdifferen- tiableprogramming:Shift/resetthepenultimatebackpropagator

    FeiWang,DanielZheng,JamesDecker,XilunWu,GrégoryMEssertel,andTiarkRompf.2019. Demystifyingdifferen- tiableprogramming:Shift/resetthepenultimatebackpropagator. Proceedings of the ACM on Programming Languages 3, ICFP (2019), 1–31

    2019

  161. [170]

    Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. 2021. Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verification.Advances in Neural Information Processing Systems34 (2021), 29909–29921

    2021

  162. [171]

    GlobalconvergenceofADMMinnonconvexnonsmoothoptimization

    YuWang,WotaoYin,andJinshanZeng.2019. GlobalconvergenceofADMMinnonconvexnonsmoothoptimization. Journal of Scientific Computing78, 1 (2019), 29–63

    2019

  163. [172]

    Eric Wong and Zico Kolter. 2018. Provable defenses against adversarial examples via the convex outer adversarial polytope. InInternational Conference on Machine Learning. PMLR, 5286–5295

    2018

  164. [173]

    Quantifyingpoliticalleaningfromtweets, retweets, and retweeters.IEEE Transactions on Knowledge and Data Engineering28, 8 (2016), 2158–2172

    FelixMingFaiWong,CheeWeiTan,SoumyaSen,andMungChiang.2016. Quantifyingpoliticalleaningfromtweets, retweets, and retweeters.IEEE Transactions on Knowledge and Data Engineering28, 8 (2016), 2158–2172

    2016

  165. [174]

    1997.Primal-dual interior-point methods

    Stephen J Wright. 1997.Primal-dual interior-point methods. SIAM

    1997

  166. [175]

    2022.Optimization for data analysis

    Stephen J Wright and Benjamin Recht. 2022.Optimization for data analysis. Cambridge University Press

    2022

  167. [176]

    DifferentiablelinearizedADMM

    XingyuXie,JianlongWu,GuangcanLiu,ZhishengZhong,andZhouchenLin.2019. DifferentiablelinearizedADMM. In International Conference on Machine Learning. PMLR, 6902–6911

    2019

  168. [177]

    Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. 2020. Fast and complete: Enabling complete neural networkverification with rapid and massivelyparallel incomplete verifiers.arXiv preprint arXiv:2011.13824 (2020)

    2020 arXiv

  169. [178]

    Zhisheng Ye, Wei Gao, Qinghao Hu, Peng Sun, Xiaolin Wang, Yingwei Luo, Tianwei Zhang, and Yonggang Wen

  170. [179]

    Jihun Yun, Aurélie C Lozano, and Eunho Yang. 2021. Adaptive proximal gradient methods for structured neural networks. Advances in Neural Information Processing Systems34 (2021), 24365–24378

    2021

  171. [180]

    Alp Yurtsever, Joel A Tropp, Olivier Fercoq, Madeleine Udell, and Volkan Cevher. 2021. Scalable semidefinite pro- gramming. SIAM Journal on Mathematics of Data Science3, 1 (2021), 171–200

    2021

  172. [181]

    Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh

  173. [182]

    Yang Zheng, Giovanni Fantuzzi, and Antonis Papachristodoulou. 2021. Chordal and factor-width decompositions for scalable semidefinite and polynomial optimization.Annual Reviews in Control52 (2021), 243–279

    2021

  174. [183]

    Surveys56, 6 (2024), 1–38

    Deep learning workload scheduling in GPU datacenters: A survey.Comput. Surveys56, 6 (2024), 1–38

    2024

  175. [189]

    Mingqiang Zhu and Tony Chan. 2008. An efficient primal-dual hybrid gradient algorithm for total variation image restoration. Ucla Cam Report34, 2 (2008). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 J. ACM, Vol. 37, No. 4, Article 111. Publication dat...

    2008

  176. [2019]

    Towards stable and efficient training of verifiably robust neural networks.arXiv preprint arXiv:1906.06316 (2019)

    1906 arXiv

  177. [2020]

    In Joint European Conference on Machine Learning and Knowledge Discovery in Databases

    Lagrangian duality for constrained deep learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 118–135

  178. [2021]

    Practical large-scale linear programming using primal-dual hybrid gradient.Advances in Neural Information Processing Systems34 (2021), 20243–20257

    2021

  179. [2024]

    Journal of Artificial Intelligence Research80 (2024), 1623–1701

    Decision-focused learning: Foundations, state of the art, benchmark and future opportunities. Journal of Artificial Intelligence Research80 (2024), 1623–1701

    2024

  180. [2025]

    arXivpreprintarXiv:2501.07018 (2025)

    Pdlp:Apracticalfirst-ordermethodforlarge-scalelinearprogramming. arXivpreprintarXiv:2501.07018 (2025)

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.