Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

Implicit Deep Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep networks reduce to a single fixed-point equation

desk verdict Solid well-posedness and robustness theory for implicit models, but Theorem 6.1 overclaims gradient existence and needs a spectral-radius fix. read the letter →

arxiv 1908.06315 v4 pith:ZYU7DYZY submitted 2019-08-17 cs.LG math.OCstat.ML

Laurent El Ghaoui , Fangda Gu , Bertrand Travacca , Armin Askari , Alicia Y. Tsai This is my paper · ORCID
classification cs.LGmath.OCstat.ML MSC 90C2649M9965K1062M4526B10
keywords implicitdeeplearningfixed-pointequationequilibriummodelsPerron-Frobeniuseigenvaluewell-posednessrobustnesscertificationadversarialattacksdifferentiation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the prediction rule of a deep network need not be defined layer by layer. Instead, a single state vector $x$ can be defined implicitly as the solution of the equilibrium equation $x = \varphi(Ax + Bu)$, with output $y = Cx + Du$. The paper shows that feedforward, residual, recurrent, convolutional, and attention-based architectures all admit such a representation, and that the same framework supports well-posedness conditions, robustness bounds, adversarial attack generation, and gradient-based training. A sympathetic reader would care because the implicit view condenses architectural variety into one object, so that analysis tools apply uniformly across architectures.

What carries the argument

The central object is the equilibrium equation $x = \varphi(Ax + Bu)$ together with its Perron-Frobenius (PF) contractivity condition $\lambda_{\mathrm{pf}}(|A|) < 1$, where $|A|$ is the entrywise absolute value and $\lambda_{\mathrm{pf}}$ its largest non-negative eigenvalue. Under a componentwise non-expansive (CONE) activation such as ReLU, tanh, or sigmoid, the condition guarantees a unique solution for every input $b$ and makes the fixed-point iteration $x^{(t+1)} = \varphi(Ax^{(t)} + b)$ converge linearly. The same PF machinery, via the Collatz-Wielandt formula, permits rescaling a model so that the convex constraint $\|A\|_\infty < 1$ replaces the non-convex PF condition, and it generates the sensitivity matrix $S = |C|(I - |A|)^{-1}|B| + |D|$ that bounds output changes and drives attacks and robustness penalties.

What would settle it

Train an implicit model with tanh activation under the constraint $\|A\|_\infty < 1$ on a regression task that a standard feedforward network fits well; if the constrained implicit model cannot approach the same accuracy, the lossless replacement claim for non-homogeneous activations fails. Alternatively, compute an adversarial perturbation within a given box for a trained implicit model and check whether the output change ever exceeds the sensitivity-matrix bound $S\sigma_u$; a violation would break the robustness theorem's assumptions.

Watch

Extended reading notes

Core claim

The paper's central claim is that implicit prediction rules of the form $y = Cx + Du$ with $x = \varphi(Ax + Bu)$ generalize the recursive rules of standard deep learning, and that a numerically tractable sufficient condition—$\lambda_{\mathrm{pf}}(|A|) < 1$ for componentwise non-expansive activations, or $\lambda_{\mathrm{pf}}(\Gamma N(A)) < 1$ for blockwise Lipschitz ones—guarantees the equilibrium equation has a unique solution computable by fixed-point iteration. On this foundation the paper derives componentwise sensitivity bounds for states and outputs, convex penalties that encourage robustness, LP and SDP relaxations that produce adversarial attacks, and an implicit-differentiation rule for training. If correct, the framework makes deep learning notation simpler and turns robustness analysis into a matrix inequality problem.

Load-bearing premise

The framework's practical claims rest on the assumption that enforcing the contractivity condition during training does not consume the expressive power needed for the task; for non-homogeneous activations such as tanh or sigmoid, the paper replaces the Perron-Frobenius condition with a max-row-sum norm constraint without proving that this replacement is lossless.

Editorial extensions

If this is right

  • Every standard deep architecture can be written as an implicit model, so robustness certificates derived for the implicit form apply to the original network.
  • The fixed-point iteration for solving the equilibrium equation costs about the same as one forward pass, so implicit models are practical to evaluate.
  • Training can be done by differentiating through the equilibrium, which itself requires solving a fixed-point equation in the gradient variables.
  • Sparsity and low-rank structure in the model matrix translate directly into computational savings and state-dimension reduction.
  • Well-posed implicit models are Lipschitz-continuous with an explicit constant, giving an a priori bound on output changes under bounded input perturbations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same fixed-point view may extend to time-varying and stochastic activations, since the contraction argument only relies on the Lipschitz structure of the map.
  • If the max-row-sum constraint is used as a surrogate for the PF condition when training non-homogeneous activations, a gap opens between the training constraint and the well-posedness guarantee; measuring that gap on tanh and sigmoid networks is a testable next step.
  • The sensitivity matrix suggests a data-free way to compare the robustness of different architectures on the same task, since it depends only on parameters, not on inputs.
  • One could combine the implicit representation with control-theoretic analyses, such as integral quadratic constraints, to certify stability of closed-loop systems containing deep networks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces a class of prediction rules based on the implicit equilibrium equation x = φ(Ax + Bu) with output y = Cx + Du, and argues that this framework generalizes feedforward, residual, recurrent, and convolutional networks. It provides sufficient conditions for well-posedness using Perron–Frobenius spectral radius bounds under componentwise non-expansive (CONE) and blockwise Lipschitz (BLIP) activations, develops composition rules, derives robustness bounds and sensitivity matrices, proposes LP and SDP relaxations for adversarial attacks, discusses sparsity and model compression, and formulates training algorithms based on implicit differentiation and lifted/Fenchel-divergence relaxations. Numerical experiments on synthetic data, MNIST, GTSRB, and adversarial attack benchmarks support the practical viability of the approach.

Significance. If the training-theory claims are corrected, the paper makes a substantial contribution: it unifies several network architectures under a single implicit-model formalism, gives tractable sufficient conditions for well-posedness, and derives robustness certificates and attack constructions that apply beyond feedforward networks. The proofs of Theorems 2.2, 2.7, and 4.1–4.4 are standard contraction arguments and appear valid, and the robustness bounds are derived from stated assumptions rather than fit to data. The experimental comparisons with feedforward networks on MNIST and GTSRB are informative, and the sensitivity-matrix attack is a simple and interesting contribution. The main weakness is in the gradient-existence theorem, which is overbroad as stated and needs a stronger well-posedness hypothesis.

major comments (3)
  1. [§6.2, Theorem 6.1; Appendix G, Lemma G.1] Theorem 6.1 is false as stated. Well-posedness of the equilibrium equation does not imply nonsingularity of I − ΦA or existence of the gradient fixed-point equation. Counterexample: n = 1, φ = tanh, A = 1. For every b, x = tanh(x + b) has a unique solution because x − tanh(x + b) is strictly increasing with limits ±∞, so A ∈ WP(φ). At b = 0 the solution is x = 0 and Φ = φ'(0) = 1, so I − ΦA = 0; the gradient equation v = Φ(Av + c) becomes v = v + c, which has no solution for c ≠ 0, and the solution map x(b) behaves like (3b)^{1/3}, so ∂x/∂b is unbounded and the loss gradient does not exist. The proof in Lemma G.1 asserts that a BLIP map has derivative blocks with norm < 1; BLIP only gives Lipschitz constants γ_l ≥ 0, which may equal 1, and the derivative can attain norm 1 (e.g., tanh at 0). The theorem should be restated under a stronger condition such as λ_pf(|A|) < 1 or sup_z ‖Φ(z)A‖ < 1, which is effectively what the implemented algorithms use.
  2. [§2.5 and §6.1] The claim in §6.1 that the PF well-posedness condition can be replaced 'without loss of generality' by the convex constraint ‖A‖∞ ≤ κ for CONE maps is not supported by §2.5. The scaling argument in §2.5 explicitly assumes φ is positively homogeneous (e.g., ReLU), because the diagonal similarity (2.7) preserves the prediction rule only when φ(Sy) = Sφ(y). For non-homogeneous CONE activations such as tanh or sigmoid, the scaled model does not compute the same prediction rule, and the stated WLOG reduction is not established. This affects the training formulation (6.4), where the norm constraint is treated as a WLOG replacement rather than as a conservative sufficient condition.
  3. [§6.3] The statement that 'given X, the problem is convex in the model matrix M' for training problem (6.4) is incorrect: the equality constraint X = φ(AX + BU) is generally nonconvex in (A, B) even for fixed X, for activations such as tanh or ReLU. The subsequent block-coordinate descent discussion appears to apply to the Fenchel divergence relaxation rather than to the exact constrained problem; the text should state this clearly and avoid asserting convexity for the exact constraint.
minor comments (3)
  1. [§4.6] The assertion that 'the bidual of a non-convex QCQP simply becomes the canonical rank relaxed version' is not a general theorem and is not established by the cited Appendix B of [10]; since (4.19) is a valid relaxation by construction, the upper-bound conclusion survives, but the 'bidual' terminology should be corrected or justified.
  2. [§3.6] The sentence 'As shown in Section 2.2, we can formulate these multiplicative units as well-posed implicit models' should refer to the multiplicative-connection discussion in §2.4 rather than §2.2.
  3. [§6.2] There is a typo in 'Due to the fact taht A satisfies'; it should read 'that'.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation found: the core well-posedness, robustness, and representation results follow from stated contraction and Lipschitz assumptions, not from fitted values or load-bearing self-citations.

full rationale

The main derivation chain is self-contained. Theorem 2.2 is a direct application of the contraction mapping theorem; Appendix A proves existence and uniqueness from lambda_pf(|A|)<1, and the proof does not presuppose the theorem's conclusion. The BLIP extension in Theorem 2.7 has an analogous proof in Appendix B using the norm matrix M = diag(gamma)N(A). The robustness bounds (Theorems 4.1-4.4) are derived from the componentwise non-expansiveness or BLIP condition and from the invertibility of I-|A| or I-Gamma N(A); no fitted parameter is later renamed as a prediction. The representation of feedforward, residual, recurrent, and convolutional networks in Section 3 is by explicit construction, with well-posedness inherited from block-triangular structure. Section 2.5's scaling argument is supported by the Collatz-Wielandt formula rather than by the paper's own results, although its use in Section 6.1 for general CONE maps silently drops the positive-homogeneity requirement; this is a correctness gap, not circularity. Similarly, the skeptic's counterexample to Theorem 6.1 concerns the false assertion in Lemma G.1 that a BLIP derivative map has block norms strictly below 1; the failure of gradient existence for phi=tanh and A=1 is a proof/correctness issue, not a circular reduction, since well-posedness is not being used to prove well-posedness. The self-citations to Fenchel lifted networks [24], lifted neural networks [5], and implicit optimization [51] are background tools for the training relaxation; the ReLU Fenchel divergence is stated directly in Eq. (6.3), and the central well-posedness and robustness results do not rest on these citations. No step was found in which a prediction is equivalent by construction to an input, a fitted parameter is relabeled as a prediction, or a uniqueness theorem from the authors' prior work is used to force the chosen framework.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The core theory relies on standard contraction arguments, Perron-Frobenius theory, and domain assumptions about activation maps. No new physical entities or fitted constants enter the derivations. The experimental hyperparameters (n, kappa, sigma_u) are free choices but do not support the main theorems.

free parameters (3)
  • state dimension n (experiments) = 75, 10, 100, 400
    Chosen per experiment, roughly matching hidden units of comparison networks; not fitted to data, but a free design choice.
  • well-posedness bound kappa = 0.5 in Section 7.1, 1 for synthetic implicit generation
    Hand-chosen hyperparameter constraining ||A||_inf; controls the contraction margin.
  • input uncertainty sigma_u = not specified in experiments
    Assumed given in the robustness analysis; no concrete values are selected or reported.
assumptions (5)
  • standard math Banach fixed point theorem / contraction mapping theorem
    Used in Theorem 2.2 and Appendix A to establish existence and uniqueness of x = phi(Ax+b) under lambda_pf(|A|) < 1.
  • standard math Perron-Frobenius theorem and Collatz-Wielandt formula
    Used to relate lambda_pf(|A|) to optimally scaled infinity norms in Section 2.5.
  • domain assumption Activation maps are CONE or BLIP with known Lipschitz constants
    All well-posedness and robustness theorems restrict to these classes; see Section 2.1.
  • domain assumption Gradient of loss is computable and the implicit function theorem applies at the fixed point
    Theorem 6.1 assumes differentiable BLIP activations and well-posed A; see Appendix G.
  • ad hoc to paper The bidual of a non-convex QCQP equals the standard rank-relaxed SDP (as in Appendix B of Boyd and Vandenberghe [10])
    Section 4.6 relies on this known result to claim that the SDP relaxation (4.19) is the canonical rank relaxation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Implicit Deep Learning." pith.science (2026). https://pith.science/paper/ZYU7DYZY

@misc{pith2026190806315,
  author       = {Pith},
  title        = {Pith review of: Implicit Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZYU7DYZY}},
  note         = {Machine review of arXiv:1908.06315}
}
read the original abstract

Implicit deep learning prediction rules generalize the recursive rules of feedforward neural networks. Such rules are based on the solution of a fixed-point equation involving a single vector of hidden features, which is thus only implicitly defined. The implicit framework greatly simplifies the notation of deep learning, and opens up many new possibilities, in terms of novel architectures and algorithms, robustness analysis and design, interpretability, sparsity, and network architecture optimization.

Figures

Figures reproduced from arXiv: 1908.06315 by the authors.

Figure 1
Figure 1. A block-diagram view of an implicit model. 1. Introduction. 1.1. Implicit prediction rules. In this paper, we consider a new class of deep learning models that are based on implicit prediction rules. Such rules are not obtained via a recursive procedure through several layers, as in current neural networks. Instead, they are based on solving a fixed-point equation in some single “state” vector x ∈ R n . Precisely, f… view at source ↗
Figure 2
Figure 2. Left: equation x = (ax + b)+ has two or no solutions, depending on the sign of b. Right: for a < 0, solution is unique for every b. There are many classes of matrices that satisfy the well-posedness property. As seen next, strictly upper-triangular matrices are well-posed with respect to any activation map that acts componentwise; such a class arises when modeling feedforward neural networks as implicit models, as s… view at source ↗
Figure 3
Figure 3. Cascade connection of two implicit models. Using the above results, we can preserve well-posedness of implicit models via composi￾tion. For example, given two models with matrix parameters (Ai , Bi , Ci , Di) and activation functions φi , i = 1, 2, we can consider a “cascaded” prediction rule: yˆ2 = C2x2 + D2u2 where u2 = ˆy1 = C1x1 + D1u1, where xi = φi(Aixi + Biui), i = 1, 2 [PITH_FULL_IMAGE:figures/full_fig_p006… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Feedback connection of two implicit models. Finally, feedback connections are also possible. Consider two well-posed implicit systems: yi = Cixi + Diui , xi = φi(Aixi + Bui), i = 1, 2 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: A max-pooling operation: the smaller image contains the maximal pixel values of each colored area. down-sample of an image, which is a smaller image that contains the largest pixel values of [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Sensitivity matrix for a 3-layer neural network with q = 10 outputs and n = 1094 states. The sensitivity matrix is a way to visualize the input-output properties of a given model [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Implicit prediction y(u) comparison with f(u) we achieve a RMSE of 0.56. For comparison purposes, we also train a neural network with 3 hidden layers of width n/3 = 25 using ADAM, mini-batches, and a tuned learning rate. We run Adam until convergence. We get a RMSE = 0…
Figure 10
Figure 10. Figure 10: RMSE across projected gradient iterations for the (A, B) block update 7.2. Comparison with neural networks. In that section we compare the performance of implicit models with that of neural networks. Experiments on both synthetic datasets and real datasets are conduct…
Figure 11
Figure 11. Figure 11: Performance comparison on a syn￾thetic dataset generated from a neural network. Av￾erage best accuracy, implicit: 0.85, neural network: 0.76. The curves are generated from 5 the different runs with the lines marked as mean and region marked as the standard deviation 0…
Figure 13
Figure 13. Figure 13: Performance comparison on MNIST. Average best accuracy, implicit: 0.976, neural net￾works: 0.972. The curves are generated from 5 dif￾ferent runs with the lines marked as mean and region marked as the standard deviation over the runs. 0 2 4 6 8 10 epochs 0.5 0.6 0.7 0…
Figure 15
Figure 15. Figure 15: Left: sensitivity values of a feed-forward network for the class “digit 0” in MNIST. Right: sensitivity values of a ResNet-20 model for the class “airplane” in CIFAR-10. Brighter colors correspond to higher sensitivity when perturbed [PITH_FULL_IMAGE:figures/full_fig…
Figure 16
Figure 16. Figure 16: Top: adversarial samples from MNIST. On the left are dense attacks with small perturbations and on the right are sparse attacks with random perturbations (perturbed pixels are marked as red). Bottom: example sparse attack on CIFAR-10. The left ones are cleaned images,…
Figure 17
Figure 17. Figure 17: Example attack on CIFAR dataset. Top: clean data. Bottom: perturbed data [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]
Figure 18
Figure 18. Figure 18: shows three examples of perturbed images under non-sparse and sparse attack. Images on the left are the results of non-sparse attack and those on the right are the results of sparse attack. The model fails to predict the label correctly under both conditions. These re…
Figure 19
Figure 19. Figure 19: Some cousins of implicit models: LTI systems (bottom left) and uncertain systems (bottom right). 9.1. Cousins of implicit models. Implicit models rely on a representation of the prediction rule where the linear operations are clearly separated from the (parameter-free…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Deep Equilibrium Models

    cs.LG 2019-09 accept novelty 5.0 of 10

    Deep equilibrium models find the fixed point of a weight-tied sequence network via root-finding and implicit differentiation, achieving constant-memory training and competitive language modeling results.

Reference graph

Works this paper leans on

60 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    B. Amos, I. Jimenez, J. Sacks, B. Boots, and J. Z. Kolter , Differentiable mpc for end-to-end planning and control, in Advances in Neural Information Processing Systems, 2018, pp. 8289–8300

    work page 2018

  2. [2]

    Amos and J

    B. Amos and J. Z. Kolter , Optnet: Differentiable optimization as a layer in neural networks , in Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, 2017, pp. 136–145

    work page 2017

  3. [3]

    Anwar, K

    S. Anwar, K. Hwang, and W. Sung , Structured pruning of deep convolutional neural networks , ACM Journal on Emerging Technologies in Computing Systems (JETC), 13 (2017), pp. 1–18

    work page 2017

  4. [4]

    J. D. Aplevich , The essentials of linear state-space systems , Wiley New York, 2000

    work page 2000

  5. [5]

    Askari, G

    A. Askari, G. Negiar, R. Sambharya, and L. E. Ghaoui , Lifted neural networks , arXiv preprint arXiv:1805.01532, (2018)

    arXiv 2018

  6. [6]

    Athalye, N

    A. Athalye, N. Carlini, and D. A. W agner , Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples , in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm¨ assan, Stockholm, Sweden, July 10-15, 2018, vol. 80 of Proceedings of Machine Learning Research, PMLR, 2018, pp. 274–283

    work page 2018

  7. [7]

    Bahdanau, K

    D. Bahdanau, K. Cho, and Y. Bengio , Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473, (2014)

    arXiv 2014

  8. [8]

    S. Bai, J. Z. Kolter, and V. Koltun , Deep equilibrium models. Preprint submitted, 2019

    work page 2019

Show all 60 references
  1. [9]

    Belkin, D

    M. Belkin, D. Hsu, S. Ma, and S. Mandal , Reconciling modern machine-learning practice and the clas- sical bias–variance trade-off, Proceedings of the National Academy of Sciences, 116 (2019), pp. 15849– 15854

    2019

  2. [10]

    Boyd and L

    S. Boyd and L. V andenberghe , Convex optimization, Cambridge university press, 2004

    2004

  3. [11]

    Carlini and D

    N. Carlini and D. A. W agner , Adversarial examples are not easily detected: Bypassing ten detec- tion methods , in Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, IMPLICIT DEEP LEARNING 35 AISec@CCS 2017, Dallas, TX, USA, November 3, 2017, ACM, 20...

    2017

  4. [12]

    Carreira-Perpinan and W

    M. Carreira-Perpinan and W. W ang , Distributed optimization of deeply nested systems , in Proceed- ings of the Seventeenth International Conference on Artificial Intelligence and Statistics, S. Kaski and J. Corander, eds., vol. 33 of Proceedings of Machine Learning Research, R...

    2014

  5. [13]

    Changpinyo, M

    S. Changpinyo, M. Sandler, and A. Zhmoginov , The power of sparsity in convolutional neural networks, arXiv preprint arXiv:1702.06257, (2017)

    2017 arXiv

  6. [14]

    T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud , Neural ordinary differential equa- tions, in Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds., Curran Associates, Inc., 201...

    2018

  7. [15]

    K. Cho, B. V an Merri ¨enboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio , Learning phrase representations using rnn encoder-decoder for statistical machine trans- lation, arXiv preprint arXiv:1406.1078, (2014)

    2014 arXiv

  8. [16]

    J. M. Cohen, E. Rosenfeld, and J. Z. Kolter , Certified adversarial robustness via randomized smooth- ing, 2019, https://arxiv.org/abs/arXiv:1902.02918

    2019 arXiv

  9. [17]

    de A vila Belbute-Peres, K

    F. de A vila Belbute-Peres, K. Smith, K. Allen, J. Tenenbaum, and J. Z. Kolter , End- to-end differentiable physics for learning and control , in Advances in Neural Information Pro- cessing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Ga...

    2018

  10. [18]

    Donti, B

    P. Donti, B. Amos, and J. Z. Kolter , Task-based end-to-end model learning in stochastic optimization, in Advances in Neural Information Processing Systems, 2017, pp. 5484–5494

    2017

  11. [19]

    G. E. Dullerud and F. Paganini , A course in robust control theory: a convex approach , vol. 36, Springer Science & Business Media, 2013

    2013

  12. [20]

    U. Evci, F. Pedregosa, A. Gomez, and E. Elsen , The difficulty of training sparse neural networks , arXiv preprint arXiv:1906.10732, (2019)

    2019 arXiv

  13. [21]

    Gao and L

    B. Gao and L. Pavel , On the properties of the softmax function with application in game theory and reinforcement learning, arXiv preprint arXiv:1704.00805, (2017)

    2017 arXiv

  14. [22]

    I. J. Goodfellow, J. Shlens, and C. Szegedy , Explaining and harnessing adversarial examples , in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015, http://arxiv.org/abs/1412.6572

    2015 arXiv

  15. [23]

    Gowal, K

    S. Gowal, K. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato, R. Arandjelovic, T. A. Mann, and P. Kohli , On the effectiveness of interval bound propagation for training verifiably robust models, CoRR, abs/1810.12715 (2018), http://arxiv.org/abs/1810.12715, https://arxiv.or...

    2018 arXiv

  16. [24]

    F. Gu, A. Askari, and L. El Ghaoui , Fenchel lifted networks: A Lagrange relaxation of neural network training, arXiv preprint arXiv:1811.08039, (2018)

    2018 arXiv

  17. [25]

    S. Han, H. Mao, and W. J. Dally , Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding , arXiv preprint arXiv:1510.00149, (2015)

    2015 arXiv

  18. [26]

    Hassibi, D

    B. Hassibi, D. G. Stork, and G. J. Wolff , Optimal brain surgeon and general network pruning , in IEEE international conference on neural networks, IEEE, 1993, pp. 293–299

    1993

  19. [27]

    K. He, X. Zhang, S. Ren, and J. Sun , Deep residual learning for image recognition , 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2016), pp. 770–778

    2016

  20. [28]

    Kawakami, Supervised sequence labelling with recurrent neural networks , Ph

    K. Kawakami, Supervised sequence labelling with recurrent neural networks , Ph. D. thesis, (2008)

    2008

  21. [29]

    Kolter, Personal communication with A

    J. Kolter, Personal communication with A. Askari . Aug. 2019

    2019

  22. [30]

    Kurakin, I

    A. Kurakin, I. J. Goodfellow, and S. Bengio , Adversarial examples in the physical world , in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Workshop Track Proceedings, OpenReview.net, 2017, https://openreview.net/forum...

    2017

  23. [31]

    Kurakin, I

    A. Kurakin, I. J. Goodfellow, and S. Bengio , Adversarial machine learning at scale , 2017, https: //arxiv.org/abs/1611.01236

    2017 arXiv

  24. [32]

    Lebedev and V

    V. Lebedev and V. Lempitsky , Fast convnets using group-wise brain damage , in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2554–2564. 36 EL GHAOUI ET AL

    2016

  25. [33]

    LeCun, J

    Y. LeCun, J. S. Denker, and S. A. Solla , Optimal brain damage , in Advances in neural information processing systems, 1990, pp. 598–605

    1990

  26. [34]

    J. Li, C. F ang, and Z. Lin , Lifted proximal operator machines, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 4181–4188

    2019

  27. [35]

    Y. Liu, X. Chen, C. Liu, and D. Song , Delving into transferable adversarial examples and black-box attacks, in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net, 2017, https://o...

    2017

  28. [36]

    Louizos, M

    C. Louizos, M. Welling, and D. P. Kingma , Learning sparse neural networks through l 0 regulariza- tion, arXiv preprint arXiv:1712.01312, (2017)

    2017 arXiv

  29. [37]

    Madry, A

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu , Towards deep learning models resistant to adversarial attacks, in International Conference on Learning Representations, 2018, https: //openreview.net/forum?id=rJzIBfZAb

    2018

  30. [38]

    Megretski and A

    A. Megretski and A. Rantzer , System analysis via integral quadratic constraints , IEEE Transactions on Automatic Control, 42 (1997), pp. 819–830

    1997

  31. [39]

    C. D. Meyer , Matrix analysis and applied linear algebra , vol. 71, Siam, 2000

    2000

  32. [40]

    Narang, E

    S. Narang, E. Elsen, G. Diamos, and S. Sengupta , Exploring sparsity in recurrent neural networks , arXiv preprint arXiv:1704.05119, (2017)

    2017 arXiv

  33. [41]

    Papernot, P

    N. Papernot, P. D. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami , The limitations of deep learning in adversarial settings , in IEEE European Symposium on Security and Privacy, EuroS&P 2016, Saarbr¨ ucken, Germany, March 21-24, 2016, IEEE, 2016, pp. 372–387, http...

    2016 doi

  34. [42]

    Papernot, P

    N. Papernot, P. D. McDaniel, X. Wu, S. Jha, and A. Swami , Distillation as a defense to adversarial perturbations against deep neural networks , in IEEE Symposium on Security and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016, IEEE Computer Society, 2016, pp. 582–597, ht...

    2016 doi

  35. [43]

    Raghunathan, J

    A. Raghunathan, J. Steinhardt, and P. S. Liang , Semidefinite relaxations for certifying robustness to adversarial examples , in Advances in Neural Information Processing Systems, 2018, pp. 10877– 10887

    2018

  36. [44]

    Sastry, Nonlinear systems: analysis, stability, and control, vol

    S. Sastry, Nonlinear systems: analysis, stability, and control, vol. 10, Springer Science & Business Media, 2013

    2013

  37. [45]

    See, M.-T

    A. See, M.-T. Luong, and C. D. Manning , Compression of neural machine translation models via pruning, arXiv preprint arXiv:1606.09274, (2016)

    2016 arXiv

  38. [46]

    Shaham, Y

    U. Shaham, Y. Yamada, and S. Negahban , Understanding adversarial training: Increasing local stabil- ity of neural nets through robust optimization , (2015), https://doi.org/10.1016/j.neucom.2018.04.027, https://arxiv.org/abs/arXiv:1511.05432

    2015 arXiv

  39. [47]

    Simonyan, A

    K. Simonyan, A. Vedaldi, and A. Zisserman , Deep inside convolutional networks: Visualising image classification models and saliency maps, in 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, Y....

    2014 arXiv

  40. [48]

    Srivastava, G

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov , Dropout: a simple way to prevent neural networks from overfitting , The journal of machine learning research, 15 (2014), pp. 1929–1958

    2014

  41. [49]

    Stallkamp, M

    J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel , The german traffic sign recognition benchmark: a multi-class classification competition, in The 2011 international joint conference on neural networks, IEEE, 2011, pp. 1453–1460

    2011

  42. [50]

    Taylor, R

    G. Taylor, R. Burmeister, Z. Xu, B. Singh, A. Patel, and T. Goldstein , Training neural networks without gradients: A scalable ADMM approach, in International conference on machine learning, 2016, pp. 2722–2731

    2016

  43. [51]

    Travacca, S

    B. Travacca, S. Moura, and L. El Ghaoui , Implicit optimization: Models and methods [to be pub- lished, cdc 2020]

    2020

  44. [52]

    W ang, P

    P.-W. W ang, P. Donti, B. Wilder, and Z. Kolter , SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver , in Proceedings of the 36th International Con- ference on Machine Learning, K. Chaudhuri and R. Salakhutdinov, eds., vol. 97 of Pr...

    2019

  45. [53]

    W ang, P

    P.-W. W ang, P. L. Donti, B. Wilder, and Z. Kolter , Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver , arXiv preprint arXiv:1905.12149, (2019)

    2019 arXiv

  46. [54]

    Wong and J

    E. Wong and J. Z. Kolter , Provable defenses against adversarial examples via the convex outer ad- versarial polytope, 2017, https://arxiv.org/abs/arXiv:1711.00851

    2017 arXiv

  47. [55]

    Y. Yu, X. Si, C. Hu, and J. Zhang , A review of recurrent neural networks: Lstm cells and network architectures, Neural computation, 31 (2019), pp. 1235–1270

    2019

  48. [56]

    J. Zeng, T. T.-K. Lau, S. Lin, and Y. Yao , Global convergence of block coordinate descent in deep learning, arXiv preprint arXiv:1803.00225, (2018)

    2018 arXiv

  49. [57]

    Zhang and M

    Z. Zhang and M. Brand , Convergent block coordinate descent for training Tikhonov regularized deep neural networks, in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, USA, 2017, Curran Associates Inc., pp. 1719–1728, http://d...

    2017

  50. [58]

    vectorized

    M. Zhu and S. Gupta , To prune, or not to prune: exploring the efficacy of pruning for model compres- sion, arXiv preprint arXiv:1710.01878, (2017). 38 EL GHAOUI ET AL. Appendix A. Proof of Theorem 2.2. Let b∈ Rn. We first prove the existence of a solution ξ∈ Rn to the equation ξ...

    2017 arXiv

  51. [59]

    Form a vector g(λ) containing the sub-gradients corresponding to each row, evaluated at λi, i∈ [n]: g(λ) =−κ1 + (|A0|− λ1T )T +1

  52. [60]

    For every i∈ [n], reset λu i =λi if gi(λ)> 0, λl i =λi if gi(λ)≤ 0

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.