Pith. sign in

REVIEW 4 major objections 5 minor 56 references

An analysis of optimization problems involving ReLU neural networks

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A ReLU network's redundancy—linear regions and unstable neurons—governs how hard it is to optimize over it; of six interventions tested on 1,080 networks, L1 regularization speeds solvers up most.

desk verdict Solid empirical comparison with a genuinely new ReLU scaling trick; the regularization speedup is believable but the reported factors need censored-runtime robustness checks. read the letter →

arxiv 2502.03016 v1 pith:XHTMCYR4 submitted 2025-02-05 math.OC cs.LG

Christoph Plate , Mirko Hahn , Alexander Klimek , Caroline Ganzer , Kai Sundmacher , Sebastian Sager This is my paper · ORCID
classification math.OCcs.LG MSC 90C1190C2668T07
keywords ReLUneuralnetworksmixed-integerprogrammingbig-MformulationboundtighteningL1regularizationlinearregionsstableneuronssurrogateoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks a practical question: when a trained ReLU neural network is embedded in a mixed-integer optimization problem, what determines how long a solver takes, and which interventions shorten that time? The authors compare four families of approaches—optimization-based bound tightening, a new a-posteriori scaling that exploits ReLU's positive homogeneity, clipped-ReLU and dropout variations during training, and L1 regularization—across 1,080 networks trained on three benchmark functions. Their central finding is a quantified trade-off: the redundancy that makes networks accurate and robust (many linear regions, few stable neurons) is exactly what makes the associated optimization hard. L1 regularization is the most effective lever, shrinking big-M coefficients, raising the percentage of stable neurons, cutting linear regions, and accelerating solvers by over an order of magnitude; the price is some loss of test accuracy at high regularization levels. A process-engineering application mentioned in the paper was solved for the first time after training with regularization, which suggests the trade-off matters in practice.

What carries the argument

The load-bearing identity is the positive homogeneity of ReLU, $\mathrm{ReLU}(c\cdot z) = c\cdot\mathrm{ReLU}(z)$ for $c > 0$, which lets any trained network be re-parameterized, row-by-row and column-by-column, into a functionally identical network with smaller weights and biases (equations 9 and 10). The scaling factors are chosen by a convex optimization problem obtained by substituting logarithms for the positive factors (problem 12), whose objective is a sum of exponentials that drives down the $\ell^1$ norm of weights and biases and with it the big-M coefficients of formulation (5). Two complexity measures carry the analysis: the number of linear regions, computed by a forward-sensitivity enumeration that walks across facets using the QuickHull algorithm, and the percentage of stable neurons, determined from pre-activation interval bounds; the paper uses both to explain the runtime behavior of Gurobi on problem (18).

What would settle it

Re-run the same six treatments on an embedded problem with more than two inputs and additional constraints linking the network output to other decision variables—for instance, the chemical-engineering superstructure problem the paper cites as having become tractable only after regularization—and test whether L1 regularization still dominates; a negative result would falsify the transfer claim. A second check targets the scaling-plus-OBBT speedup of 0.467, which the paper itself notes may be driven by a few outlier instances (Figure 5b): recompute the geometric mean after removing the extreme instances and compare the median speedup instead.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that for the big-M mixed-integer formulation of a feed-forward ReLU network, the computational cost of global optimization is governed by three linked quantities: the magnitude of the big-M coefficients (which grow exponentially with depth under interval arithmetic), the percentage of neurons whose activation is stable over the input domain, and the number of linear regions. Every method the paper tests acts on these quantities. LP-based bound tightening reduces big-M coefficients by roughly half and geometric-mean runtime to 0.57 of baseline; the novel ReLU scaling alone barely helps (0.936), but scaling followed by bound tightening achieves tighter hidden-layer bounds than either alone (geometric-mean bound width 0.16 of baseline) and a runtime ratio of 0.467. Clipped ReLU shrinks bounds but costs extra binary variables and gains little. Dropout is the antithesis, multiplying the number of linear regions by three to four, cutting stable neurons by about 20 percentage points, and slowing solvers four- to sixfold. The most effective method is L1 regularization: at $\lambda = 10^{-3}$, big-M coefficients drop to a geometric mean of 0.009 of baseline, stable neurons rise, linear regions fall to under a third, and runtime falls to 0.028 of baseline, making almost all instances with up to ten layers solvable within the 300-second time limit.

Load-bearing premise

The benchmark evidence covers only the direct minimization of a single feed-forward ReLU network's output on three two-dimensional test functions with default Gurobi settings and a 300-second time limit, and the paper's ranking of methods is assumed to carry over to the embedded problems with extra variables and constraints that motivate the work (Section 4).

Editorial extensions

If this is right

  • Unregularized networks deeper than four layers are effectively unsolvable on the paper's benchmarks, while moderate L1 regularization ($\lambda$ between $10^{-5}$ and $10^{-3}$) makes almost all ten-layer instances tractable, so regularization is the method of choice when a surrogate may later be embedded in an optimization.
  • Sequential application of the new scaling method and LP-based bound tightening produces tighter hidden-layer big-M bounds (geometric-mean width 0.16 of baseline) than either alone and removes the exponential bound growth with depth in the tested ten-layer networks.
  • Because clipped ReLU reduces big-M coefficients yet barely changes runtimes, the extra binary variable it introduces largely cancels the benefit of tighter bounds.
  • Dropout training, which the paper uses as its antithesis, raises linear-region counts three- to fourfold and runtime four- to sixfold, quantitatively confirming that redundancy is a cost rather than a free good.
  • Because minimizing the output of a network is the same problem class as adversarial-robustness verification with a perturbation constraint around an input, the speedups measured for problem (18) should transfer directly to certification tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test it, but the redundancy-cost law it quantifies should hold for any training-time intervention: techniques that inflate piecewise-linear complexity (ensembling, data augmentation, wider layers) should raise MIP solve times, while complexity-reducing ones (pruning, stability regularization, distillation) should lower them; regressing Gurobi runtime on linear-region count acros
  • The scaling method works on any pretrained network without changing the function it encodes, so scaling plus bound tightening forms a drop-in preprocessing pair for legacy surrogates that were never trained with optimization in mind; the paper's geometric-mean speedup of 0.467 for the pair may be outlier-driven, as its own parity plot suggests, so the median speedup is worth reporting.
  • The scaling objective (12) minimizes the $\ell^1$ norm of weights and biases; composing it with lossless compression that removes layers of provably stable neurons could shrink both norm and network size, a combination the paper leaves untested.
  • Linear-region counts could be turned into an a-priori difficulty score for embedded optimization, letting a user choose a training regime before ever calling a solver; the paper reports the correlation but does not build the predictor.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies how different training-time and post-training modifications to feed-forward ReLU neural networks affect the time needed to solve mixed-integer optimization problems in which the network appears as an embedded constraint or objective. It surveys big-M formulations, interval-arithmetic and LP-based bound tightening, clipped ReLU activations, dropout, and ℓ1 regularization, and it proposes a new a posteriori scaling procedure that exploits the positive homogeneity of ReLU to rescale weights and biases (without changing the encoded function) so as to reduce big-M coefficients. The numerical study trains 1080 networks on three 2D benchmark functions (Peaks, Ackley, Himmelblau) and solves the unconstrained minimization problem min_x h(x) with Gurobi under a 300-second time limit. The main empirical conclusions are that ℓ1 regularization is the most effective way to speed up subsequent optimization, that the proposed scaling helps mainly when combined with bound tightening, that clipped ReLU gives only marginal speedups, and that dropout degrades performance. The paper also quantifies a trade-off between the redundancy of a ReLU model (measured by number of linear regions and percentage of stable neurons) and the computational cost of optimizing over it.

Significance. If the empirical claims hold, the paper is a useful systematic benchmark for practitioners choosing between training-time interventions and post-processing tools for optimization with embedded ReLU networks. Its strengths include a clean, parameter-free derivation of the ReLU scaling transform (Section 2.3), a large controlled benchmark (1080 networks, three functions, consistent solver settings), and transparent reporting of solved-instance counts alongside runtime ratios and parity plots. The finding that ℓ1 regularization is substantially more effective than post-training bound tightening or scaling for the tested setting is actionable and qualitatively consistent with prior work on ReLU stability. The main limitations are that the speedup evidence is based on runtime ratios over only the instances solved by both configurations, and that the regularization comparison does not separate the effect of reduced redundancy from the effect of degraded surrogate accuracy.

major comments (4)
  1. [Section 2.3, Eqs. (11)-(12)] The optimization problem defining the scaling method is not a correct transcription of the equivalent-scaling transformation described in Eq. (10). Eq. (10) introduces scaling factors only for layers 1 through J-1, with the final layer compensated by 1/c_j^(J-1). However, Eq. (11) contains a term with 1/c_i^(J), and Eq. (12) not only repeats that misindexing but also includes a k=J term in the main weight sum and includes the final-layer bias in the bias sum, even though final-layer biases are not scaled. As printed, the objective being solved is not the one corresponding to the class of functionally equivalent networks defined by (10). Please correct the formulas and confirm whether the numerical results were obtained with the corrected objective or with the printed one.
  2. [Table 2 and Section 3.1] The geometric-mean time ratios in the last column of Table 2 are computed only over instances solved to global optimality by both the adapted and the baseline configuration. This restriction is particularly severe for the ℓ1 regularization rows: the baseline (lambda = 0) solves only 151 of 360 instances, while the regularized configurations solve 287 to 358. The ratio therefore excludes precisely the harder instances in which the baseline hits the 300-second time limit, so the reported speedups (geometric means from 0.028 to 0.280) may substantially overstate the practical benefit. The paper should provide a robustness analysis that accounts for censored instances, for example by imputing the time limit, reporting performance profiles, or reporting time ratios separately by function and depth.
  3. [Section 3.3 and Figure 6] The claim that ℓ1 regularization is the most effective method is partially confounded by surrogate accuracy. Figure 6 shows that larger regularization parameters noticeably degrade the test-set MAPE, especially for Ackley's function, and the reported speedups are obtained on the networks that result from these regularization levels. Faster optimization of a less accurate surrogate does not by itself establish that regularization is preferable in an application where approximation quality matters. The paper should separate the redundancy-reduction effect from the accuracy loss, for example by comparing networks with matched MAPE or by reporting an accuracy-corrected performance metric alongside raw runtime.
  4. [Section 3.1 and Section 2.2.2] It is not clear whether the runtimes reported for the OBBT comparison in Table 2 include the time spent solving the auxiliary LPs in (8). If the reported times are only the Gurobi solution times for (18), then the claimed OBBT speedup of 0.57 is not an end-to-end speedup because the bound-tightening effort is excluded. Please state explicitly how the reported times are measured and, if the OBBT time is excluded, add the overhead and report total time including bound tightening.
minor comments (5)
  1. [Section 2.3, Eq. (12)] The logarithm transformation is undefined for exactly zero weights or biases, which can occur in principle under ℓ1 regularization; the paper should state how such entries are handled (for example, by omitting the corresponding term or using an epsilon-regularized logarithm).
  2. [Section 3.3, paragraph after Figure 6] The text says 'regularization proved the most effective method,' but the comparison mixes training-time interventions with post-processing methods and different instance sets; it would be more precise to say 'most effective among the tested methods on these benchmarks.'
  3. [Section 1 and Section 4] The abstract and conclusions state a general trade-off between model redundancy and computational cost, but the supporting experiments are limited to unconstrained minimization of a single network output on three 2D functions (problem (18)). The paper already acknowledges this in Section 4, but the title and abstract would benefit from an explicit scope qualifier such as 'for the benchmark setting considered.'
  4. [General] No information is given about code or data availability; providing the trained network instances and the enumeration/scaling scripts would materially improve reproducibility of the reported geometric means and linear-region counts.
  5. [Section 2.1.2] The linear-region enumeration assumes that no two neurons switch simultaneously on a facet and states that this behavior was not observed; it would help to report how frequently the check for empty-interior regions triggered, since the assumption is relied upon for the correctness of the region counts.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central speedup and trade-off claims are empirical measurements against an external solver, and the scaling method follows from the ReLU homogeneity identity without fitted constants.

full rationale

The only proposed derivation in the paper is the equivalence transformation in Section 2.3: from ReLU(c v) = c ReLU(v) (Eq. 9), the authors derive the scaled weights and biases (Eq. 10) and the convex log-space scaling objective (Eq. 12). This derivation contains no fitted parameter, no assumption that includes the target result, and no imported uniqueness theorem; it simply re-expresses functional equivalence under positive homogeneity. The claims that follow, such as big-M reduction, stable-neuron counts, linear-region counts, and runtime ratios, are measured outcomes obtained by comparing Gurobi v11.0.1 with default settings on 1080 trained networks (Table 2), not predictions recovered from the method's own outputs. Regularization and dropout effects are likewise measured against baseline networks and an external solver, so no fitted input is renamed as a prediction. Self-citations are present, e.g., Klimek et al. (2024) in Section 4 and Joseph-Duran et al. (2014) as modeling context, but neither is load-bearing for the benchmark ranking or the scaling derivation. Section 4 explicitly labels transfer to other contexts as an extrapolation ('we believe that the findings are also applicable in other contexts'), which is a stated belief rather than a circular derivation. A skeptical concern about the geometric-mean time ratios being restricted to instances solved by both configurations is a robustness issue, not a circularity issue, and does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central methodological contribution, equivalent scaling, introduces no new fitted constants or postulated entities; it optimizes the existing weights. The experimental conclusions, however, rest on hand-chosen hyperparameters (lambda, M, time limit) and on the validity of the LP relaxation and the linear-region enumeration assumptions.

free parameters (3)
  • L1 regularization coefficient lambda = 0, 1e-7, 1e-6, 1e-5, 1e-4, 1e-3
    Chosen by hand to probe the effect of regularization strength on runtime and accuracy; not fit to data. The paper's conclusion that regularization speeds up optimization depends on this set of levels.
  • Clipped ReLU threshold M = 2 and 5
    Chosen by hand for the clipped ReLU experiments; the effect of clipping on big-M coefficients and runtime depends on M.
  • Gurobi time limit = 300 seconds
    Chosen as the experimental stopping criterion; it determines the 'solved within time limit' counts and thus the runtime ratio statistics.
assumptions (4)
  • standard math ReLU positive homogeneity: ReLU(c x) = c ReLU(x) for c > 0
    Basis of the equivalent scaling transform in Eq. (9) and the functional equivalence of the scaled network (Eq. 10).
  • domain assumption LP relaxation of (8) yields valid bounds on neuron pre-activations
    Used in Section 2.2.2 to justify LP-based bound tightening as a source of valid big-M coefficients.
  • domain assumption Generic position: no facet of a linear region is created by two neurons switching simultaneously
    Assumed by the QuickHull-based linear region enumeration in Section 2.1.2; the authors state 'We do not observe this behavior' but do not prove it.
  • domain assumption The log-transformed scaling problem (12) is solved to global optimality
    Problem (12) is convex, so the authors rely on a convex solver finding the global minimum, but no solver tolerance or verification is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An analysis of optimization problems involving ReLU neural networks." pith.science (2026). https://pith.science/paper/XHTMCYR4

@misc{pith2026250203016,
  author       = {Pith},
  title        = {Pith review of: An analysis of optimization problems involving ReLU neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XHTMCYR4}},
  note         = {Machine review of arXiv:2502.03016}
}
read the original abstract

Solving mixed-integer optimization problems with embedded neural networks with ReLU activation functions is challenging. Big-M coefficients that arise in relaxing binary decisions related to these functions grow exponentially with the number of layers. We survey and propose different approaches to analyze and improve the run time behavior of mixed-integer programming solvers in this context. Among them are clipped variants and regularization techniques applied during training as well as optimization-based bound tightening and a novel scaling for given ReLU networks. We numerically compare these approaches for three benchmark problems from the literature. We use the number of linear regions, the percentage of stable neurons, and overall computational effort as indicators. As a major takeaway we observe and quantify a trade-off between the often desired redundancy of neural network models versus the computational costs for solving related optimization problems.

Figures

Figures reproduced from arXiv: 2502.03016 by the authors.

Figure 1
Figure 1. Equivalent scaling of ReLU ANNs. Scalar factor [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Surface plots of the benchmark functions for surrogate model training and optimization. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Comparison of pre-activation bounds U (k) for functionally equivalent neural networks with ten hidden layers. The original bounds derived via interval arithmetic shown in 3a are characterized by the typical exponential increase due to forward propagation of the input bounds. Solving auxiliary LPs yields tighter bounds, although the exponential increase is still present, as shown in 3b. Comparable bounds can be compu… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Parity plots comparing percentage of stable neurons and computational times of optimally [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Parity plots comparing computational times for optimally solved instances of ( [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Mean absolute percentage error on test set, number of linear regions, percentage of stable [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Linear regions for ReLU networks approximating the Peaks function ( [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Comparison of pre-activation bounds U (k) for neural networks with ten hidden layers and clipped ReLU formulation (17) with M = 5.0 as activation function. Compared to the bounds derived via interval arithmetic for the regular ReLU activation shown in Figure 3a on page…
Figure 9
Figure 9. Figure 9: Mean absolute percentage error on the test set, number of linear regions, percentage of fixed [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 34 canonical work pages

  1. [1]

    Mathematical Programming 183(1):3--39, doi:10.1007/s10107-020-01474-5

    Anderson R, Huchette J, Ma W, Tjandraatmadja C, Vielma JP (2020) Strong mixed-integer programming formulations for trained neural networks. Mathematical Programming 183(1):3--39, doi:10.1007/s10107-020-01474-5

    doi:10.1007/s10107-020-01474-5 2020

  2. [2]

    doi:10.48550/arXiv.2312.16699, 2312.16699

    Badilla F, Goycoolea M, Mu \ n oz G, Serra T (2023) Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networks . doi:10.48550/arXiv.2312.16699, 2312.16699

    arXiv doi:10.48550/arxiv.2312.16699 2023

  3. [3]

    ACM Trans Math Softw 22(4):469--483, doi:10.1145/235815.235821, ://doi.org/10.1145/235815.235821

    Barber CB, Dobkin DP, Huhdanpaa H (1996) The quickhull algorithm for convex hulls. ACM Trans Math Softw 22(4):469--483, doi:10.1145/235815.235821, ://doi.org/10.1145/235815.235821

    work page arXiv 1996

  4. [4]

    INFORMS J on Computing 34(4):2229--2248, doi:10.1287/ijoc.2022.1181

    Bertsimas D, Stellato B (2022) Online Mixed-Integer Optimization in Milliseconds . INFORMS J on Computing 34(4):2229--2248, doi:10.1287/ijoc.2022.1181

    work page arXiv 2022

  5. [5]

    Management Science 62(5):1511--1531, doi:10.1287/mnsc.2015.2363

    Bertsimas D, O'Hair A, Relyea S, Silberholz J (2016) An Analytics Approach to Designing Combination Chemotherapy Regimens for Cancer . Management Science 62(5):1511--1531, doi:10.1287/mnsc.2015.2363

    work page arXiv 2016

  6. [6]

    Springer Science & Business Media

    Bynum ML, Hackebeil GA, Hart WE, Laird CD, Nicholson BL, Siirola JD, Watson JP, Woodruff DL (2021) Pyomo--optimization modeling in python, vol 67, 3rd edn. Springer Science & Business Media

    work page 2021

  7. [7]

    Operations Research Letters 57:107194, doi:10.1016/j.orl.2024.107194, ://www.sciencedirect.com/science/article/pii/S0167637724001305

    Cacciola M, Frangioni A, Lodi A (2024) Structured pruning of neural networks for constraints learning. Operations Research Letters 57:107194, doi:10.1016/j.orl.2024.107194, ://www.sciencedirect.com/science/article/pii/S0167637724001305

    arXiv 2024

  8. [8]

    Physics Reports 1044:1--68

    Camps-Valls G, Gerhardus A, Ninad U, Varando G, Martius G, Balaguer-Ballester E, Vinuesa R, Diaz E, Zanna L, Runge J (2023) Discovering causal relations and equations from data. Physics Reports 1044:1--68

    work page 2023

Show all 56 references
  1. [9]

    Journal of Machine Learning Research 23(349):1--8

    Ceccon F, Jalving J, Haddad J, Thebelt A, Tsay C, Laird CD, Misener R (2022) OMLT : Optimization & Machine Learning Toolkit . Journal of Machine Learning Research 23(349):1--8

    2022

  2. [10]

    (2015) Keras

    Chollet F, et al. (2015) Keras. https://keras.io

    2015

  3. [11]

    Mathematics of Control, Signals and Systems 2(4):303--314, doi:10.1007/BF02551274

    Cybenko G (1989) Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems 2(4):303--314, doi:10.1007/BF02551274

    1989 doi

  4. [12]

    Chemical Engineering & Technology 29(4):449--453, doi:10.1002/ceat.200500310

    Fernandes FaN (2006) Optimization of Fischer-Tropsch Synthesis Using Neural Networks . Chemical Engineering & Technology 29(4):449--453, doi:10.1002/ceat.200500310

    2006 doi

  5. [13]

    Constraints 23(3):296--309, doi:10.1007/s10601-018-9285-6

    Fischetti M, Jo J (2018) Deep neural networks and mixed integer linear optimization. Constraints 23(3):296--309, doi:10.1007/s10601-018-9285-6

    2018 doi

  6. [14]

    SIAM Journal on Applied Algebra and Geometry 6(2):216--242, doi:10.1137/20M1368902, ://doi.org/10.1137/20M1368902, https://doi.org/10.1137/20M1368902

    Grigsby JE, Lindsey K (2022) On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks. SIAM Journal on Applied Algebra and Geometry 6(2):216--242, doi:10.1137/20M1368902, ://doi.org/10.1137/20M1368902, https://doi.org/10.1137...

    2022 doi

  7. [15]

    Computers & Chemical Engineering 131:106580, doi:10.1016/j.compchemeng.2019.106580

    Grimstad B, Andersson H (2019) ReLU Networks as Surrogate Models in Mixed-Integer Linear Programs . Computers & Chemical Engineering 131:106580, doi:10.1016/j.compchemeng.2019.106580

    2019

  8. [16]

    ://www.gurobi.com

    Gurobi Optimization, LLC (2024) Gurobi Optimizer Reference Manual . ://www.gurobi.com

    2024

  9. [17]

    In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, MIT Press, Cambridge, MA, USA, NIPS '15, pp 1135--1143

    Han S, Pool J, Tran J, Dally WJ (2015) Learning both weights and connections for efficient neural networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, MIT Press, Cambridge, MA, USA, NIPS '15, pp 1135--1143

    2015

  10. [18]

    doi:10.48550/arXiv.1412.5567, 1412.5567

    Hannun A, Case C, Casper J, Catanzaro B, Diamos G, Elsen E, Prenger R, Satheesh S, Sengupta S, Coates A, Ng AY (2014) Deep Speech : Scaling up end-to-end speech recognition. doi:10.48550/arXiv.1412.5567, 1412.5567

    2014 doi arXiv

  11. [19]

    Mathematical Programming Computation 3(3):219--260

    Hart WE, Watson JP, Woodruff DL (2011) Pyomo: modeling and solving mathematical programs in python. Mathematical Programming Computation 3(3):219--260

    2011

  12. [20]

    In: Proceedings of the 31st International Conference on Neural Information Processing Systems , Curran Associates Inc., Red Hook, NY, USA, NIPS '17, pp 2263--2273

    Hein M, Andriushchenko M (2017) Formal guarantees on the robustness of a classifier against adversarial manipulation. In: Proceedings of the 31st International Conference on Neural Information Processing Systems , Curran Associates Inc., Red Hook, NY, USA, NIPS '17, pp 2263--2273

    2017

  13. [21]

    Neural Networks 4(2):251--257, doi:10.1016/0893-6080(91)90009-T

    Hornik K (1991) Approximation capabilities of multilayer feedforward networks. Neural Networks 4(2):251--257, doi:10.1016/0893-6080(91)90009-T

    1991 doi

  14. [22]

    doi:10.48550/arXiv.2305.00241, 2305.00241

    Huchette J, Mu \ n oz G, Serra T, Tsay C (2023) When Deep Learning Meets Polyhedral Theory : A Survey . doi:10.48550/arXiv.2305.00241, 2305.00241

    2023 doi arXiv

  15. [23]

    W ater R esources M anagement 28(1):41--63, doi:10.1007/s11269-013-0468-z

    Joseph-Duran B, Jung M, Ocampo-Martinez C, Sager S, Cambrano G (2014) M inimization of S ewage N etwork O verflow. W ater R esources M anagement 28(1):41--63, doi:10.1007/s11269-013-0468-z

    2014 doi

  16. [24]

    ://arxiv.org/abs/1412.6980, 1412.6980

    Kingma DP, Ba J (2017) Adam: A method for stochastic optimization. ://arxiv.org/abs/1412.6980, 1412.6980

    2017 arXiv

  17. [25]

    In: ESCAPE34 , ://www.aidic.it/BOA/24/BOA2401.pdf

    Klimek A, Ganzer C, Sundmacher K (2024) Enhancing superstructure optimization via embedded neural networks in optimal process design for sustainable aviation fuel ( SAF ) production. In: ESCAPE34 , ://www.aidic.it/BOA/24/BOA2401.pdf

    2024

  18. [26]

    In: Pereira F, Burges C, Bottou L, Weinberger K (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., vol 25

    Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Pereira F, Burges C, Bottou L, Weinberger K (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., vol 25

    2012

  19. [27]

    doi:10.48550/arXiv.2202.05198, 2202.05198

    Kronqvist J, Misener R, Tsay C (2024) P-split formulations: A class of intermediate formulations between big- M and convex hull for disjunctive constraints. doi:10.48550/arXiv.2202.05198, 2202.05198

    2024 doi arXiv

  20. [28]

    doi:10.48550/arXiv.1905.11428, 1905.11428

    Kumar A, Serra T, Ramalingam S (2019) Equivalent and Approximate Transformations of Deep Neural Networks . doi:10.48550/arXiv.1905.11428, 1905.11428

    2019 doi arXiv

  21. [29]

    arXiv preprint arXiv:240209092

    Kunc V, Kl \'e ma J (2024) Three decades of activations: A comprehensive survey of 400 activation functions for neural networks. arXiv preprint arXiv:240209092

    2024

  22. [30]

    https://doi.org/10.5281/zenodo.5601907, doi:https://doi.org/10.5281/zenodo.5601907, ://github.com/ChemEngAI/ReLU_ANN_MILP

    Lueg L, Grimstad B, Mitsos A, Schweidtmann AM (2021) reluMIP : Open source tool for MILP optimization of ReLU neural networks. https://doi.org/10.5281/zenodo.5601907, doi:https://doi.org/10.5281/zenodo.5601907, ://github.com/ChemEngAI/ReLU_ANN_MILP

    2021 doi

  23. [31]

    Computers & Chemical Engineering 179:108411, doi:10.1016/j.compchemeng.2023.108411

    Misener R, Biegler L (2023) Formulating data-driven surrogate models for process optimization. Computers & Chemical Engineering 179:108411, doi:10.1016/j.compchemeng.2023.108411

    2023

  24. [32]

    In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, MIT Press, Cambridge, MA, USA, NIPS '14, vol 2, pp 2924--2932

    Mont \'u far G, Pascanu R, Cho K, Bengio Y (2014) On the number of linear regions of deep neural networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, MIT Press, Cambridge, MA, USA, NIPS '14, vol 2, pp 2924--2932

    2014

  25. [33]

    Chemical Engineering Research and Design 84(8):635--644, doi:10.1205/cherd.05096

    Mujtaba IM, Aziz N, Hussain MA (2006) Neural Network Based Modelling and Control in Batch Reactor . Chemical Engineering Research and Design 84(8):635--644, doi:10.1205/cherd.05096

    2006 doi

  26. [34]

    Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: An imperative style, high-performance deep learn...

    2019

  27. [35]

    In: Proceedings of the 34th International Conference on Machine Learning , PMLR, pp 2847--2854

    Raghu M, Poole B, Kleinberg J, Ganguli S, Sohl-Dickstein J (2017) On the Expressive Power of Deep Neural Networks . In: Proceedings of the 34th International Conference on Machine Learning , PMLR, pp 2847--2854

    2017

  28. [36]

    Journal of Global Optimization 81(1):109--152, doi:10.1007/s10898-020-00949-1

    R \"o ssig A, Petkovic M (2021) Advances in verification of ReLU neural networks. Journal of Global Optimization 81(1):109--152, doi:10.1007/s10898-020-00949-1

    2021 doi

  29. [37]

    Chemie Ingenieur Technik 93(12):2029--2039, doi:10.1002/cite.202100083

    Schweidtmann A, Esche E, Fischer A, Kloft M, Repke J, Sager S, Mitsos A (2021) Machine learning in chemical engineering: A perspective. Chemie Ingenieur Technik 93(12):2029--2039, doi:10.1002/cite.202100083

    2021 doi

  30. [38]

    Journal of Optimization Theory and Applications 180(3):925--948, doi:10.1007/s10957-018-1396-0

    Schweidtmann AM, Mitsos A (2019) Deterministic Global Optimization with Artificial Neural Networks Embedded . Journal of Optimization Theory and Applications 180(3):925--948, doi:10.1007/s10957-018-1396-0

    2019 doi

  31. [39]

    Serra T, Kumar A, Ramalingam S (2020) Lossless Compression of Deep Neural Networks . In: Hebrard E, Musliu N (eds) Integration of Constraint Programming , Artificial Intelligence , and Operations Research , Springer International Publishing, Cham, pp 417--430, doi:10.1007/978-...

    2020 doi

  32. [40]

    In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics , Speech and Signal Processing ( ICASSP ), pp 4192--4196, doi:10.1109/ICASSP40776.2020.9053540

    Smyrnis G, Maragos P, Retsinas G (2020) Maxpolynomial Division with Application To Neural Network Simplification . In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics , Speech and Signal Processing ( ICASSP ), pp 4192--4196, doi:10.1109/ICASSP40776.2020.9053540

    2020

  33. [41]

    Journal of Machine Learning Research 15(56):1929--1958, ://jmlr.org/papers/v15/srivastava14a.html

    Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: A Simple Way to Prevent Neural Networks from Overfitting . Journal of Machine Learning Research 15(56):1929--1958, ://jmlr.org/papers/v15/srivastava14a.html

    2014

  34. [42]

    Suzuki T, Abe H, Murata T, Horiuchi S, Ito K, Wachi T, Hirai S, Yukishima M, Nishimura T (2020) Spectral Pruning : Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error . In: Proceedings of the Twenty-Ninth International Joint Conference on Artifi...

    2020 doi

  35. [43]

    Journal of the Royal Statistical Society Series B (Methodological) 58(1):267--288, 2346178

    Tibshirani R (1996) Regression Shrinkage and Selection via the Lasso . Journal of the Royal Statistical Society Series B (Methodological) 58(1):267--288, 2346178

    1996

  36. [44]

    doi:10.48550/arXiv.1711.07356, 1711.07356

    Tjeng V, Xiao K, Tedrake R (2019) Evaluating Robustness of Neural Networks with Mixed Integer Programming . doi:10.48550/arXiv.1711.07356, 1711.07356

    2019 doi arXiv

  37. [45]

    Tong J, Cai J, Serra T (2024) Optimization over Trained Neural Networks : Taking a Relaxing Walk . In: Dilkina B (ed) Integration of Constraint Programming , Artificial Intelligence , and Operations Research , Springer Nature Switzerland, Cham, pp 221--233, doi:10.1007/978-3-0...

    2024 doi

  38. [46]

    doi:10.48550/arXiv.2102.04373, 2102.04373

    Tsay C, Kronqvist J, Thebelt A, Misener R (2021) Partition-based formulations for mixed-integer optimization of trained ReLU neural networks. doi:10.48550/arXiv.2102.04373, 2102.04373

    2021 doi arXiv

  39. [47]

    doi:10.48550/arXiv.2312.08074, 2312.08074

    Turner M, Chmiela A, Koch T, Winkler M (2024) PySCIPOpt-ML : Embedding Trained Machine Learning Models into Mixed-Integer Programs . doi:10.48550/arXiv.2312.08074, 2312.08074

    2024 doi arXiv

  40. [48]

    Nature Methods 17:261--272, doi:10.1038/s41592-019-0686-2

    Virtanen P, Gommers R, Oliphant TE, Haberland M, Reddy T, Cournapeau D, Burovski E, Peterson P, Weckesser W, Bright J, van der Walt SJ, Brett M, Wilson J, Millman KJ, Mayorov N, Nelson ARJ, Jones E, Kern R, Larson E, Carey CJ, Polat I, Feng Y, Moore EW, VanderPlas J, Laxalde D...

    2020

  41. [49]

    1809.03008

    Xiao KY, Tjeng V, Shafiullah NM, Madry A (2019) Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability . 1809.03008

    2019 arXiv

  42. [50]

    In: 8th International Conference on Learning Representations, ://iclr.cc/virtual_2020/poster_SkeFl1HKwr.html

    Zhang X, Wu D (2020) Empirical Studies on the Properties of Linear Regions in Deep Neural Networks . In: 8th International Conference on Learning Representations, ://iclr.cc/virtual_2020/poster_SkeFl1HKwr.html

    2020

  43. [51]

    , " * write output.state after.block = add.period write newline

    ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mi...

  44. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap dupl...

  45. [53]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or...

  47. [55]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.