Pith. sign in

REVIEW 3 major objections 5 minor 68 references

Learning by solving differential equations

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Higher-order Runge-Kutta optimizers for deep learning become practical when momentum is applied to their gradient estimates, closing the large-batch generalization gap and beating Adam on MNIST and Fashion-MNIST.

desk verdict A candid, useful empirical study of RK4 for deep learning with honest limitations, but its central 'better than Adam' claim is under-supported by unmatched tuning and missing error bars. read the letter →

arxiv 2505.13397 v1 pith:QD35S7M4 submitted 2025-05-19 cs.LG cs.NAmath.NAstat.ML

Benoit Dherin , Michael Munn , Hanna Mazzawi , Michael Wunder , Sourabh Medapati , Javier Gonzalvo This is my paper · ORCID
classification cs.LGcs.NAmath.NAstat.ML
keywords Runge-KuttamethodsgradientflowODEsolversdeeplearningoptimizationmomentumpreconditioningadaptiverategeneralizationgap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Training a neural network by gradient descent is numerically solving the differential equation $\dot{\theta} = -\nabla L(\theta)$ with Euler's method, the simplest possible ODE solver. The paper argues that higher-order Runge-Kutta (RK) solvers, which track the gradient-flow trajectory far more closely, can be made into practical deep-learning optimizers if they are combined with ideas from modern optimizers. It first documents that vanilla RK4 is competitive on simple workloads but develops a generalization gap in the large-batch regime, because following gradient flow too precisely removes the implicit flatness bias that first-order discretization error provides. It then shows that preconditioning, an adaptive learning rate, and momentum each help, and that momentum applied to RK gradient estimates closes the gap, giving better test accuracy than Adam on MNIST and Fashion-MNIST. The evidence for this central comparison comes from multi-layer perceptron models on those two datasets, with per-dataset tuning of the added hyperparameters.

What carries the argument

The central object is the RK gradient estimate $g^*(\theta, h) = \sum_i b_i g(\theta_i)$, where the stage points $\theta_i = \theta - h \sum_j a_{ij} g(\theta_j)$ probe the gradient field nearby and the coefficients $a_{ij}, b_i$ are chosen so the update matches the Taylor expansion of the true gradient-flow solution to a prescribed order. The paper attaches three mechanisms to this object: a modified AdaGrad preconditioner $A'_n = (1 + \operatorname{diag}\, G_n)^{-1/2}$, a rescaled adaptive step size $h_{DALR}(\theta) = c / (1 + (c/2)(\|H g\| / \|g\|)^p)$, and momentum applied to the RK gradient itself. The momentum mechanism is the load-bearing one for the paper's strongest claim, since it fully closes the large-batch generalization gap that rules out vanilla RK4.

What would settle it

Run RK4-with-momentum and Adam with the same tuning budget on a large-scale workload such as an ImageNet-scale vision transformer or a large transformer language model, using large batches; if RK4-with-momentum fails to match Adam's test accuracy, or its per-step wall-clock cost becomes prohibitive, the paper's practical claim that these RK modifications benefit deep learning collapses. A cheaper check is to reproduce the full-batch MNIST result with the paper's reported hyperparameters and verify that the accuracy advantage over Adam persists across more than five seeds.

Watch

Extended reading notes

Core claim

The central discovery is that the value of higher-order ODE solvers for deep learning lies not in using them as-is but in grafting modern optimizer machinery onto their gradient estimates. A vanilla RK4 step evaluates the gradient at four nearby points and combines them, yielding an update that follows the exact gradient-flow solution to $\mathcal{O}(h^5)$ per step, but this precision is a double-edged sword: it stabilizes training while stripping away the regularizing effect of discretization error, producing the large-batch generalization gap. The paper's main result is that applying momentum to RK gradients, via $m_{n+1} = \beta m_n + g^*(\theta_n, h)$ and $\theta_{n+1} = \theta_n - h m_{n+1}$, bridges that gap and achieves better test accuracy than both Adam and vanilla RK4 on full-batch MNIST and Fashion-MNIST. It also establishes a preconditioning lemma showing that a positive-definite symmetric preconditioner keeps the loss decreasing along the modified gradient flow, and uses it to justify a modified AdaGrad preconditioner, along with a rescaled Drift-Adjusted Learning rate that each individually close part of the gap.

Load-bearing premise

The central practical claim rests on the assumption that the results observed on small MLP workloads with per-dataset tuning of the added hyperparameters transfer to the more complex workloads where such optimizers would matter; the paper explicitly leaves that transfer to future work.

Editorial extensions

If this is right

  • On simple workloads, vanilla RK4 can match or beat tuned baselines while exposing only a learning rate to tune and maintaining no gradient accumulators.
  • Momentum applied to RK gradients closes the large-batch generalization gap, making RK4-with-momentum more accurate than Adam on full-batch MNIST and Fashion-MNIST.
  • Preconditioning with a modified AdaGrad matrix and a rescaled Drift-Adjusted Learning rate each individually narrow the large-batch gap, though neither alone fully closes it.
  • Runge-Kutta optimizers require multiple gradient evaluations per step; their wall-clock time is comparable to Adam when all stage gradients fit in device memory, but roughly doubles when they do not.
  • The stability gained by following gradient flow more precisely comes at the cost of losing the implicit regularization induced by first-order discretization error, which explains the large-batch gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the large-batch gap is truly caused by missing implicit regularization, then pairing RK4 with explicit regularizers such as weight decay, label smoothing, or injected noise should extend its benefits beyond MNIST-scale tasks; this is not tested in the paper.
  • Editorial inference: the momentum-on-RK-gradients scheme can be read as a deterministic analogue of stochastic noise injection; a continuous-time analysis of the scheme could predict how $\beta$ should scale with curvature to maintain the generalization benefit.
  • Editorial inference: the CIFAR-10 ResNet-18 experiments with momentum and DALR hint that the modifications transfer beyond MLPs, but those runs used no augmentation, schedule, or weight decay, so they fall short of showing competitiveness in production-scale settings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the application of higher-order Runge-Kutta (RK) ODE solvers to deep learning, viewing neural network training as numerically solving the gradient flow ODE. It benchmarks vanilla RK4 against strong baselines on several workloads, identifies three limitations (wall-clock time, stiffness, and a large-batch generalization gap), and proposes three modifications to RK4: AdaGrad-like preconditioning, an adaptive learning rate (DALR), and momentum applied to RK gradients. The paper derives the order conditions for RK methods, proves a preconditioning lemma, and presents experiments on MNIST and Fashion-MNIST with MLPs (plus smaller CIFAR experiments) showing that the modifications bridge the generalization gap and, for the momentum version, claim better test accuracy than Adam. The conclusion explicitly acknowledges that the modifications were tested only on relatively simple workloads and that extending them to more complex settings is future work.

Significance. If its empirical claims are established, the paper makes a useful contribution by connecting a classical numerical-analysis framework to modern deep learning optimizers and by showing that simple modifications to vanilla RK4 can close the gap with Adam on small-scale full-batch tasks. The theoretical parts are solid: the order-condition derivation in Appendix A is correct, the preconditioning lemma (Lemma 4.1) is valid, and the paper's explicit statement of limitations in Section 5 is an honest and welcome feature. However, the central practical claim that the modified RK methods 'achieve better test accuracy than Adam' rests on experiments whose comparison control is not yet sufficient: Adam's hyperparameters are not tuned as thoroughly as those of the proposed methods, and no error bars or significance tests are reported for the key figures. These issues are fixable with additional experiments and re-analysis, so the paper is potentially valuable but not yet convincing on its main empirical claim.

major comments (3)
  1. [Section 4.3, Fig. 4, Appendix C.4] The claim that RK4 with momentum 'achieves better test accuracy than both Adam and vanilla RK4' is not supported by a controlled comparison. In Appendix C.4, Adam's learning rate is tuned while its beta parameters retain Optax defaults, but RK4+momentum has both its learning rate and momentum beta tuned per dataset (MNIST: 0.004, beta=0.95; Fashion-MNIST: 0.001, beta=0.95). Since the experiments are full-batch, Adam's default beta1=0.9 and beta2=0.999 may be poorly matched to deterministic gradients, and the reported gap could close or even reverse if Adam's betas were tuned per dataset. The authors should tune Adam's betas (or at least provide a sensitivity analysis over beta values) using the same search budget as for the proposed method.
  2. [Figures 2-4 and Appendix C.2-C.4] The learning curves in Figures 2-4 are presented without error bars or confidence intervals, and no significance test is reported for the final test-accuracy differences. With only 5 seeds and differences on the order of a few tenths of a percent, the claim that the modifications 'confer benefits' over Adam may be within seed noise. The authors should report mean plus/minus standard error (or confidence intervals) for the final test accuracy, and ideally perform a paired significance test across the shared seeds.
  3. [Section 3.1, Table 1] The claim that vanilla RK4 is 'competitive' with strong baselines is confounded by batch-size differences between the baseline and RK4 in several workloads. For example, Fashion-MNIST CNN uses baseline batch size 512 vs. RK4 batch size 64, CIFAR-10 uses baseline batch size 128 vs. RK4 batch size 512, and CIFAR-100 uses baseline batch size 128 vs. RK4 batch size 256. Because batch size affects both stochastic noise and the number of epochs per training step, the comparison is not purely an optimizer comparison. To support the competitiveness claim, the authors should match batch sizes for both optimizers, or at least report results with matched batch size as a sensitivity check.
minor comments (5)
  1. [Appendix C.2] The acronym 'ADGR' is introduced without definition; it is used in the captions of Figures 10 and 11 and in the text of Section 4.1, but the paper never spells out what it stands for.
  2. [References] References [29] and [30] are duplicates (both point to He et al., Deep residual learning for image recognition); this causes confusion in the caption of Figure 18, which cites the second copy.
  3. [Section 3.1] The phrase 'having only the learning rate train to tune' should read 'having only the learning rate to tune'.
  4. [Various captions] The word 'batchsize' appears in several figure captions and appendix headings; it should be written as 'batch size' for consistency with the rest of the text.
  5. [Section 5] The conclusion states that the modifications 'indeed improve upon vanilla RK4 and, in some cases, outperform Adam as well'; given the paper's own limitation statement that only simple workloads were tested, this claim should be explicitly scoped to the small-scale settings to avoid overgeneralization.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity found: the RK derivations are self-contained, and the empirical claims rest on disclosed experiments rather than on equations or self-citations that reduce to their own inputs.

full rationale

The paper's only derivation chain is the order-condition calculation for RK methods (Sec. 2, Appendix A): Eq. (5) expands a general RK step and Eq. (6) matches it to the Taylor expansion of the exact gradient-flow solution (Eq. (2)). That is standard, self-contained mathematics and does not assume the target claim. The preconditioning lemma (Lemma 4.1) is proven in the text from positive definiteness of A(theta), and the AdaGrad-like preconditioner (Eq. 12) is introduced as a practical choice, with its benefit verified empirically rather than derived from the lemma. The DALR learning rate (Eq. 15) is stated as a rescaling of the DAL idea and its two parameters (p, c) are tuned per dataset in Appendix C.3; no claim is made that the formula itself predicts the test-accuracy improvement. The momentum scheme (Eqs. 16-17) is a defined update rule; the statement that it 'achieves better test accuracy than both Adam and vanilla RK4' is an empirical result from Fig. 4 with explicitly reported hyperparameters in Appendix C.4, not a quantity forced by the update equations. The self-citations ([5], [13], [51], [52]) are motivational or attributive (natural metric, geometric complexity, DAL); the paper re-derives or states the needed formulas and does not invoke a uniqueness theorem or an unverified prior result to exclude alternatives. The skeptic's point about Adam's decay parameters being left at Optax defaults while RK4 gets per-dataset beta tuning is a comparison-fairness and external-validity concern, not circularity: no fitted parameter is renamed as a prediction, and no experimental output is set equal to an input by construction. Accordingly, no specific circular step can be exhibited under the required standard.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical entities. It relies on standard numerical analysis (RK theory), widely used continuous-time models of optimization, and a set of heuristic modifications whose parameters (DALR p, c; momentum beta; preconditioner constant) are tuned per workload. The main empirical claims therefore rest on the assumption that these simple workloads and tuned settings are representative.

free parameters (5)
  • DALR exponent p = 0.8 for MNIST and Fashion-MNIST
    Tuned per workload in Eq. (15); controls sensitivity of adaptive learning rate to gradient curvature.
  • DALR cap c = 4.0 for MNIST, 1.0 for Fashion-MNIST
    Tuned per workload; caps the adaptive learning rate and was introduced ad hoc in Eq. (15).
  • Momentum beta = 0.95 for MNIST and Fashion-MNIST; 0.99 for CIFAR-10
    Standard momentum hyperparameter, tuned per workload in Eq. (16).
  • Preconditioner constant 1 = 1 (replaces AdaGrad epsilon)
    The modified AdaGrad preconditioner A'_n = (1 + diag G_n)^(-1/2) uses 1 instead of epsilon; chosen because it 'worked better in practice'.
  • RK4 learning rate and batch size = Varies per workload (e.g., 0.003, batch 16 on MNIST DNN)
    The paper tunes only learning rate and batch size for RK4; these are the only free parameters in the vanilla RK4 benchmark.
assumptions (6)
  • standard math Standard Runge-Kutta order conditions determine the order of the method
    Used throughout Sec. 2 and Appendix A to construct RK2/RK3/RK4; standard numerical analysis result.
  • domain assumption Gradient flow ODE is the relevant continuous-time model for deep learning optimization
    The entire approach is predicated on treating neural network training as numerical integration of ˙theta = -∇L(θ); this is a common assumption in the literature.
  • domain assumption Higher-order ODE solvers that track gradient flow more closely yield more stable training
    Stated in Sec. 2 as a theoretical motivation; not proven for deep learning losses.
  • domain assumption The natural metric G(θ) = I + g g^T from [5] is relevant for preconditioning
    Used in Sec 4.1 and Appendix B to motivate the modified AdaGrad preconditioner; relies on cited work [5].
  • domain assumption Implicit regularization from discretization error or stochastic noise helps generalization
    Used to explain the large-batch generalization gap in Sec 3.2; based on cited works [9,23].
  • standard math The loss remains a Lyapunov function for the preconditioned gradient flow when A is positive definite
    Lemma 4.1 proof; this is a standard result from Lyapunov theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning by solving differential equations." pith.science (2026). https://pith.science/paper/QD35S7M4

@misc{pith2026250513397,
  author       = {Pith},
  title        = {Pith review of: Learning by solving differential equations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QD35S7M4}},
  note         = {Machine review of arXiv:2505.13397}
}
read the original abstract

Modern deep learning algorithms use variations of gradient descent as their main learning methods. Gradient descent can be understood as the simplest Ordinary Differential Equation (ODE) solver; namely, the Euler method applied to the gradient flow differential equation. Since Euler, many ODE solvers have been devised that follow the gradient flow equation more precisely and more stably. Runge-Kutta (RK) methods provide a family of very powerful explicit and implicit high-order ODE solvers. However, these higher-order solvers have not found wide application in deep learning so far. In this work, we evaluate the performance of higher-order RK solvers when applied in deep learning, study their limitations, and propose ways to overcome these drawbacks. In particular, we explore how to improve their performance by naturally incorporating key ingredients of modern neural network optimizers such as preconditioning, adaptive learning rates, and momentum.

Figures

Figures reproduced from arXiv: 2505.13397 by the authors.

Figure 1
Figure 1. For MNIST (top row) and Fashion MNIST (bottom row) trained on MLP, vanilla RK4 achieves better test accuracy compared to Adam for small batches (left column) while we observe a generalization gap for large batches (right column). See Appendix C.1 for experiment details. workloads. Most notably, these challenges are 1) the increase in compute time per step, 2) convergence difficulties in case of gradient field stiffn… view at source ↗
Figure 9
Figure 9. However, RK’s wall-clock time sharply increases if gradient data exceeds memory capacity. [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 2
Figure 2. RK4 with AdaGrad-like preconditioning helps bridge the gap between Adam and vanilla [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figures from the paper (16 more)
Figure 3
Figure 3. Figure 3: RK4 combined with an adaptive learning rate bridges the generalization gap in the large [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]
Figure 4
Figure 4. Figure 4: Combining momentum with RK4 bridges the generalization gap in the large batch regime [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: RK4 is competitive with Adam on MNIST with a 3 hidden layer MLP of 500 neurons [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: RK4 is competitive with Adam on Fashion MNIST with a 3 hidden layer MLP of 500 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: RK4 suffers from a generalization gap w.r.t. Adam on MNIST with a 3 hidden layer MLP [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: RK4 suffers from a generalization gap w.r.t. Adam on Fashion MNIST with a 3 hidden [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Wall-clock time versus steps between Adam and vanilla RK4 for MNIST ( [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: RK4 with preconditioning competes effectively with Adam on MNIST when using a [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: RK4 with preconditioning competes effectively with Adam on Fashion MNIST when [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: RK4 with adaptive learning rate (DALR) competes effectively with Adam on MNIST [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: RK4 with adaptive learning rate (DALR) competes effectively with Adam on Fashion [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: RK4 with momentum competes effectively with Adam on MNIST when using a 3-layer [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: RK4 with momentum competes effectively with Adam on Fashion MNIST when using a [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: The wall-clock time for Adam and RK4 on CIFAR-10 is essentially the same for small [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 17
Figure 17. Figure 17: The figure displays side-by-side training curves (left) versus test curves (right), illustrating [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: The figure displays side-by-side training curves for momentum (left) and adaptive learning [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 46 canonical work pages

  1. [1]

    Natural gradient works efficiently in learning.Neural computation, 10(2):251– 276, 1998

    Shun-ichi Amari. Natural gradient works efficiently in learning.Neural computation, 10(2):251– 276, 1998

    work page 1998

  2. [2]

    Scalable second order optimization for deep learning

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020

    arXiv 2002

  3. [3]

    Stochastic runge-kutta methods and adaptive sgd-g2 stochastic gradient descent

    Imen Ayadi and Gabriel Turinici. Stochastic runge-kutta methods and adaptive sgd-g2 stochastic gradient descent. In 2020 25th International Conference on Pattern Recognition (ICPR), 2021

    work page 2020

  4. [4]

    Liapunov Functions and Stability in Control Theory

    Andrea Bacciotti and Lionel Rosier. Liapunov Functions and Stability in Control Theory . Lecture Notes in Control and Information Sciences. Springer, London, 1 edition, 2001

    work page 2001

  5. [5]

    Barrett and Benoit Dherin

    David G.T. Barrett and Benoit Dherin. Implicit gradient regularization. In ICLR, 2021

    work page 2021

  6. [6]

    Modular duality in deep learning

    Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning. arXiv preprint arXiv:2410.21265, 2024

    arXiv 2024

  7. [7]

    On symplectic optimization

    Michael Betancourt, Michael I Jordan, and Ashia C Wilson. On symplectic optimization. arXiv preprint arXiv:1802.03653, 2018

    arXiv 2018

  8. [8]

    A. A. Brown and M. C. Bartholomew-Biggs. Some effective methods for unconstrained optimization based on the solution of systems of ordinary differential equations. J. Optim. Theory Appl., 62(2):211–224, August 1989

    work page 1989

Show all 68 references
  1. [9]

    On the implicit bias of adam

    Matias D Cattaneo, Jason M Klusowski, and Boris Shigida. On the implicit bias of adam. arXiv:2309.00079, 2023

    2023 arXiv

  2. [10]

    J. Cortés. Finite-time convergent gradient flows with applications to network consensus. Automatica, 42(11):1993–2000, 2006

    1993

  3. [11]

    Dahl, Frank Schneider, Peter Mattson, et al

    George E. Dahl, Frank Schneider, Peter Mattson, et al. Benchmarking neural network training algorithms. arXiv preprint arXiv:2306.07179, 2023

    2023 arXiv

  4. [12]

    The road less scheduled

    Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled. arXiv preprint arXiv:2405.15682, 2024. 10

    2024 arXiv

  5. [13]

    Why neural networks find simple solutions: The many regularizers of geometric complexity

    Benoit Dherin, Michael Munn, Mihaela Rosca, and David Barrett. Why neural networks find simple solutions: The many regularizers of geometric complexity. In NeurIPS, 2022

    2022

  6. [14]

    Corridor geometry in gradient-based optimization, 2024

    Benoit Dherin and Mihaela Rosca. Corridor geometry in gradient-based optimization, 2024

    2024

  7. [15]

    Adam: A method for stochastic optimization

    Kingma Diederik. Adam: A method for stochastic optimization. (No Title), 2014

    2014

  8. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  9. [17]

    Incorporating nesterov momentum into adam

    Timothy Dozat. Incorporating nesterov momentum into adam. In ICLR Workshop, 2016

    2016

  10. [18]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159, 2011

    2011

  11. [19]

    Towards hyperparameter-agnostic dnn training via dynamical system insights

    Carmel Fiscko, Aayushya Agarwal, Yihan Ruan, Soummya Kar, Larry Pileggi, and Bruno Sinopoli. Towards hyperparameter-agnostic dnn training via dynamical system insights. arXiv preprint arXiv:2310.13901, 2023

    2023 arXiv

  12. [20]

    Conformal symplectic and relativistic optimization

    Guilherme França, Jeremias Sulam, Daniel Robinson, and René Vidal. Conformal symplectic and relativistic optimization. In NeurIPS, 2020

    2020

  13. [21]

    Admm and accelerated admm as continuous dynamical systems

    Guilherme França, Daniel P Robinson, and René Vidal. Admm and accelerated admm as continuous dynamical systems. In International Conference on Machine Learning , pages 1554–1562. PMLR, 2018

    2018

  14. [22]

    Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization

    Guilherme França, Daniel P Robinson, and René Vidal. Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization. Physical Review E, 103(5), 2021

    2021

  15. [23]

    Implicit regularization in heavy-ball momentum accelerated stochastic gradient descent

    Avrajit Ghosh, He Lyu, Xitong Zhang, and Rongrong Wang. Implicit regularization in heavy-ball momentum accelerated stochastic gradient descent. ICLR, 2023

    2023

  16. [24]

    Gilmer, George E

    Justin M. Gilmer, George E. Dahl, Zachary Nado, Priya Kasimbeg, and Sourabh Medapati. init2winit: a jax codebase for initialization, optimization, and tuning research. github, 2023

    2023

  17. [25]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842–1850. PMLR, 2018

    2018

  18. [26]

    Geometric numerical integration

    Ernst Hairer, Marlis Hochbruck, Arieh Iserles, and Christian Lubich. Geometric numerical integration. Oberwolfach Reports, 3(1):805–882, 2006

    2006

  19. [27]

    Solving Ordinary Differential Equations I: Nonstiff Problems, volume 8 of Springer Series in Computational Mathematics

    Ernst Hairer, Syvert P Nørsett, and Gerhard Wanner. Solving Ordinary Differential Equations I: Nonstiff Problems, volume 8 of Springer Series in Computational Mathematics. Springer, 2 edition, 1993

    1993

  20. [28]

    Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems, volume 14 of Springer Series in Computational Mathematics

    Ernst Hairer and Gerhard Wanner. Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems, volume 14 of Springer Series in Computational Mathematics. Springer, Berlin, Heidelberg, 2 edition, 1996

    1996

  21. [30]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016

    2016

  22. [31]

    Optax: composable gradient transformation and optimisation, in jax!, 2020

    Matteo Hessel, David Budden, Fabio Viola, Mihaela Rosca, Eren Sezener, and Tom Hennigan. Optax: composable gradient transformation and optimisation, in jax!, 2020

    2020

  23. [32]

    Yang, Zachary Nado, Sourabh Medapati, Philipp Hennig, Michael Rabbat, and George E

    Priya Kasimbeg, Frank Schneider, Runa Eschenhagen, Juhan Bae, Chandramouli Shama Sastry, Mark Saroufim, BOYUAN FENG, Less Wright, Edward Z. Yang, Zachary Nado, Sourabh Medapati, Philipp Hennig, Michael Rabbat, and George E. Dahl. Accelerating neural network training: An analys...

    2025

  24. [33]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015

    2015

  25. [34]

    Continuous time analysis of momentum methods

    Nikola B Kovachki and Andrew M Stuart. Continuous time analysis of momentum methods. Journal of Machine Learning Research, 22(17):1–40, 2021

    2021

  26. [35]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. https://www.cs.toronto.edu/ kriz/learning-features-2009-TR.pdf, 2009

    2009

  27. [36]

    John M. Lee. Introduction to Smooth Manifolds, volume 218 of Graduate Texts in Mathematics. Springer, 2nd edition, 2012

    2012

  28. [37]

    Visualizing the loss landscape of neural nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In NeurIPS, 2018

    2018

  29. [38]

    Implicit stochastic gradient descent for training physics-informed neural networks

    Ye Li, Song-Can Chen, and Sheng-Jun Huang. Implicit stochastic gradient descent for training physics-informed neural networks. arXiv preprint arXiv:2303.01767, 2023

    2023 arXiv

  30. [39]

    Understanding the difficulty of training transformers

    Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), April 2020

    2020

  31. [40]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  32. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019

    2019

  33. [42]

    Aggregated momentum: Stability through passive damping

    James Lucas, Shengyang Sun, Richard Zemel, and Roger Grosse. Aggregated momentum: Stability through passive damping. In International Conference on Learning Representations, 2019

    2019

  34. [43]

    Optimizing neural networks with kronecker-factored approx- imate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approx- imate curvature. In International conference on machine learning, pages 2408–2417. PMLR, 2015

    2015

  35. [44]

    An empirical model of large-batch training

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018

    2018 arXiv

  36. [45]

    A dynamical systems perspective on nesterov acceleration

    Michael Muehlebach and Michael I Jordan. A dynamical systems perspective on nesterov acceleration. In International Conference on Machine Learning, pages 4656–4662. PMLR, 2019

    2019

  37. [46]

    Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution

    Antonia Orvieto, Simon Lacoste-Julien, and Nicolas Loizou. Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. In NeurIPS, 2022

    2022

  38. [47]

    An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes

    Antonio Orvieto and Lin Xiao. An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes. arXiv preprint arXiv:2407.04358, 2024

    2024 arXiv

  39. [48]

    Boris T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964

    1964

  40. [49]

    On the curvature of the loss landscape

    Alison Pouplin, Hrittik Roy, Sidak Pal Singh, and Georgios Arvanitidis. On the curvature of the loss landscape. arXiv preprint arXiv:2307.04719, 2023

    2023 arXiv

  41. [50]

    Training generative adversarial networks by solving ordinary differential equations

    Chongli Qin, Yan Wu, Jost Tobias Springenberg, Andy Brock, Jeff Donahue, Timothy Lillicrap, and Pushmeet Kohli. Training generative adversarial networks by solving ordinary differential equations. In Advances in Neural Information Processing Systems, 2020

    2020

  42. [51]

    Mihaela Rosca, Yan Wu, Benoit Dherin, and David G.T. Barrett. Discretization drift in two- player games. In ICML, 2021

    2021

  43. [52]

    On a continuous time model of gradient descent dynamics and instability in deep learning

    Mihaela Rosca, Yan Wu, Chongli Qin, and Benoit Dherin. On a continuous time model of gradient descent dynamics and instability in deep learning. In TMLR, 2023. 12

    2023

  44. [53]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323:533–536, 1986

    1986

  45. [54]

    Acceleration via symplectic discretiza- tion of high-resolution differential equations

    Bin Shi, Simon S Du, Weijie J Su, and Michael I Jordan. Acceleration via symplectic discretiza- tion of high-resolution differential equations. In Advances in Neural Information Processing Systems, pages 5744–5752, 2019

    2019

  46. [55]

    A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale

    Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, and Michael Rabbat. A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale. a...

    2023 arXiv

  47. [56]

    Improving optimizers by runge-kutta method: A case study of sgd and adam

    Dan Su, Qihai Jiang, Enhong Liu, and Mei Liu. Improving optimizers by runge-kutta method: A case study of sgd and adam. In 2024 12th International Conference on Intelligent Control and Information Processing (ICICIP), 2024

    2024

  48. [57]

    A differential equation for modeling nes- terov’s accelerated gradient method: theory and insights.Journal of Machine Learning Research, 17:1–43, 2016

    Weijie Su, Stephen Boyd, and Emmanuel Candès. A differential equation for modeling nes- terov’s accelerated gradient method: theory and insights.Journal of Machine Learning Research, 17:1–43, 2016

    2016

  49. [58]

    On the importance of initial- ization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, volume 28, pages 1139–1147. PMLR, 2013

    2013

  50. [59]

    Spike no more: Stabilizing the pre-training of large language models, 2025

    Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. Spike no more: Stabilizing the pre-training of large language models, 2025

    2025

  51. [60]

    Rmsprop: Divide the gradient by a running average of its recent magnitude

    Tijmen Tieleman and Geoffrey Hinton. Rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA Neural Networks for Machine Learning, 2012

    2012

  52. [61]

    Understanding and mitigating gradient flow pathologies in physics-informed neural networks

    Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing , 43(5):A3055–A3081, 2021

    2021

  53. [62]

    Small-scale proxies for large-scale transformer training instabilities

    Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies fo...

    2024

  54. [63]

    Structured pre- conditioners in adaptive optimization: A unified analysis

    Shuo Xie, Tianhao Wang, Sashank Reddi, Sanjiv Kumar, and Zhiyuan Li. Structured pre- conditioners in adaptive optimization: A unified analysis. arXiv preprint arXiv:2503.10537, 2025

    2025 arXiv

  55. [64]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016

    2016 arXiv

  56. [65]

    Matthew D. Zeiler. Adadelta: An adaptive learning rate method.arXiv preprint arXiv:1212.5701, 2012

    2012 arXiv

  57. [66]

    Direct runge-kutta discretiza- tion achieves acceleration

    Jingzhao Zhang, Aryan Mokhtari, Suvrit Sra, and Ali Jadbabaie. Direct runge-kutta discretiza- tion achieves acceleration. In Advances in Neural Information Processing Systems, 2018

    2018

  58. [67]

    Lookahead optimizer: k steps forward, 1 step back

    Michael R Zhang, James Lucas, Geoffrey Hinton, and Jimmy Ba. Lookahead optimizer: k steps forward, 1 step back. In Advances in neural information processing systems, pages 9591–9601, 2019. 13 A Runge-Kutta updates This section presents the formulas for Runge-Kutta (RK) updates...

    2019

  59. [68]

    In the main paper, we benchmarked RK4, which is the classical 4th order method, and it has an error of sizeO(h5)

    to illustrate the general approach for finding RK coefficients. In the main paper, we benchmarked RK4, which is the classical 4th order method, and it has an error of sizeO(h5). Recall that a RK method is given by a matrixA = (aij) and a vectorb = (bi). The corresponding RK up...

  60. [69]

    Therefore, the correct way to write the gradient flow ODE is with the help of an underlying metric tensorG(θ) on the parameter space given by ˙θ(t) = G(θ(t))−1∇L(θ(t))

    for more details):µ→G−1µ. Therefore, the correct way to write the gradient flow ODE is with the help of an underlying metric tensorG(θ) on the parameter space given by ˙θ(t) = G(θ(t))−1∇L(θ(t)). (44) In this form, the gradient flow ODE has the exact same form in every coordina...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.