{"id":"47834966-f036-4ef5-baf8-c7eed603f321","arxiv_id":"2505.13397","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Runge-Kutta optimizers adapted with momentum, preconditioning, or adaptive learning rates can close the large-batch generalization gap and match Adam on small MLP workloads.","lead":"This paper tests Runge-Kutta, a family of numerical ODE solvers, as optimizers for neural networks, and finds they can match or beat Adam on simple tasks when combined with momentum, preconditioning, or adaptive learning rates. The results are limited to small workloads, and the paper identifies where vanilla Runge-Kutta fails, such as on large batches and complex models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fig 4's 'better than Adam' claim rests on unequal tuning: Adam only gets LR tuning with default betas, while RK4+momentum gets LR and beta tuned per dataset; without matched Adam tuning or error bars, the central empirical claim is not established.","rationale":"The reader's weakest assumption is that the small-scale MNIST/Fashion-MNIST results are representative enough to support a general practical claim. I agree that transferability is a limitation, but there is a more proximal, testable problem: the central 'better than Adam' comparison is not controlled. Adam is evaluated with default decay hyperparameters and only a tuned learning rate, while the proposed RK4+momentum method receives per-dataset tuning of both learning rate and momentum beta. Since the claimed advantage is the core of the strongest claim and the supporting figure lacks error bars and statistical testing, this is the least secure load-bearing point. I still find the paper honest and the ODE background sound; the issue is empirical comparison control, not internal inconsistency. The reader's CONDITIONAL verdict remains appropriate, so I recommend UNCHANGED, with a request for matched tuning, error bars, and ideally a more complex workload before the practical claim can be accepted at higher confidence.","tokens_in":18631,"tokens_out":4152,"duration_ms":41723,"concrete_test":"Re-run the Fig. 4 large-batch protocol with a matched hyperparameter search for Adam: tune beta1 in {0.0, 0.9, 0.95, 0.99}, beta2 in {0.99, 0.999, 0.9999}, and learning rate in {1e-4, 3e-4, 1e-3, 3e-3} on each dataset with the same 5 seeds, and report mean plus/minus standard error of best test accuracy. If tuned Adam is within one standard error of RK4+momentum on both datasets, the 'better than Adam' claim is not supported; if RK4+momentum remains ahead by more than two standard errors, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing empirical claim (Sec. 4.3, Fig. 4) is that momentum-RK4 'achieves better test accuracy than both Adam and vanilla RK4' on MNIST and Fashion-MNIST in the large-batch regime. The comparison is not controlled: Appendix C.4 states that for the baseline, Adam's learning rate was tuned (0.001) while decay parameters were set to Optax defaults, whereas RK4+momentum had both its learning rate and momentum beta tuned separately per dataset (MNIST: 0.004, beta=0.95; Fashion-MNIST: 0.001, beta=0.95). In full-batch training, Adam's default beta1=0.9 and beta2=0.999 are not obviously optimal; they may be poorly matched to deterministic gradients, and a per-dataset tuning of Adam's betas could close or even reverse the reported gap. Additionally, Fig. 4 is presented as learning curves without reported error bars or a significance test, so the 5-seed differences may be within seed noise. If the advantage disappears under matched hyperparameter tuning, the central practical claim reduces to a demonstration that a hand-tuned variant can match Adam on two small MLP workloads. The paper's own Sec. 5 limitation statement concedes that the modifications were tested only on simple workloads and that transfer to more complex workloads is future work, so this comparison-control issue is the main gating factor for the paper's central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the application of higher-order Runge-Kutta (RK) ODE solvers to deep learning, viewing neural network training as numerically solving the gradient flow ODE. It benchmarks vanilla RK4 against strong baselines on several workloads, identifies three limitations (wall-clock time, stiffness, and a large-batch generalization gap), and proposes three modifications to RK4: AdaGrad-like preconditioning, an adaptive learning rate (DALR), and momentum applied to RK gradients. The paper derives the order conditions for RK methods, proves a preconditioning lemma, and presents experiments on MNIST and Fashion-MNIST with MLPs (plus smaller CIFAR experiments) showing that the modifications bridge the generalization gap and, for the momentum version, claim better test accuracy than Adam. The conclusion explicitly acknowledges that the modifications were tested only on relatively simple workloads and that extending them to more complex settings is future work.","tokens_in":18907,"tokens_out":3641,"duration_ms":34761,"significance":"If its empirical claims are established, the paper makes a useful contribution by connecting a classical numerical-analysis framework to modern deep learning optimizers and by showing that simple modifications to vanilla RK4 can close the gap with Adam on small-scale full-batch tasks. The theoretical parts are solid: the order-condition derivation in Appendix A is correct, the preconditioning lemma (Lemma 4.1) is valid, and the paper's explicit statement of limitations in Section 5 is an honest and welcome feature. However, the central practical claim that the modified RK methods 'achieve better test accuracy than Adam' rests on experiments whose comparison control is not yet sufficient: Adam's hyperparameters are not tuned as thoroughly as those of the proposed methods, and no error bars or significance tests are reported for the key figures. These issues are fixable with additional experiments and re-analysis, so the paper is potentially valuable but not yet convincing on its main empirical claim.","major_comments":[{"comment":"The claim that RK4 with momentum 'achieves better test accuracy than both Adam and vanilla RK4' is not supported by a controlled comparison. In Appendix C.4, Adam's learning rate is tuned while its beta parameters retain Optax defaults, but RK4+momentum has both its learning rate and momentum beta tuned per dataset (MNIST: 0.004, beta=0.95; Fashion-MNIST: 0.001, beta=0.95). Since the experiments are full-batch, Adam's default beta1=0.9 and beta2=0.999 may be poorly matched to deterministic gradients, and the reported gap could close or even reverse if Adam's betas were tuned per dataset. The authors should tune Adam's betas (or at least provide a sensitivity analysis over beta values) using the same search budget as for the proposed method.","section":"Section 4.3, Fig. 4, Appendix C.4"},{"comment":"The learning curves in Figures 2-4 are presented without error bars or confidence intervals, and no significance test is reported for the final test-accuracy differences. With only 5 seeds and differences on the order of a few tenths of a percent, the claim that the modifications 'confer benefits' over Adam may be within seed noise. The authors should report mean plus/minus standard error (or confidence intervals) for the final test accuracy, and ideally perform a paired significance test across the shared seeds.","section":"Figures 2-4 and Appendix C.2-C.4"},{"comment":"The claim that vanilla RK4 is 'competitive' with strong baselines is confounded by batch-size differences between the baseline and RK4 in several workloads. For example, Fashion-MNIST CNN uses baseline batch size 512 vs. RK4 batch size 64, CIFAR-10 uses baseline batch size 128 vs. RK4 batch size 512, and CIFAR-100 uses baseline batch size 128 vs. RK4 batch size 256. Because batch size affects both stochastic noise and the number of epochs per training step, the comparison is not purely an optimizer comparison. To support the competitiveness claim, the authors should match batch sizes for both optimizers, or at least report results with matched batch size as a sensitivity check.","section":"Section 3.1, Table 1"}],"minor_comments":[{"comment":"The acronym 'ADGR' is introduced without definition; it is used in the captions of Figures 10 and 11 and in the text of Section 4.1, but the paper never spells out what it stands for.","section":"Appendix C.2"},{"comment":"References [29] and [30] are duplicates (both point to He et al., Deep residual learning for image recognition); this causes confusion in the caption of Figure 18, which cites the second copy.","section":"References"},{"comment":"The phrase 'having only the learning rate train to tune' should read 'having only the learning rate to tune'.","section":"Section 3.1"},{"comment":"The word 'batchsize' appears in several figure captions and appendix headings; it should be written as 'batch size' for consistency with the rest of the text.","section":"Various captions"},{"comment":"The conclusion states that the modifications 'indeed improve upon vanilla RK4 and, in some cases, outperform Adam as well'; given the paper's own limitation statement that only simple workloads were tested, this claim should be explicitly scoped to the small-scale settings to avoid overgeneralization.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives the community something real: the first broad benchmark of vanilla RK4 against strong modern baselines (Adam, Momentum, NAdamW), and it is admirably honest about where RK4 fails. The benchmark table and the discussion of wall-clock time and the large-batch generalization gap are the most valuable parts. The mathematical background is correct, and the three proposed modifications (preconditioning, adaptive learning rate, momentum) are clearly motivated and grounded in the ODE view of optimizers. The appendix documents the setups in enough detail to reproduce the main experiments, except that no code is provided.\n\nThe soft spots are in the empirical claims, not the theory. The stress-test concern about Fig. 4 is on point: the momentum-RK4 result is compared against an Adam baseline that only got its learning rate tuned, while RK4+momentum had both learning rate and beta tuned per dataset. That is not a controlled comparison. With full-batch training, Adam's default betas are not obviously optimal, and a matched tuning of Adam could easily close or reverse the reported gap. The learning curves in Figs. 2–4 have no error bars or significance tests, so the 5-seed differences may be noise. The paper's own conclusion concedes the modifications were tested only on simple MLP workloads; the CIFAR-10 results in the appendix are a step in the right direction but still limited and without error bars.\n\nNone of this makes the paper a waste of time. The central conceptual claim—that higher-order RK can be made practical with the right modifications—is plausible and worth investigating. But the strongest advertised result, \"better than Adam,\" is not established by the evidence as presented. The honest framing and the negative results (vanilla RK4's poor performance on ImageNet, the memory cost) are genuinely useful for anyone thinking about ODE-based optimizers.\n\nWho will get value from this? Researchers working on optimizer design, especially those interested in the connection between numerical analysis and deep learning. It deserves a serious referee, but the authors should be pushed to share code, add error bars, and redo the Adam comparison with matched tuning. I would not cite the headline claim as established, but I would cite the benchmark and the negative results.","headline":"A candid, useful empirical study of RK4 for deep learning with honest limitations, but its central 'better than Adam' claim is under-supported by unmatched tuning and missing error bars.","tokens_in":19487,"tokens_out":2085,"would_cite":true,"duration_ms":20978,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Higher-order Runge-Kutta optimizers for deep learning become practical when momentum is applied to their gradient estimates, closing the large-batch generalization gap and beating Adam on MNIST and Fashion-MNIST.","keywords":["Runge-Kutta methods","gradient flow","ODE solvers","deep learning optimization","momentum","preconditioning","adaptive learning rate","generalization gap"],"falsifier":"Run RK4-with-momentum and Adam with the same tuning budget on a large-scale workload such as an ImageNet-scale vision transformer or a large transformer language model, using large batches; if RK4-with-momentum fails to match Adam's test accuracy, or its per-step wall-clock cost becomes prohibitive, the paper's practical claim that these RK modifications benefit deep learning collapses. A cheaper check is to reproduce the full-batch MNIST result with the paper's reported hyperparameters and verify that the accuracy advantage over Adam persists across more than five seeds.","tokens_in":18414,"feed_emoji":"⚙️","tokens_out":11281,"duration_ms":90361,"temperature":0.7,"pith_summary":"Training a neural network by gradient descent is numerically solving the differential equation $\\dot{\\theta} = -\\nabla L(\\theta)$ with Euler's method, the simplest possible ODE solver. The paper argues that higher-order Runge-Kutta (RK) solvers, which track the gradient-flow trajectory far more closely, can be made into practical deep-learning optimizers if they are combined with ideas from modern optimizers. It first documents that vanilla RK4 is competitive on simple workloads but develops a generalization gap in the large-batch regime, because following gradient flow too precisely removes the implicit flatness bias that first-order discretization error provides. It then shows that preconditioning, an adaptive learning rate, and momentum each help, and that momentum applied to RK gradient estimates closes the gap, giving better test accuracy than Adam on MNIST and Fashion-MNIST. The evidence for this central comparison comes from multi-layer perceptron models on those two datasets, with per-dataset tuning of the added hyperparameters.","feed_headline":"RK4 with momentum beats Adam on MNIST and Fashion-MNIST","feed_subtitle":"A momentum term on Runge-Kutta gradient estimates closes the large-batch gap that held RK optimizers back.","key_machinery":"The central object is the RK gradient estimate $g^*(\\theta, h) = \\sum_i b_i g(\\theta_i)$, where the stage points $\\theta_i = \\theta - h \\sum_j a_{ij} g(\\theta_j)$ probe the gradient field nearby and the coefficients $a_{ij}, b_i$ are chosen so the update matches the Taylor expansion of the true gradient-flow solution to a prescribed order. The paper attaches three mechanisms to this object: a modified AdaGrad preconditioner $A'_n = (1 + \\operatorname{diag}\\, G_n)^{-1/2}$, a rescaled adaptive step size $h_{DALR}(\\theta) = c / (1 + (c/2)(\\|H g\\| / \\|g\\|)^p)$, and momentum applied to the RK gradient itself. The momentum mechanism is the load-bearing one for the paper's strongest claim, since it fully closes the large-batch generalization gap that rules out vanilla RK4.","core_discovery":"The central discovery is that the value of higher-order ODE solvers for deep learning lies not in using them as-is but in grafting modern optimizer machinery onto their gradient estimates. A vanilla RK4 step evaluates the gradient at four nearby points and combines them, yielding an update that follows the exact gradient-flow solution to $\\mathcal{O}(h^5)$ per step, but this precision is a double-edged sword: it stabilizes training while stripping away the regularizing effect of discretization error, producing the large-batch generalization gap. The paper's main result is that applying momentum to RK gradients, via $m_{n+1} = \\beta m_n + g^*(\\theta_n, h)$ and $\\theta_{n+1} = \\theta_n - h m_{n+1}$, bridges that gap and achieves better test accuracy than both Adam and vanilla RK4 on full-batch MNIST and Fashion-MNIST. It also establishes a preconditioning lemma showing that a positive-definite symmetric preconditioner keeps the loss decreasing along the modified gradient flow, and uses it to justify a modified AdaGrad preconditioner, along with a rescaled Drift-Adjusted Learning rate that each individually close part of the gap.","pith_inferences":["Editorial inference: if the large-batch gap is truly caused by missing implicit regularization, then pairing RK4 with explicit regularizers such as weight decay, label smoothing, or injected noise should extend its benefits beyond MNIST-scale tasks; this is not tested in the paper.","Editorial inference: the momentum-on-RK-gradients scheme can be read as a deterministic analogue of stochastic noise injection; a continuous-time analysis of the scheme could predict how $\\beta$ should scale with curvature to maintain the generalization benefit.","Editorial inference: the CIFAR-10 ResNet-18 experiments with momentum and DALR hint that the modifications transfer beyond MLPs, but those runs used no augmentation, schedule, or weight decay, so they fall short of showing competitiveness in production-scale settings."],"forward_implications":["On simple workloads, vanilla RK4 can match or beat tuned baselines while exposing only a learning rate to tune and maintaining no gradient accumulators.","Momentum applied to RK gradients closes the large-batch generalization gap, making RK4-with-momentum more accurate than Adam on full-batch MNIST and Fashion-MNIST.","Preconditioning with a modified AdaGrad matrix and a rescaled Drift-Adjusted Learning rate each individually narrow the large-batch gap, though neither alone fully closes it.","Runge-Kutta optimizers require multiple gradient evaluations per step; their wall-clock time is comparable to Adam when all stage gradients fit in device memory, but roughly doubles when they do not.","The stability gained by following gradient flow more precisely comes at the cost of losing the implicit regularization induced by first-order discretization error, which explains the large-batch gap."],"supporting_citations":[{"why":"Supplies the order-condition theory that defines RK methods and the RK4 update used throughout.","marker":"[27]"},{"why":"Provides the stiffness and adaptive-step-size background that motivates the adaptive learning-rate modification.","marker":"[28]"},{"why":"Defines Adam, the principal baseline that RK4 variations are compared against.","marker":"[33]"},{"why":"Defines AdaGrad, whose diagonal preconditioner the paper modifies for RK updates.","marker":"[18]"},{"why":"Explains the implicit bias of Adam and the generalization behavior used to diagnose RK4's large-batch gap.","marker":"[9]"},{"why":"Introduces implicit gradient regularization from discretization error, the mechanism whose absence creates the gap.","marker":"[5]"},{"why":"Introduces the Drift-Adjusted Learning rate that the paper rescales for use with RK methods.","marker":"[52]"},{"why":"Provides the classical momentum recurrence the paper adapts to average RK gradients.","marker":"[58]"},{"why":"Shows that direct Runge-Kutta discretization can produce accelerated optimization, supporting the premise that RK solvers are relevant.","marker":"[66]"},{"why":"Provides a prior benchmark of RK4 on deep-learning workloads that this paper extends by comparing against modern baselines.","marker":"[56]"}],"fun_headline_variants":["Momentum closes RK4's generalization gap, beats Adam","Runge-Kutta plus momentum outperforms Adam on MNIST","Grafting momentum onto RK4 solves large-batch gap","RK4 with momentum: a new optimizer that beats Adam","High-order ODE solvers get a boost from momentum"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central practical claim rests on the assumption that the results observed on small MLP workloads with per-dataset tuning of the added hyperparameters transfer to the more complex workloads where such optimizers would matter; the paper explicitly leaves that transfer to future work.","fun_headline_variants_meta":{"raw":{"variants":["Momentum closes RK4's generalization gap, beats Adam","Runge-Kutta plus momentum outperforms Adam on MNIST","Grafting momentum onto RK4 solves large-batch gap","RK4 with momentum: a new optimizer that beats Adam","High-order ODE solvers get a boost from momentum"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00046,"raw_usage":{"total_tokens":2297,"prompt_tokens":929,"completion_tokens":1368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1284}},"tokens_in":545,"tokens_out":1368,"duration_ms":10337,"temperature":1.0,"reasoning_tokens":1284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:14:13.364306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RK4-with-momentum and Adam with the same tuning budget on a large-scale workload such as an ImageNet-scale vision transformer or a large transformer language model, using large batches; if RK4-with-momentum fails to match Adam's test accuracy, or its per-step wall-clock cost becomes prohibitive, the paper's practical claim that these RK modifications benefit deep learning collapses. A cheaper check is to reproduce the full-batch MNIST result with the paper's reported hyperparameters and verify that the accuracy advantage over Adam persists across more than five seeds.","supporting_citations":[{"cited_title":"Solving Ordinary Differential Equations I: Nonstiff Problems, volume 8 of Springer Series in Computational Mathematics","cited_arxiv_id":null,"evidence_quote":"Supplies the order-condition theory that defines RK methods and the RK4 update used throughout."},{"cited_title":"Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems, volume 14 of Springer Series in Computational Mathematics","cited_arxiv_id":null,"evidence_quote":"Provides the stiffness and adaptive-step-size background that motivates the adaptive learning-rate modification."},{"cited_title":"Adaptive subgradient methods for online learning and stochastic optimization","cited_arxiv_id":null,"evidence_quote":"Defines AdaGrad, whose diagonal preconditioner the paper modifies for RK updates."},{"cited_title":"Barrett and Benoit Dherin","cited_arxiv_id":null,"evidence_quote":"Introduces implicit gradient regularization from discretization error, the mechanism whose absence creates the gap."},{"cited_title":"On a continuous time model of gradient descent dynamics and instability in deep learning","cited_arxiv_id":null,"evidence_quote":"Introduces the Drift-Adjusted Learning rate that the paper rescales for use with RK methods."},{"cited_title":"On the importance of initial- ization and momentum in deep learning","cited_arxiv_id":null,"evidence_quote":"Provides the classical momentum recurrence the paper adapts to average RK gradients."},{"cited_title":"Direct runge-kutta discretiza- tion achieves acceleration","cited_arxiv_id":null,"evidence_quote":"Shows that direct Runge-Kutta discretization can produce accelerated optimization, supporting the premise that RK solvers are relevant."},{"cited_title":"Improving optimizers by runge-kutta method: A case study of sgd and adam","cited_arxiv_id":null,"evidence_quote":"Provides a prior benchmark of RK4 on deep-learning workloads that this paper extends by comparing against modern baselines."}],"review_version":1}