Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks

Blake Woodworth; Pierfrancesco Beneventano

REVIEW 3 major objections 7 minor 26 references

Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gradient descent on the simplest self-multiplying network converges linearly to a global minimum even at stepsizes about twice the classical limit, and provably selects a flatter minimizer than gradient flow, because discrete steps shrink…

desk verdict The paper's core idea is right and worth engaging, but a false step in the region-C argument means the explicit-rate theorem is not proved as written. read the letter →

arxiv 2501.09137 v2 pith:AIMT6Q3B submitted 2025-01-15 cs.LG math.OCstat.ML

Pierfrancesco Beneventano , Blake Woodworth This is my paper · ORCID

classification cs.LGmath.OCstat.ML MSC 68T0790C2690C3065K05

keywords gradientdescentimplicitregularizationedgeofstabilitylinearconvergencedepth-2networksharpnessflownon-convexoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies the simplest possible self-multiplying model of a neural network — a depth-2 linear net with one scalar input and one scalar output, whose training loss is $L(a,b)=\tfrac{1}{2}(a^\top b-\Phi)^2$ — and asks how gradient descent (GD) behaves on it. The authors prove that GD converges at an explicit linear rate to a global minimum for stepsizes up to about $2/\lambda$, twice the classical stability threshold, and that it still converges, though logarithmically slowly, for even larger stepsizes. They also characterize which global minimum GD selects: each coordinate imbalance $a_i^2-b_i^2$ shrinks under the discrete update while being conserved by gradient flow, so GD's limit has strictly lower norm and lower Hessian sharpness than the flow's. The paper's central trade-off is that the same imbalance that gets regularized sets the convergence rate, so stronger implicit regularization costs training speed — which reframes edge-of-stability training, where real networks' sharpness hovers near $2/\eta$, as a regime where flatness is bought by delayed convergence.

What carries the argument

The central machinery is a three-scalar reparametrization of the $2d$-dimensional dynamics: the residual $\varepsilon=a^\top b-\Phi$, the scale $\lambda=\|a\|^2+\|b\|^2$ (the Hessian's top eigenvalue at a minimizer), and the imbalance $Q=\sum_i|a_i^2-b_i^2|$, together with the exact updates $\varepsilon(t+1)=\varepsilon(t)\,[1-\eta\lambda(t)+\eta^2\varepsilon(t)(\varepsilon(t)+\Phi)]$, $\lambda(t+1)=(1+\eta^2\varepsilon(t)^2)\lambda(t)-4\eta\varepsilon(t)(\varepsilon(t)+\Phi)$, and $Q_i(t+1)=(1-\eta^2\varepsilon(t)^2)Q_i(t)$. The load-bearing identity is the paper's Lemma 2: the quantity $\alpha=\lambda^2-8\varepsilon(\varepsilon+\Phi)+4\varepsilon^2$ is conserved along gradient flow and strictly reduced by every GD step by a positive term of order $\eta^2\varepsilon^2Q^2$, which simultaneously bounds $\lambda$ along the trajectory and drives the imbalance down. The convergence proof then splits the state space into three regions ($\varepsilon>0$; $\varepsilon<0<a^\top b$; $a^\top b<0$) and establishes a trajectory-wise Polyak–Łojasiewicz inequality — the bound $\|\nabla L\|^2\ge \mu L$ with a locally positive constant — whose $\mu$ is controlled below by $Q(\tau)$, where $\tau$ is the first step with $a^\top b>0$; because $\varepsilon$ and $Q$ both evolve multiplicatively, $Q(\tau)$ cannot vanish, and that positive constant is what converts the near-contraction of $\varepsilon$ into the explicit linear rate.

What would settle it

Simulate the scalar dynamics with, say, $\Phi=1$, $d=2$, an initialization with $Q(0)\neq 0$, and stepsize $\eta=2/\sqrt{\lambda(0)^2+4\Phi^2}$; Theorem 2 predicts linear residual contraction whose rate is governed by $Q$ at the first crossing of $a^\top b=0$, with the iteration count to reach $L\le\delta$ bounded by the displayed $O(\cdot)$ expression — a materially faster or slower measured decay would break the claimed rate. A sharper check targets the slow regime: for $\eta$ just above $2/\lambda(0)$ the paper predicts only logarithmic decay, of the form of the ODE $\dot x=-x^2$ (residual decaying like $1/t$); observing exponential decay there would falsify the slow-convergence claim. Both checks are direct numerical experiments on the closed-form updates (6)–(8).

Watch

Extended reading notes

Core claim

On the objective $L(a,b)=\tfrac{1}{2}(a^\top b-\Phi)^2$ with $\Phi\ge 0$, the paper claims that gradient descent with stepsize in the range $0<\eta<\min\{1/(2|\varepsilon(0)|),\,2/\sqrt{\lambda(0)^2+4\Phi^2}\}$ converges to a global minimum at a linear rate whose dependence on $\eta$, the initialization, and $\Phi$ is explicit, and that for larger stepsizes it still converges, but only logarithmically slowly, at a rate comparable to the ODE $\dot x=-x^2$, so residuals decay like $1/t$ rather than exponentially. The selection claim is that each imbalance $Q_i=a_i^2-b_i^2$ evolves as $Q_i(t+1)=(1-\eta^2\varepsilon(t)^2)Q_i(t)$, so the limit satisfies $|Q_i(0)|\exp(-\sqrt{\eta}\,\varepsilon(0)^2/\Phi)<|Q_i(\infty)|<|Q_i(0)|\exp(-\eta^2\sum_t\varepsilon(t)^2)$, strictly below the gradient-flow limit in which every $Q_i$ is conserved. Since the sharpness at a global minimum equals $\lambda=\|a\|^2+\|b\|^2$, minimized at $\lambda=2\Phi$ when the layers are balanced, GD provably terminates at a flatter minimum than gradient flow, larger stepsizes flatten it further, and the imbalance never fully vanishes except on a measure-zero set of initializations.

Load-bearing premise

The argument depends on the scalar-input reduction of the paper's Lemma 3: only for univariate inputs does the whole empirical risk collapse into the single square $(a^\top b-\Phi)^2$ whose residual $\varepsilon$ and imbalance $Q$ the proofs track as scalars; for vector-valued inputs the loss becomes a sum of coupled residuals and the single-scalar analysis no longer applies, a case the paper explicitly leaves open.

Editorial extensions

If this is right

On this model, GD converges linearly to a global minimum at stepsizes up to about $2/\lambda$, twice the classical threshold, with a rate explicit in $\eta$, the initialization, and $\Phi$ — a property prior work on the same objective established only without a rate.
GD terminates at a strictly flatter minimizer than gradient flow: the imbalance $Q$, the norm $\lambda$, and the sharpness are all strictly smaller, and larger stepsizes give strictly more flattening.
Gradient flow is not a quantitatively reliable proxy for gradient descent even on this simplest nonlinear model, because the quantity the flow conserves here is the quantity the discrete step actively shrinks.
The speed–regularization trade-off is genuine: the stronger the implicit regularization of $Q$, the slower the convergence, because the same $Q(\tau)$ that bounds the rate is the quantity being regularized.
Training at the edge of stability, with $\eta$ above $2/\lambda$, can still converge but only slowly, and that slowness is precisely what buys the additional flattening — a provable sense in which large-step training regularizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that its speed–regularization coupling suggests a deliberate schedule: keeping $\eta$ large late in training converts slow convergence into extra flatness, a lever worth testing on deeper models.
The mechanism is generic enough to predict, for matrix factorization and deeper linear nets, that layer-wise imbalance measures decrease under GD, stay constant under GF, and decrease faster at larger stepsizes — a checkable signature outside this paper's model.
The trajectory-wise PL argument suggests a template for other non-convex problems whose global PL constant is zero: identify the scalar that both controls the slowest direction and is itself reduced by the discrete update, and use it as the effective rate constant.
The logarithmic-slowdown prediction is a falsifiable fingerprint of this mechanism in real networks: in the edge-of-stability regime the loss should decay polynomially rather than exponentially; exponential decay there would indicate that the toy model's mechanism is not what real training is doing.

Share X Bluesky LinkedIn Reddit HN

Formalized claims in Lean

Claim #1: On the objective $L(a,b)=\tfrac{1}{2}(a^\top b-\Phi)^2$ with $\Phi\ge 0$, the paper claims that gradient descent with stepsize in the range $0<\eta<\min\{1/(2|\varepsilon(0)|),\,2/\sqrt{\lambda(0)^2+4\Phi^2}\}$ converges to a global minimum at a linear rate whose dependence on $\eta$, the initialization, and $\Phi$ is explicit, and that for larger stepsizes it still converges, but only logarithmic

/-- @claim 1 On the objective $L(a,b)=\tfrac{1}{2}(a^\top b-\Phi)^2$ with $\Phi\ge 0$, the paper claims that gradient descent with stepsize in the range $0<\eta<\min\{1/(2|\varepsilon(0)|),\,2/\sqrt{\lambda(0)^2+4\Phi^2}\}$ converges to a global minimum at a linear rate whose dependence on $\eta$, the initialization, and $\Phi$ is explicit, and that for larger stepsizes it still converges, but only logarithmic -/
def central_claim : Prop :=

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the high-level story is good: GD on the product-parameterized loss 1/2(a^T b - Phi)^2 converges to a global minimum with an explicit rate, and it systematically reduces the imbalance Q while gradient flow conserves it, so larger steps flatten the solution. Second, the proof of the explicit rate has a genuine bug. The phenomenon may be true, but the main theorem is not established by the current argument.

What is new and useful: the paper sharpens Wang et al. (2022), who proved convergence without a rate, to an explicit rate; it characterizes the limit point via the Q imbalance; and it points out the speed-regularization trade-off. The update equations (6)-(8) are correct, the reduction of the empirical loss to the scalar form is clean, and the paper is self-contained. No hidden fitting.

The soft spot: Proposition 2 asserts that when the trajectory starts with a^T b < 0, by the time it crosses into a^T b > 0 the imbalance satisfies Q(tau) > 2 sqrt(eta Phi). That is false as stated. A concrete admissible initialization: Phi=1, a=(0.5,0.1), b=(-0.2,-0.2), eta=0.4. Here epsilon(0)=-1.12, Q(0)=0.24, and the hypotheses on eta are satisfied, but one GD step gives Q(1) ~ 0.192 while 2 sqrt(eta Phi) ~ 1.265. Since Q(0) is already smaller than the claimed bound, the proposition cannot hold. The algebraic source is that Lemma 7 uses lambda^2 - 4(a^T b)^2 = Q^2 as an equality, which is only true in d=1; for d>1 there is an additional nonnegative cross term (Lemma 10 states the inequality but does not carry it through). Proposition 2 feeds directly into the lower bound on the PL constant and hence into the explicit time in Theorem 2, so that theorem is not justified for d>1 as written. The statement of Theorem 2 also contains the log of a possibly negative quantity, which suggests the final formulas need care.

In proportion: this is a proof gap, not a refutation of the paper's intuition. In the counterexample the dynamics still appears to converge linearly, just at a rate controlled by Q(tau) rather than by the larger Q(0). So the result is likely salvageable with a correct lower bound on Q(tau). But as it stands, the central claim is unproven.

This paper is for readers who care about implicit regularization, edge-of-stability, and linear nets. I'd take it seriously as a submission, but only with major revision. My recommendation: send it to a competent referee who will check the region-C algebra; if the authors fix Proposition 2 and the dimensional identity, this should be a solid contribution. Until then, the explicit rate claim should not be taken on faith.

Referee Report

3 major / 7 minor

Summary. The paper analyzes gradient descent (GD) on the depth-2 linear network loss L(a,b)=1/2(a^T b−Φ)^2, a,b∈R^d, through three scalar descriptors: the residual ε=a^T b−Φ, the norm λ=∥a∥²+∥b∥², and the per-coordinate imbalance Q=Σ_i|a_i²−b_i²|, with discrete updates (6)–(8). The main claims are: Theorem 1 states that for η<min{1/(2|ε(0)|), 2/λ̄}, GD reaches a minimizer for which each |Q_i| is strictly between |Q_i(0)|exp(−√ηε(0)²/Φ) and |Q_i(0)|exp(−η²Σ_t ε(t)²), so GD implicitly regularizes Q—and hence solution sharpness—strictly more than gradient flow, which conserves each Q_i. Theorem 2 claims an explicit two-phase linear convergence bound for the same step-size range, and a slower logarithmic-convergence regime for a larger step-size range, framed as an edge-of-stability phenomenon. The proof partitions the state space into three regions (A: ε>0; B: ε<0<a^T b; C: a^T b<0), establishes a trajectory-dependent PL lower bound controlled by Q(τ)—the imbalance at the time the trajectory exits Region C—and bounds that crossing time in Proposition 2. The paper concludes with a speed-versus-regularization trade-off and implications for using gradient flow as a surrogate for GD.

Significance. If the main theorems were correct, this would be a clean and useful contribution: it would add explicit rates and a sharp GD-versus-GF comparison to the known convergence result of Wang et al. (2022), and would give one of the simplest rigorous demonstrations that discretization itself can flatten the selected minimum. The paper is self-contained, the reparameterization is elegant, the Q-decay identity (8) is exact and easily verified, the upper bound in Theorem 1 follows directly from it, the numerical illustration in Figure 3 is reproducible, and the qualitative predictions (larger η ⇒ smaller final Q; GD differs from GF) are falsifiable. These sound elements should be credited. However, the added value over prior work rests precisely on the explicit quantitative statements, and those statements are not established as written: Proposition 2 is refuted by an explicit admissible initialization (see Major Comment 1), the exact algebraic identity used in Lemmas 7 and 10 fails for d>1 (Major Comment 2), and Lemma 20—the only support for Theorem 1's lower bound—is an incomplete fragment (Major Comment 3).

major comments (3)

[Appendix E, Proposition 2; Theorem 2 (Appendix F.5)] Proposition 2 is false as stated, and the counterexample is an admissible Region-C initialization satisfying all of the proposition's hypotheses (Appendix E). Take Φ=1, a(0)=(0.5,0.1), b(0)=(−0.2,−0.2); then a(0)^T b(0)=−0.12<0, ε(0)=−1.12, λ(0)=0.34, Q(0)=0.24, λ̄=√(λ(0)²+4)≈2.029, and η=0.4 satisfies η<min{1/|ε(0)|,2/λ̄}=min{0.893,0.986}; note also η|ε(0)|=0.448<1/2, so this is the 'small step' case of Appendix E.3, not a borderline-large-step case. One GD step via (4) gives a(1)^T b(1)=0.0082>0, so τ=1, while (8) gives Q(1)=(1−η²ε(0)²)Q(0)=0.1918<1.2649=2√(ηΦ), contradicting the claimed bound 2√(ηΦ)<Q(τ). The same example invalidates Lemma 15's bounds: w0−4ηz0²=0.24−2.007<0, so c1 is not real, and the claimed lower bound τ1≥1/(ηw0^{3/2})≈21.3 contradicts the observed τ1=1. This is load-bearing: Appendix F.5, Eq. (79), converts the Q(τ) lower bound into the explicit iteration count of Theorem 2, and Appendix H's Lemma 20 converts the same (failed) bound into the lower bound of Theorem 1. The trajectory itself still converges (Region B takes over after τ=1, with ε(2)/ε(1)≈0.92), so the qualitative phenomenon may be salvageable, but the explicit-rate claim—the stated novelty relative to Wang et al. (2022)—is not proven, and no uniform lower bound on Q(τ) in terms of initial data is available from the current argument. Additionally, the 'log(a(0)^T b(0))' term in Theorem 2's bound (10) and in Proposition 2's τ bound is undefined for a(0)^T b(0)<0, i.e., for exactly the Region-C case these results are meant to cover (presumably log|a(0)^T b(0)| was intended), and Theorem 2's displayed rate (11) does not match the closing bound (79), which depends on Q(τ) rather than on Q(0)exp(min{−a(0)^T b(0),0}).
[Appendix C.2, Lemma 7; Appendix E.1, Lemma 10] Lemma 7 and Lemma 10 rely on an identity that fails for d>1. In Lemma 7's proof the degree-2 term −2η²ε²(λ²−4(ε+Φ)²) is declared 'exactly equal to −2η²ε²Q²', which requires λ²−4(a^T b)²=Q²; this holds only for d=1 (or a∥b). For d>1, λ²−4(a^T b)²=Q²+4Σ_{i<j}(a_i b_j−a_j b_i)²; e.g., a=(3,4), b=(−1,−2) gives λ²−4(a^T b)²=900−484=416 while Q²=400. Lemma 10's proof is also invalid: it asserts λ²=Q²+4∥a∥²∥b∥² 'wlog' with a_i>b_i for all i, but Q=Σ|Q_i| is not invariant under such a normalization, and the identity fails when the Q_i have mixed signs (a=(1,4), b=(2,3) gives λ²=900 but Q²+4∥a∥²∥b∥²=984; the correct identity is λ²=(ΣQ_i)²+4∥a∥²∥b∥²). Consequently, the exact formula in Lemma 2, α(t+1)=α(t)−2η²ε(t)²Q(t)²|1−η²ε(t)²|, is not established for d>1, and the derivations that depend on it—Lemma 8's bound λ(t)≤λ̄ and the sharpness comparison of Section 4 (GD's limit has lower λ than GF's, via α-monotonicity)—need to be redone with the corrected expansion. In my own numerical check (a=(3,4), b=(−1,−2), Φ=1, η=0.01) the actual decrease of α is −11.90 versus −11.35 predicted by Lemma 7, so the direction may survive, but the stated identity is wrong and the higher-order terms differ as well.
[Appendix H, Lemma 20 and Theorem 1] Lemma 20 in Appendix H, the only support for the lower bound in Theorem 1, is incomplete and internally inconsistent. Its hypothesis 'assume there exists t̃ such that for all t≥t̃ we have η|ε(0)|<1/2' must read η|ε(t)|<1/2, and the proof stops in mid-sentence at 'Qi(t)' with no conclusion. More substantively, the proof's final conversion to |Qi(t)|≥Q(0)exp(−√ηε(0)²/Φ) 'plugs in the lower bound in Lemma 16', i.e., the very Q(τ)≥2√(ηΦ) bound refuted by the counterexample in Major Comment 1; and the proof invokes Theorem 2 for exponential convergence, making Theorem 1's lower bound contingent on Theorem 2, whose own proof (Appendix F.5) depends on Proposition 2. There is no strict cycle (Theorem 2's proof does not call Lemma 20), but the dependency chain means both quantitative claims stand or fall together. As written, the claim 0<|Q_i(0)|exp(−√ηε(0)²/Φ)<|Q_i(∞)| is unsupported; the upper bound in Theorem 1 follows directly from (8) and is not affected. Note also that both Theorems 1 and 2 divide by Φ and therefore require Φ>0, while the text only assumes Φ≥0.

minor comments (7)

[Section 3 / Lemma 4] The loss is defined as L=1/2(a^T b−Φ)² in Eq. (3), but Lemma 4 and its proof assert 'L=ε²'; the factor 1/2 is dropped inconsistently and propagates into the O(·) constants of Theorem 2.
[Throughout] The manuscript needs a careful proofread; examples include 'inbalance' (Sections 4 and 7), 'abnalyze' (Appendix D.2), 'our work hint to a possible...benefit' (Section 5), and the garbled footnote 3, which says the case 'Q(t) ≠ 0 for some t' is handled in Appendix F when it must mean Q(t)=0.
[Figure 2] The regions A, B, C are defined only in the proof sketch of Section 6, and the figure's labels and axes do not make the partition self-explanatory; add the definitions (ε>0; ε<0<a^T b; a^T b<0) to the caption and mark the boundaries.
[Theorem 2, second case] The interval min{√2/|ε|, 2/√(λ(0)²+4Φ²)} < η < min{2/|ε|, 2/(λ+2ε(ε+Φ)/λ³)} mixes initial quantities (λ(0)) with time-dependent ones (ε, λ) without specifying the time index, so it is not a checkable condition on the initialization.
[Eq. (14)] The rescaling notation 'λ ↶ [...] λ' with a reversed arrow is nonstandard and easy to misread, and the assertion that 'all our proofs work on L̄' is not traced through the theorems, whose hypotheses are stated only for the normalized loss (2).
[Appendix G] The series expansions for η1 and η2 in Eqs. (80)–(81) are never used in the proofs and the displayed terms are not a consistent expansion ('14α8' follows a quadratic); either derive and use them or remove the section.
[Appendix E.3, Lemma 15] 'Sedrakyan's lemma' is invoked without a statement or reference, and the notation 'wτ−2' is ambiguous between w_{τ−2} and w_τ−2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper derives its convergence and implicit-regularization claims by direct algebra from the GD update equations, with no fitted parameters, no imported prior results, and no self-citation chain.

full rationale

The derivation chain is self-contained. Equation (4) is the definition of gradient descent on L(a,b), and the reparametrization into epsilon(t), lambda(t), and Q(t) is exact algebra: equations (6)-(8) follow by expanding the GD update, not by assuming a conclusion. Lemma 3 reduces the empirical risk to L(a,b) with an explicit constant independent of (a,b), so the simplified objective is not a hidden restatement of the target result. The convergence proof uses bounding sequences (Definition 2) and invariants/decay identities (Lemmas 5-7) that are established directly, rather than imported from the authors' own prior work; indeed the paper contains no self-citations and the cited prior work (Wang et al. 2022, Lewkowycz et al. 2020, etc.) is contextual and not load-bearing. The central claims -- linear rate in Theorem 2, imbalance bounds in Theorem 1, and the GF-versus-GD comparison in Section 4 -- are all consequences of the exact recurrences, not renamings or fits. The skeptical finding that Proposition 2's lower bound Q(tau)>2*sqrt(eta*Phi) may fail for some d=2 initializations identifies a possible algebraic error in Lemma 10, but a false or unsupported inequality is a correctness gap, not circularity. It does not make any prediction equivalent to its input by construction, so the circularity score remains 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper is a self-contained mathematical analysis. It does not fit any constants to data; the step size eta, target Phi, and initialization (a(0), b(0)) are inputs, not fitted parameters. The analysis relies on standard calculus and on the domain assumption that the problem is the univariate depth-2 linear network objective. The condition Q(0) != 0 is an initialization assumption in Theorem 2, handled separately in Appendix D.2.

assumptions (3)

standard math Standard calculus and the definition of gradient descent for the quadratic loss L = 1/2(a^T b - Phi)^2
Used to derive the update equations (4)-(8) and the conserved/contracted quantities in Lemmas 5-7.
domain assumption The univariate reduction (Lemma 3): the empirical loss for scalar inputs collapses to L = 1/2(a^T b - Phi)^2 up to a constant
This makes the problem one-dimensional in terms of residuals epsilon; it does not generalize to vector-valued inputs.
domain assumption Initialization Q(0) != 0 in Theorem 2
The linear rate bound relies on a positive lower bound on the imbalance; the Q = 0 case is treated separately in Appendix D.2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks." pith.science (2026). https://pith.science/paper/AIMT6Q3B

@misc{pith2026250109137,
  author       = {Pith},
  title        = {Pith review of: Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AIMT6Q3B}},
  note         = {Machine review of arXiv:2501.09137}
}

abstract

We study the gradient descent (GD) dynamics of a depth-2 linear neural network with a single input and output. We show that GD converges at an explicit linear rate to a global minimum of the training loss, even with a large stepsize -- about $2/\textrm{sharpness}$. It still converges for even larger stepsizes, but may do so very slowly. We also characterize the solution to which GD converges, which has lower norm and sharpness than the gradient flow solution. Our analysis reveals a trade off between the speed of convergence and the magnitude of implicit regularization. This sheds light on the benefits of training at the ``Edge of Stability'', which induces additional regularization by delaying convergence and may have implications for training more complex models.

Figures

Figures reproduced from arXiv: 2501.09137 by the authors.

**Figure 1.** offers a geometric intuition: GF conserves the quantities Qi by curving away from the origin: Q˙ i = 2(ai a˙ i − bi b˙ i) = 2 ai(− ε bi) − bi(− ε ai) = 0. 2A similar result holds for larger stepsizes, but its statement is more involved. See Appendix H. GF Trajectory GD step a = b = √ Φ a b [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** Schematic of GD behaviors in three different regions: (A) ε > 0, (B) ε < 0 < a ⊤b, (C) a ⊤b < 0. See text for details. To establish the rate of convergence for small and moderate step sizes we now must ensure λ(t) never becomes too small, i.e., the dynamics stays away from the origin. The key idea here is to notice that λ ≥ Q and studying the evolution of its size partitioning the parameter space into the three “re… view at source ↗

**Figure 3.** Gradient descent on (2) with Φ = 1 and various step sizes and initial scales. Left: Ratio Q(T)/Q(0) showing how little Q changes when η λ(0) is small and how strongly it is reduced for large η λ(0). Right: The time to converge to a small residual, illustrating slower convergence in cases with stronger Q-regularization. The chaotic behavior appears when η ≥ 1/ε. dynamics are inherently different from gradient descen… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

T., Suarez, F., and Zhang, Y

Ahn, K., Bubeck, S., Chewi, S., Lee, Y. T., Suarez, F., and Zhang, Y. Learning threshold neurons via edge of stability. In Advances in Neural Information Processing Systems , volume 36, 2024

work page 2024
[3]

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks , October 2019

Arora, S., Cohen, N., Golowich, N., and Hu, W. A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks , October 2019. URL http://arxiv.org/abs/1810.02281. arXiv:1810.02281 [cs, stat]

arXiv 2019
[4]

P., Selman, B., and Weinberger, K

Bjorck, N., Gomes, C. P., Selman, B., and Weinberger, K. Q. Understanding batch normalization. In Advances in neural information processing systems, volume 31, 2018

work page 2018
[5]

E., and Nocedal, J

Bottou, L., Curtis, F. E., and Nocedal, J. Optimization Methods for Large - Scale Machine Learning , February 2018. URL http://arxiv.org/abs/1606.04838. arXiv: 1606.04838

arXiv 2018
[6]

and Bruna, J

Chen, L. and Bruna, J. Beyond the edge of stability via two-step gradient updates. In International Conference on Machine Learning , pp.\ 4330--4391. PMLR, 2023

work page 2023
[7]

Z., and Talwalkar, A

Cohen, J., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations , 2021

work page 2021
[8]

Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks

Gidel, G., Bach, F., and Lacoste-Julien, S. Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks . In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/f39ae9ff3a81f499230c4126e01f421b-Abstract.html

work page 2019

Show all 26 references

[9]

and Schmidhuber, J

Hochreiter, S. and Schmidhuber, J. Flat minima. Neural Computation, 9 0 (1): 0 1--42, 1997. Publisher: MIT Press

1997
[10]

The Break - Even Point on Optimization Trajectories of Deep Neural Networks , February 2020

Jastrzebski, S., Szymczak, M., Fort, S., Arpit, D., Tabor, J., Cho, K., and Geras, K. The Break - Even Point on Optimization Trajectories of Deep Neural Networks , February 2020. URL http://arxiv.org/abs/2002.09572. arXiv: 2002.09572

2020 arXiv
[11]

S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P

Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima, 2016

2016
[12]

B., and Müller, K.-R

LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. Efficient backprop. In Neural networks: Tricks of the trade , pp.\ 9--50. Springer, 2002

2002
[13]

The large learning rate phase of deep learning: the catapult mechanism, March 2020

Lewkowycz, A., Bahri, Y., Dyer, E., Sohl-Dickstein, J., and Gur-Ari, G. The large learning rate phase of deep learning: the catapult mechanism, March 2020. URL http://arxiv.org/abs/2003.02218. arXiv:2003.02218 [cs, stat]

2020 arXiv
[14]

Towards explaining the regularization effect of initial large learning rate in training neural networks

Li, Y., Wei, C., and Ma, T. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in neural information processing systems, volume 32, 2019

2019
[15]

M., Rauhut, H., and Terstiege, U

Nguegnang, G. M., Rauhut, H., and Terstiege, U. Convergence of gradient descent for learning linear neural networks. Advances in Continuous and Discrete Models, 2024 0 (1): 0 1--28, 2024. Publisher: Springer

2024
[16]

The Effect of Network Width on Stochastic Gradient Descent and Generalization : an Empirical Study

Park, D., Sohl-Dickstein, J., Le, Q., and Smith, S. The Effect of Network Width on Stochastic Gradient Descent and Generalization : an Empirical Study . In Proceedings of the 36th International Conference on Machine Learning , pp.\ 5042--5051. PMLR, May 2019. URL https://proce...

2019
[17]

Polyak, B. T. Gradient methods for the minimisation of functionals. USSR Computational Mathematics and Mathematical Physics, 3 0 (4): 0 864--878, 1963. Publisher: Elsevier

1963
[18]

M., McClelland, J

Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, February 2014. URL http://arxiv.org/abs/1312.6120. arXiv:1312.6120 [cond-mat, q-bio, stat]

2014 arXiv
[19]

Smith, S. L. and Le, Q. V. A Bayesian Perspective on Generalization and Stochastic Gradient Descent , February 2018. URL http://arxiv.org/abs/1710.06451. arXiv: 1710.06451

2018 arXiv
[20]

D., and Vidal, R

Tarmoun, S., Franca, G., Haeffele, B. D., and Vidal, R. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning , pp.\ 10153--10161. PMLR, 2021

2021
[21]

Large Learning Rate Tames Homogeneity : Convergence and Balancing Effect

Wang, Y., Chen, M., Zhao, T., and Tao, M. Large Learning Rate Tames Homogeneity : Convergence and Balancing Effect . In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=3tbDrs77LJ5

2022
[22]

and Ziyin, L

Xu, Y. and Ziyin, L. Three Mechanisms of Feature Learning in the Exact Solution of a Latent Variable Model , May 2024. URL http://arxiv.org/abs/2401.07085. arXiv:2401.07085

2024 arXiv
[23]

Linear convergence of gradient descent for finite width over-parametrized linear networks with general initialization

Xu, Z., Min, H., Tarmoun, S., Mallada, E., and Vidal, R. Linear convergence of gradient descent for finite width over-parametrized linear networks with general initialization. In International Conference on Artificial Intelligence and Statistics , pp.\ 2262--2284. PMLR, 2023

2023
[24]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[25]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[26]

,# (7),01444 '9=82<.342C 2! !22222222222222222222222222222222222222222222222222

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

T., Suarez, F., and Zhang, Y

Ahn, K., Bubeck, S., Chewi, S., Lee, Y. T., Suarez, F., and Zhang, Y. Learning threshold neurons via edge of stability. In Advances in Neural Information Processing Systems , volume 36, 2024

work page 2024

[3] [3]

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks , October 2019

Arora, S., Cohen, N., Golowich, N., and Hu, W. A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks , October 2019. URL http://arxiv.org/abs/1810.02281. arXiv:1810.02281 [cs, stat]

arXiv 2019

[4] [4]

P., Selman, B., and Weinberger, K

Bjorck, N., Gomes, C. P., Selman, B., and Weinberger, K. Q. Understanding batch normalization. In Advances in neural information processing systems, volume 31, 2018

work page 2018

[5] [5]

E., and Nocedal, J

Bottou, L., Curtis, F. E., and Nocedal, J. Optimization Methods for Large - Scale Machine Learning , February 2018. URL http://arxiv.org/abs/1606.04838. arXiv: 1606.04838

arXiv 2018

[6] [6]

and Bruna, J

Chen, L. and Bruna, J. Beyond the edge of stability via two-step gradient updates. In International Conference on Machine Learning , pp.\ 4330--4391. PMLR, 2023

work page 2023

[7] [7]

Z., and Talwalkar, A

Cohen, J., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations , 2021

work page 2021

[8] [8]

Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks

Gidel, G., Bach, F., and Lacoste-Julien, S. Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks . In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/f39ae9ff3a81f499230c4126e01f421b-Abstract.html

work page 2019