Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regime

Amin Karbasi; Peiyuan Zhang

REVIEW 5 major objections 4 minor 34 references

Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regime

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Quadratic losses can trigger Edge of Stability: with $\beta=w_+^2-w_-^2$, gradient descent at step-sizes above the classical threshold converges linearly to an interpolator, and the bias depends on which side of $\eta\mu=1$ the step-size…

desk verdict Plausible and potentially important claim that quadratic loss can produce EoS, but the main theorems are proved only for a reduced system; the gap is real. read the letter →

arxiv 2412.08025 v1 pith:P52R7XUG submitted 2024-12-11 math.OC cs.LGstat.ML

Peiyuan Zhang , Amin Karbasi This is my paper · ORCID

classification math.OCcs.LGstat.ML MSC 90C3037N3065K10

keywords edgeofstabilitygradientdescentdiagonallinearnetworkquadraticlossimplicitbiasphasetransitionbifurcationoverparameterizedregression

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contrary to the belief that a quadratic loss cannot produce the Edge of Stability, this paper shows that it can: when the regression vector is parameterized as $\beta = w_+^2 - w_-^2$, gradient descent with a step-size above the classical $2/L$ threshold converges to a linear interpolator under explicit conditions. The formal proof works for one-sample regression in $d=2$, with loss $l(s)=s^2/4$, and the experiments indicate the phenomenon requires $d\ge2$, a nonzero target, and a non-degenerate input. The paper also proves linear convergence rates and bounds the distance $\|\beta_\infty-\beta^*\|$ in two regimes, showing that the limit changes character when $\eta\mu$ crosses $1$. If correct, the right question for EoS is not whether the loss grows subquadratically, but what the parameterization does to the landscape.

What carries the argument

The carrying object is the residual iteration obtained from the quadratic parameterization. GD on $\beta=w_+^2-w_-^2$ is converted exactly into a recurrence for $(a_t,a'_t,b_t,b'_t)$, with $a_{t+1}=(1-\eta r_t)^2a_t$, $b_{t+1}=(1-x\eta r_t)^2b_t$, and symmetric updates for the primed variables, where $r_t=(1+x^2)(a_t-a'_t+x(b_t-b'_t))-\mu$. After an initial phase the primed variables are treated as negligible, reducing the residual update to $r_{t+1}=-(1-\alpha_t+\beta_tr_t)r_t$, matching the paper's toy model $r_{t+1}=-(1-\alpha_t)r_t-\beta_tr_t^2$. The sign of $\alpha_t=2-2\eta(\mu-c_x b_t)$ controls everything: $\alpha_t>0$ gives damped oscillation and contraction, while $\alpha_t<0$ gives envelope expansion. The hard part is proving that for $|x|<1/(\eta\mu)$ the map eventually crosses from $\alpha_t<0$ to $\alpha_t>0$ and stays there; this is established through a bifurcation analysis of the two-dimensional map $(g,h)$, including a no-2-periodic-orbits lemma that rules out the non-convergent alternatives.

What would settle it

Run the exact GD iterates for $d=2$, $x=(1,0.5)$, $\mu=1$, $\eta=1.1$, $\alpha=10^{-3}$ and record $r_t=\langle\beta_{w_t},x\rangle-\mu$ together with $a'_t,b'_t$; if $|r_t|$ does not eventually obey the claimed $\exp(-\Theta(\eta\mu-1)(t-\bar t))$ contraction, or if $a'_t+b'_t$ is ever comparable to the envelope of $r_t$ after the initial phase, the reduction behind the theorems fails. A second check: sweep $\eta\mu$ upward from $1$ and record the first step-size at which the orbit has period larger than $2$; if that period arises inside the interval below $(3\sqrt{2}-2)/2$ for $|x|<1/(\eta\mu)$, the phase-transition lemma is false.

Watch

Extended reading notes

Core claim

Under Assumption 1 ($d=2$, $x=(1,x)$, $y=\mu$, $\beta^*=(\mu,0)$, $w_{0,\pm}=\alpha\mathbf{1}$, $l(a)=a^2/4$), the paper proves two theorems. For $\eta\mu\in(0,1)$, once the residual changes sign after some time $t_0$, GD converges linearly, $|\langle\beta_{w_t}-\beta_\infty,x\rangle|\le C_1e^{-\Theta(\eta\mu)(t-t_0)}|\langle\beta_{w_{t_0}}-\beta_\infty,x\rangle|$, and $\|\beta_\infty-\beta^*\|\le O(\alpha^{C_2})$; the sparse prior is recovered as $\alpha\to0$. For $\eta\mu\in(1,\min\{(3\sqrt{2}-2)/2,\,1+1/(4C)\})$ with $\alpha^2\ll\eta\mu-1$ and $|x|<1/(\eta\mu)$, GD again converges linearly, $|\langle\beta_{w_t}-\beta_\infty,x\rangle|\le C_3e^{-\Theta(\eta\mu-1)(t-\bar t)}|\langle\beta_{w_{\bar t}}-\beta_\infty,x\rangle|$, and $\|\beta_\infty-\beta^*\|\le C(\eta\mu-1)$; here the sparse prior is not recovered even as $\alpha\to0$. These results are a formal demonstration that a quadratic loss can sit in the EoS regime.

Load-bearing premise

The load-bearing premise is that after a short initial phase the two auxiliary variables $a'_t$ and $b'_t$ can be frozen at zero, so the original GD trajectory is exactly described by the reduced two-variable map; if that reduction leaks error into the residual, or if a chaotic orbit is reachable before contraction, the convergence rates in the theorems do not apply to the actual iterates.

Editorial extensions

If this is right

Subquadratic growth of the loss is not required for EoS; a quadratic loss combined with the quadratic parameterization is sufficient, so EoS analyses should track the full parameterized landscape.
For $\eta\mu<1$, $\|\beta_\infty-\beta^*\|\le O(\alpha^{C})$ means the sparse prior is recovered in the $\alpha\to0$ limit, extending the small-step-size rich-regime bias into the EoS regime.
For $\eta\mu>1$, $\|\beta_\infty-\beta^*\|\le C(\eta\mu-1)$ means the sparse prior is not recovered even with vanishing initialization; the step-size itself controls the bias.
In both regimes the residual contracts at a linear rate, so the EoS trajectory converges to a linear interpolator at a quantified speed rather than merely oscillating.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension of the same phase-transition argument is the $n$-sample diagonal network with $d>n$: one would predict a sharp change in the distance to the sparse prior as the step-size crosses the analogue of $\eta\mu=1$, measurable by sweeping $\eta$ at fixed $\alpha$.
The $\eta\mu>1$ error bound hints that large step-sizes act as an implicit regularizer pushing the solution away from the sparse prior; this could be checked by probing whether test error follows the $\eta\mu-1$ scale rather than the $\alpha$ scale.
The upper limit $(3\sqrt{2}-2)/2$ in Theorem 2 is a proof artifact, so the real bifurcation boundary between 2-cycles and longer orbits in the reduced $(r,s)$ map should be located by numerical continuation; doing so would yield a sharper conjecture for the maximal convergent step-size.
If one can bound the reduction error from freezing $a'_t,b'_t$ instead of assuming it away, the same scheme would likely extend the convergence proof beyond the stated $\eta\mu$ interval and to general $d$, because the hard part is the scalar phase transition rather than the linear algebra.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

X, here's my take on 2412.08025. The headline claim—that you can get Edge of Stability with a purely quadratic loss when beta = w_+^2 - w_-^2—is new and, based on the experiments, probably true. The paper does a real service by showing that the subquadratic-loss necessary condition from Ahn et al. and Song & Yun does not apply to parameterized regression. The exact reduction to the four-variable system (Lemma 5) is clean, and the phase-transition argument for ημ>1 is a genuine novelty.

The softness is in the proofs. The shift from the exact dynamics to the reduced (r,s) system is done by declaring a'_t = b'_t = 0 for t ≥ t0 (Appendix A.2, after Eq. 10). Lemma 7 only bounds those variables at the single time τ at the end of the initial phase. Once r_t changes sign, (1+ηr_t)^2 > 1 during positive r_t segments, so a'_t and b'_t can grow; no uniform bound is given. So Theorems 1 and 2 prove linear convergence for the reduced system, not for the GD iterates of the stated model. That is a real gap, not a stylistic quibble.

Lemma 14 is similar: the statement that chaotic/divergent orbits have 'negligible total measure' is asserted, not proved, and Proposition 1 explicitly assumes them away. So the ημ>1 branch has a second unresolved issue. There's also a small inconsistency in Lemma 7: the statement says a'_τ ≤ Θ(α^4/μ), the proof line gives Θ(μ/α^4); one of those is wrong. The Mathematica root computations behind Lemma 13 are not included, which makes verification harder.

None of this convinces me the central claim is false. The experiments look consistent, and the reduction gap may be fixable with a more careful error bound on (a'_t,b'_t). But as it stands, the paper's main theorems are conditional on an unproven simplification.

Who should read this? People working on EoS theory and implicit bias of diagonal linear networks. It would be a useful contribution once the gap is closed. I'd send it to serious referees and ask them to focus on the dimensional reduction and the chaotic-orbit exclusion.

Referee Report

5 major / 4 minor

Summary. The paper studies constant-step-size gradient descent on the one-sample linear regression loss L(β)=⟨x,β⟩²/4 under the quadratic parameterization β=w_+²−w_-² (a depth-2 diagonal linear network). It claims that the Edge of Stability can occur even though l is quadratic, contrary to earlier claims that subquadratic growth is necessary. After an empirical study of regimes, the paper restricts to d=2 with x=(1,x), y=µ, β*=(µ,0), initialization w0,±=α1, and states two theorems: for ηµ∈(0,1) and assuming sign-changing residuals for t≥t0, GD converges linearly to an interpolator with ‖β∞−β*‖=O(α^C); for ηµ∈(1,min{(3√2−2)/2,1+1/(4C)}) and |x|<1/(ηµ), the same conclusion holds with ‖β∞−β*‖≤C(ηµ−1). The proof reduces the four-dimensional GD recursion exactly to a quadruplet, then approximates two of the four variables by zero and analyzes a two-dimensional map (r_t,s_t).

Significance. If the proof gaps were closed, the paper would be a significant contribution: it gives a concrete example where a quadratic loss exhibits EoS, an exact algebraic reduction of GD on a diagonal linear network to a low-dimensional map, and falsifiable predictions for the bias (α^C versus ηµ−1 scaling) that differ sharply between the two EoS subregimes. The empirical section is careful and identifies plausible necessary conditions (d≥2, nondegenerate x, y≠0). However, the rigorous bridge from the reduced system to the original GD trajectory is missing, and the existence of the sign-changing regime is assumed rather than proved, so the advertised theoretical result is not yet established.

major comments (5)

[Appendix A.2, Eq. (10)] The reduction from the four-variable GD dynamics to the two-variable system is not justified for t>τ. Lemma 7 bounds a'_τ and b'_τ only at the end of the initial phase, and the proof then sets a'_t ≡ b'_t ≡ 0 for all t≥t0. In the EoS sign-changing regime r_t is positive on alternating steps, so the factors (1+ηr_t)^2 and (1+xηr_t)^2 in Lemma 5 exceed 1 and a'_t,b'_t can in principle grow after τ. No uniform bound on these variables, or on their contribution to the γ_t term in Eq. (9) and to r_t in Lemma 6, is given. Therefore the linear convergence and bias bounds in Theorems 1 and 2 are established only for the reduced (r_t,s_t) system, not for the original GD iterates.
[Appendix A.4, Lemma 14] Lemma 14 is the load-bearing step for the ηµ>1 regime: it asserts that the (r_t,s_t) system eventually crosses α_t from negative to positive. The proof rules out divergence and chaos by saying their total measure is negligible, but no measure-theoretic or dynamical-systems argument is supplied. Since Proposition 1 explicitly assumes non-divergence and non-chaos, the convergence claim for ηµ>1 remains conditional on exactly the outcome that Lemma 14 is supposed to prove. A rigorous exclusion of chaotic/divergent orbits, or an explicit invariant set with a Lyapunov argument, is needed.
[Appendix A.3, Lemma 10 and Theorem 1] The contraction computation in Lemma 10 gives |r_{t+2}| ≤ (2−2ηµ)^2 |r_t|, which is a contraction only when ηµ>1/2. The proof then states |r_t| ≤ exp(Θ(2ηµ−1)(t−t0))|r_{t0}|; the exponent changes sign at ηµ=1/2 and is not derived from the preceding factor. If the sign-change assumption excludes ηµ≤1/2 in the EoS regime, that exclusion must be stated and proved. As written, Theorem 1's linear-rate claim for the full stated interval ηµ∈(0,1) is not supported by the proof.
[Appendix A.2, Lemma 7] There is an internal inconsistency in Lemma 7: the statement bounds a'_τ by Θ(α^4/µ), while the proof concludes a'_τ ≤ Θ(µ/α^4); the product bound a'_τ a_τ ≤ Θ(α^4) with a_τ≥µ/8 gives a'_τ ≤ O(α^4/µ). Moreover, none of these bounds extends to t>τ. This matters because the smallness of a'_τ is part of the justification for the reduction criticized above; the lemma, as written, does not establish even its stated bound.
[Theorem 1 and abstract] The paper's central claim that 'EoS occurs even when l is quadratic' is not established by the theorems. Theorem 1 assumes r_t r_{t+1}<0 for all t≥t0, which is the defining signature of the EoS trajectory; the proof therefore establishes convergence and bias for trajectories already in EoS, but not that such sign-changing trajectories exist for the stated parameter ranges. The occurrence claim remains an empirical observation (Claim 1). Please separate the conditional convergence result from the existence claim, or prove that the sign-change condition holds for some open set of parameters.

minor comments (4)

[Appendix A.2] The initialization line 'a0 = a′0 = b0 = b′0 = µ/(2(1+x²))' should read 'α²/(2(1+x²))'; Lemma 7's proof uses the α² form.
[Appendix A.4, Lemma 13] The proof relies on 'Mathematica Symbolic Calculation' for the roots of the 2-periodic system without displaying the polynomial system or verifiable output; please include the algebra or a computer-verifiable derivation.
[Lemma 8] The displayed update has an extra factor: it should be 'b_{t+1} = (1−x·η r_t)^2 · b_t > b_t' rather than 'b_{t+1} = (1−x·η r_t)^2 · b_t · b′_t > b_t'.
[Theorem 2] The symbol t used for the start of the convergence phase is visually close to t0 and to the subscript of r_t; please use a distinct symbol such as t_1 to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the convergence proofs are self-contained analytic derivations from the GD update, with conditional assumptions and proof gaps that are soundness issues, not input-output equivalences.

full rationale

The paper's central derivation starts from the exact GD update and derives an equivalent four-variable recurrence (Lemma 5) and a residual recurrence (Lemma 6) without importing the target convergence or bias conclusions. The parameters eta, mu, x, and alpha are inputs, and no parameter is fitted to the predicted bias or convergence rate. The reduction to a two-variable system by setting a'_t = b'_t = 0 is an approximation that is only partially justified by Lemma 7 and empirical plots; this is a genuine proof gap between the reduced and original dynamics, but it is not circular because the reduced system is not defined in terms of the convergence result. Theorem 1's sign-change assumption is explicitly a conditional restriction used to exclude the GF regime, rather than a restatement of the EoS definition, which is given in terms of sharpness crossing the 2/eta threshold. Lemma 14's dismissal of chaotic or divergent orbits as 'negligible' without a measure-theoretic argument is an unsupported soundness assertion, not a circular reduction. The paper contains no load-bearing self-citations and invokes no uniqueness theorem by its own authors. Therefore no step in the derivation reduces by construction to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central derivation does not fit constants to data; alpha, eta, mu, and x are inputs. The main extra imports are a modeling simplification and an exclusion of chaotic behavior, both unproven, plus unverified CAS root computations. These raise soundness risk but do not constitute invented physical entities.

assumptions (4)

domain assumption The auxiliary variables a_t' and b_t' can be set to zero after the initial phase without changing the convergence proof.
Used to obtain the reduced iteration in Eq. (10), Appendix A.2. Lemma 7 only bounds them at the end of the initial phase by O(alpha^C), but does not bound their effect on r_t over the entire oscillatory trajectory, so this is an unproven simplification.
domain assumption The excluded events 'divergence' and 'chaos' for the map (g,h) do not occur under the theorem parameters.
Lemma 14 (Appendix A.4) rules out fixed and 2-periodic points but then dismisses the complementary chaotic/divergent case as negligible total measure; Proposition 1 turns this into an assumption.
standard math Symbolic root computations reported from Mathematica are correct.
Lemma 13 states 'the computation result is also obtained by Mathematica Symbolic Calculation' and no notebook, code, or certificate is provided.
standard math Stability and periodicity conclusions from bifurcation theory in the cited textbooks apply to the maps (g,h).
Used in Lemma 13, Lemma 14, and Proposition 1 to exclude stable cycles. This is standard but relies on cited references rather than a self-contained proof.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regime." pith.science (2026). https://pith.science/paper/P52R7XUG

@misc{pith2026241208025,
  author       = {Pith},
  title        = {Pith review of: Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regime},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P52R7XUG}},
  note         = {Machine review of arXiv:2412.08025}
}

abstract

Classical optimization theory requires a small step-size for gradient-based methods to converge. Nevertheless, recent findings challenge the traditional idea by empirically demonstrating Gradient Descent (GD) converges even when the step-size $\eta$ exceeds the threshold of $2/L$, where $L$ is the global smooth constant. This is usually known as the Edge of Stability (EoS) phenomenon. A widely held belief suggests that an objective function with subquadratic growth plays an important role in incurring EoS. In this paper, we provide a more comprehensive answer by considering the task of finding linear interpolator $\beta \in R^{d}$ for regression with loss function $l(\cdot)$, where $\beta$ admits parameterization as $\beta = w^2_{+} - w^2_{-}$. Contrary to the previous work that suggests a subquadratic $l$ is necessary for EoS, our novel finding reveals that EoS occurs even when $l$ is quadratic under proper conditions. This argument is made rigorous by both empirical and theoretical evidence, demonstrating the GD trajectory converges to a linear interpolator in a non-asymptotic way. Moreover, the model under quadratic $l$, also known as a depth-$2$ diagonal linear network, remains largely unexplored under the EoS regime. Our analysis then sheds some new light on the implicit bias of diagonal linear networks when a larger step-size is employed, enriching the understanding of EoS on more practical models.

Figures

Figures reproduced from arXiv: 2412.08025 by the authors.

**Figure 1.** Comparison between EoS and GF regime, represented by blue and red lines, under parameterized linear regression in (1) with l(a) = a 2 /4. The plots from left to right illustrate the trajectory of regression weight βwt (star and triangle mark the stable points), the decrease of objective and ηSt, respectively, where St is the sharpness at iteration t. EoS is featured by the ηSt > 2. Unlike previous assertions, we obs… view at source ↗

**Figure 2.** Empirical verification for the Claim 1. In the left two columns of plots, we run with configurations that obey Claim 1 and EoS occurs if we increase step-size. In contrast, we set d = 1 in the third column and y = 0 in the fourth column, under these settings GD becomes divergent without triggering EoS when we increase the step-size. Note that we use a modified initialization w0,+ = 2α1, w0,− = α1 in the last column … view at source ↗

**Figure 3.** Influence of η and different asymptotic properties of rt along GD trajectory. When we increase the step-size, it displays, from left to right, GF regime, different subregimes of EoS, chaos, and divergence. In particular, when x is larger than some threshold (see Theorem 2 for details), GD does not converge when µη > 1. Parameter configuration: µ = 1, α = 0.01. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: α decides the length of the intermediate phase in ηµ > 1: the gap between the start of oscillation t0 and the start of convergence t is proportional to log(1/α). This is because in the intermediate phase, rt remains roughly as a constant and causes bt to increase almos…

**Figure 5.** Figure 5: Relationship between error kβ∞ − β ∗ k, α and η: the x-axis is α and y-axis is the error. The left plot characterizes the error under µη > 1 and the right plot is for regime µη < 1. Rest parameters: x = 0.5, µ = 1. The x-axis of both plots are in α. Effect of α, sparsi…

**Figure 6.** Figure 6: Empirical verification for the necessity of overparameterization under the multi-sample case. We plot the loss function L(wt) and the sharpness of GD when it admits more than one sample. When the model is overparameterized (d > n), EoS occurs when we increase the step-…

**Figure 7.** Figure 7: Toy model dynamics of rt in (3) under different regimes. The left and the middle plots utilize constant α with different signs and correspond to the oscillating contracting and the expanding regimes. The right plot uses varying αt = 1 20 · tanh( t−50 10 ) , which exhib…

**Figure 8.** Figure 8: Equivalent dynamics of quadruplet (at, a′ t , bt, b′ t) in Lemma 3. We compare EoS regime (η = 1.2) and GF regime (η = 0.5) in terms of sharpness, trajectory and convergence of |rt|, where ∆at = at − a ′ t and ∆bt = bt − b ′ t . In the first plot, we only picture the t…

**Figure 9.** Figure 9: Stable behavior of a ′ t and b ′ t iterations in Lemma 5. Under initialization a0 = a ′ 0 = b0 = b ′ 0 = µ 2(1+x2 , both a ′ t and b ′ t will decrease to zero in initial several iterations. We use different step-sizes in each plot, which corresponds to EoS with ηµ > 1,…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 18 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Learning threshold neurons via the "edge of stability"

Kwangjun Ahn, S \'e bastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via the" edge of stability". arXiv preprint arXiv:2212.07469, 2022 a

work page Pith review arXiv 2022
[3]

Understanding the unstable convergence of gradient descent

Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. In International Conference on Machine Learning, pp.\ 247--257. PMLR, 2022 b

work page 2022
[4]

Sgd with large step sizes learns sparse features

Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning, pp.\ 903--925. PMLR, 2023

work page 2023
[5]

Understanding gradient descent on the edge of stability in deep learning

Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning, pp.\ 948--1024. PMLR, 2022

work page 2022
[6]

On the implicit bias of initialization shape: Beyond infinitesimal mirror descent

Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake E Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In International Conference on Machine Learning, pp.\ 468--477. PMLR, 2021

work page 2021
[7]

On gradient descent convergence beyond the edge of stability

Lei Chen and Joan Bruna. On gradient descent convergence beyond the edge of stability. arXiv preprint arXiv:2206.04172, 2022

arXiv 2022
[8]

From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression

Xuxing Chen, Krishnakumar Balasubramanian, Promit Ghosal, and Bhavya Agrawalla. From stability to chaos: Analyzing gradient descent dynamics in quadratic regression. arXiv preprint arXiv:2310.01687, 2023

work page Pith review arXiv 2023

Show all 34 references

[9]

Gradient descent on neural networks typically occurs at the edge of stability

Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065, 2021

2021 arXiv
[10]

Adaptive gradient methods at the edge of stability

Jeremy M Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E Dahl, et al. Adaptive gradient methods at the edge of stability. arXiv preprint arXiv:2207.14484, 2022

2022 arXiv
[11]

Self-stabilization: The implicit bias of gradient descent at the edge of stability

Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594, 2022

2022 arXiv
[12]

(s) gd over diagonal linear networks: Implicit regularisation, large stepsizes and edge of stability

Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (s) gd over diagonal linear networks: Implicit regularisation, large stepsizes and edge of stability. arXiv preprint arXiv:2302.08982, 2023

2023 arXiv
[13]

Implicit regularization in matrix factorization

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems, 30, 2017

2017
[14]

Mirrorless mirror descent: A natural derivation of mirror descent

Suriya Gunasekar, Blake Woodworth, and Nathan Srebro. Mirrorless mirror descent: A natural derivation of mirror descent. In International Conference on Artificial Intelligence and Statistics, pp.\ 2305--2313. PMLR, 2021

2021
[15]

Catastrophic fisher explosion: Early phase fisher matrix impacts generalization

Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand, Giancarlo B Kerg, Huan Wang, Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof J Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In International Conference on Machine Learn...

2021
[16]

The large learning rate phase of deep learning: the catapult mechanism

Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218, 2020

2003 arXiv
[17]

Analyzing sharpness along gd trajectory: Progressive sharpening and edge of stability

Zhouzi Li, Zixuan Wang, and Jian Li. Analyzing sharpness along gd trajectory: Progressive sharpening and edge of stability. arXiv preprint arXiv:2207.12678, 2022

2022 arXiv
[18]

Benign oscillation of stochastic gradient descent with large learning rates

Miao Lu, Beining Wu, Xiaodong Yang, and Difan Zou. Benign oscillation of stochastic gradient descent with large learning rates. arXiv preprint arXiv:2310.17074, 2023

2023 arXiv
[19]

The multiscale structure of neural network loss functions: The effect on optimization and origin

Chao Ma, Lei Wu, and Lexing Ying. The multiscale structure of neural network loss functions: The effect on optimization and origin. arXiv preprint arXiv:2204.11326, 2022

2022 arXiv
[20]

Implicit bias of the step size in linear diagonal neural networks

Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning, pp.\ 16270--16295. PMLR, 2022

2022
[21]

Saddle-to-saddle dynamics in diagonal linear networks

Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. Advances in Neural Information Processing Systems, 36: 0 7475--7505, 2023

2023
[22]

Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity

Scott Pesme, Loucas Pillaud-Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. Advances in Neural Information Processing Systems, 34: 0 29218--29230, 2021

2021
[23]

An introduction to dynamical systems: continuous and discrete, volume 19

Rex Clark Robinson. An introduction to dynamical systems: continuous and discrete, volume 19. American Mathematical Soc., 2012

2012
[24]

Trajectory alignment: Understanding the edge of stability phenomenon via bifurcation theory

Minhak Song and Chulhee Yun. Trajectory alignment: Understanding the edge of stability phenomenon via bifurcation theory. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

2023
[25]

Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering

Steven H Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018

2018
[26]

Implicit regularization for optimal sparse recovery

Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems, 32, 2019

2019
[27]

Kernel and rich regimes in overparametrized models

Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp.\ 3635--3673. PMLR, 2020

2020
[28]

Implicit bias of gradient descent for logistic regression at the edge of stability

Jingfeng Wu, Vladimir Braverman, and Jason D Lee. Implicit bias of gradient descent for logistic regression at the edge of stability. Advances in Neural Information Processing Systems, 36, 2024

2024
[29]

A walk with sgd

Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. arXiv preprint arXiv:1802.08770, 2018

2018 arXiv
[30]

High-dimensional linear regression via implicit regularization

Peng Zhao, Yun Yang, and Qiao-Chu He. High-dimensional linear regression via implicit regularization. Biometrika, 109 0 (4): 0 1033--1046, 2022

2022
[31]

Understanding edge-of-stability training dynamics with a minimalist example

Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, and Rong Ge. Understanding edge-of-stability training dynamics with a minimalist example. arXiv preprint arXiv:2210.03294, 2022

2022 arXiv
[32]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[33]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[34]

reference

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

2000

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Learning threshold neurons via the "edge of stability"

Kwangjun Ahn, S \'e bastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via the" edge of stability". arXiv preprint arXiv:2212.07469, 2022 a

work page Pith review arXiv 2022

[3] [3]

Understanding the unstable convergence of gradient descent

Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. In International Conference on Machine Learning, pp.\ 247--257. PMLR, 2022 b

work page 2022

[4] [4]

Sgd with large step sizes learns sparse features

Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning, pp.\ 903--925. PMLR, 2023

work page 2023

[5] [5]

Understanding gradient descent on the edge of stability in deep learning

Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning, pp.\ 948--1024. PMLR, 2022

work page 2022

[6] [6]

On the implicit bias of initialization shape: Beyond infinitesimal mirror descent

Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake E Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In International Conference on Machine Learning, pp.\ 468--477. PMLR, 2021

work page 2021

[7] [7]

On gradient descent convergence beyond the edge of stability

Lei Chen and Joan Bruna. On gradient descent convergence beyond the edge of stability. arXiv preprint arXiv:2206.04172, 2022

arXiv 2022

[8] [8]

From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression

Xuxing Chen, Krishnakumar Balasubramanian, Promit Ghosal, and Bhavya Agrawalla. From stability to chaos: Analyzing gradient descent dynamics in quadratic regression. arXiv preprint arXiv:2310.01687, 2023

work page Pith review arXiv 2023