Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

Han Bao

REVIEW 2 major objections 5 minor 40 references

Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that gradient descent on separable logistic regression reaches weak alignment with the max-margin direction in doubly exponential time, and that this early-stage alignment is tight.

desk verdict Genuinely new non-asymptotic early-alignment bound, but the discrete-time proof has a fixable step-size/initialization gap and the lower bound statement overclaims its range. read the letter →

arxiv 2608.04382 v1 pith:PFTOANDV submitted 2026-08-05 cs.LG stat.ML

Han Bao This is my paper · ORCID

classification cs.LGstat.ML MSC 68T0568Q3290C25

keywords implicitbiaslogisticregressiongradientdescentmax-marginearly-stagealignmentradial-tangentialdecompositionlinearseparabilitytightbounds

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies gradient descent on linearly separable logistic regression with an exponential loss and asks when the normalized parameter vector aligns with the max-margin direction. Prior asymptotic theory says that the alignment error decays slowly, like a polylogarithmic rate at large times. The paper proves that a weaker form of alignment—error V(t)=1-⟨u(t),u*⟩ at most δ, for δ above a data-dependent threshold near 1-γ—is reached in O(exp(exp(-δ))) iterations, and that this doubly exponential time is tight. This gives a theoretical account of the common empirical observation that parameters correlate with the max-margin direction very early, long before the slow asymptotic phase. It matters because it separates an early alignment stage from the asymptotic tail, with direct consequences for how implicit bias manifests in practical training time.

What carries the argument

The argument is carried by the radial/tangential decomposition w(t)=r(t)u(t), the alignment Lyapunov function V(t)=1-⟨u(t),u*⟩, and a geometric lemma lower-bounding the key drift term ⟨Zα(t), P⊥_{u(t)}u*⟩ ≥ κ0(V(t)) > 0 for V in (δ̄,1). The drift term couples the exponentially weighted average Zα of the data to the component of the max-margin direction orthogonal to u, and the geometric lemma shows this coupling stays uniformly positive once u is not nearly antipodal to u*. The remaining ingredients are the self-bounding property γL ≤ ∥∇L∥ ≤ L of the exponential loss, which yields risk and radial estimates, and a temperature-monotonicity lemma for the exponential weights. By avoiding any asymptotic expansion of u(t), the differential or difference inequality for V(t) is solved directly, giving the doubly exponential alignment time and its matching lower bound.

What would settle it

Compute the drift term ⟨Zα, P⊥_u u*⟩ for every unit vector u with V=1-⟨u,u*⟩ ∈ (δ̄,1) on a separable dataset with exponential weights; if any state gives a value below κ0(V), the geometric lemma that underlies both upper bounds is false. Alternatively, simulate gradient descent on a dataset with margin γ=0.2 and a near-unity data-mean margin, and check whether the measured alignment error V(t) at t=$η^{{-1}}$ exp(exp(-0.9)) is strictly above 0.9, which the upper bound would contradict.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a non-asymptotic law for the tangential alignment V(t). After an O(1) escape stage from poor initialization, gradient descent reaches V(t) ≤ δ within O(exp(exp(-δ))) time, for any δ strictly above a data-dependent threshold δ̄ ≤ 1-γ; when the data mean is well aligned, the threshold improves to around 1-√γ. The same statement holds for discrete-time gradient descent under a small step-size condition, with the iteration count multiplied by 1/η. The paper also proves a matching lower bound: before Ω(exp(exp(-δ))) iterations the alignment error is still at least δ, so the doubly exponential rate is not an artifact of the proof. This is a finite-time, early-stage phenomenon that asymptotic expansions miss; the proof works directly on the alignment dynamics using dataset geometry.

Load-bearing premise

The load-bearing premise is the small step-size condition η ≤ c0 min{ρL(0)^{-1}, γ²}; if the step size exceeds this, the discretization-error control in the discrete-time proof no longer holds, and the claimed O($η^{{-1}}$ exp(exp(-δ))) iteration bound is unproven.

Editorial extensions

If this is right

For any fixed alignment tolerance δ above the threshold, the weak alignment time is independent of how slowly the asymptotic tail converges; the parameter direction is effectively decided in the early phase.
The iteration bound O(η^{-1} exp(exp(-δ))) also holds for the logistic loss with adjusted constants, so the early-alignment phenomenon is not specific to the exponential loss.
The lower bound shows the doubly exponential time is tight: no proof can guarantee weak alignment substantially faster than Ω(η^{-1} exp(exp(-δ))) under the same assumptions.
Weak alignment up to 1-γ, and up to 1-√γ for favorably aligned data means, is a strictly stronger finite-time guarantee than the classical perceptron-style guarantee of 1-γ, and it comes with an explicit time bound.
The result separates training into an O(1) escape stage, a doubly exponential weak-alignment stage, and a slow asymptotic phase, giving a precise picture of the two-stage dynamics observed in simulations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: if the same drift bound can be established for other exponentially tailed losses and for related optimizers such as mirror descent, the doubly exponential alignment time would become a general property of separable linear classification rather than a quirk of exponential-loss logistic regression.
Editorial extension: the paper leaves the large-step-size regime open; an empirical check of whether step sizes above the stated threshold shorten or lengthen weak-alignment time would test the paper's speculation that discretization error can accelerate alignment.
Editorial extension: a probabilistic version over random data could turn the data-dependent threshold δ̄ into a high-probability statement in terms of the margin distribution, potentially bridging the gap between the worst-case threshold near 1-γ and the stronger √γ behavior seen in simulations.
Editorial extension: the doubly exponential form suggests weak alignment behaves like a sharp threshold event—the parameter direction snaps into roughly the right half-space almost immediately, then spends the rest of the early phase refining the correlation only logarithmically slowly.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This paper proves something that was actually missing in the implicit-bias literature: a finite-time, non-asymptotic bound showing that gradient descent on separable logistic regression achieves weak alignment ( ≥ 1 - δ) in O(exp(exp(-δ))) steps, i.e., essentially O(1) in the iteration count for any moderate δ. The asymptotic Soudry-style rate is O(1/log² t), so this early-stage alignment is a genuinely different phenomenon, and the paper is the first to prove it without asymptotic expansions.

The upper bound in continuous time (Theorem 8) checks out. The key step is Lemma 6, a purely geometric lower bound on in terms of V. That lemma is the real contribution; it cleanly avoids the asymptotic expansion and gives a dataset-dependent threshold for how small δ can be. The radial/tangential flow decomposition is standard, but the geometric control is new and well executed. The matching lower bound (Theorem 10) is the right idea, but its statement overclaims: it says V(t) ≥ δ for all δ > 0, which cannot hold when δ exceeds the initial gap, and the log-log term is ill-defined near t = 0. The proof in the appendix implicitly requires δ to be below the right threshold. That is a fixable statement error, not a fatal one.

The bigger soft spot is in the discrete-time analysis. Lemma 19's expansion of exp(η<∇L, z>) uses an absolute O constant, which requires η∥∇L(w(t))∥ to be uniformly bounded. Assumption 14 only gives ηL(0) ≤ c0 ρ, and ρ can be arbitrarily large; the induction in Remark 20 does not close a uniform bound independent of ρ. So Theorem 23's iteration count is not established as stated for large initial norm. The gap is likely repairable by adding ηL(0) ≤ C as an explicit condition, but then the O(η^{-1} exp(exp(-δ))) bound gains a ρ-dependent factor, which should be stated honestly. This is the main substantive criticism.

The paper is honest about its limitations—it only claims weak alignment, not perfect alignment, and it discusses the threshold δ and the open problem of improving it. The citation pattern is fine; the self-citation [3] appears in the large-stepsize context, not as a prop.

I'd send this to a serious referee. The continuous-time result is new and the proof is convincing; the discrete-time gap is real but likely fixable. If I were handling it, I'd ask for a revision that corrects Theorem 10's range and tightens the discrete-time stepsize assumption. The paper deserves a place in the literature once those are addressed.

Referee Report

2 major / 5 minor

Summary. This paper studies the early-stage dynamics of gradient descent (and gradient flow) for logistic regression with linearly separable data. Defining V(t)=1−⟨u(t),u*⟩ as the tangential alignment error to the max-margin direction, the main claim is that after an O(1) escape stage from poor initialization, the normalized parameter weakly aligns: V(t)≤δ is achieved after O(exp(exp(−δ))) time in continuous time and O(η^{-1}exp(exp(−δ))) iterations in discrete time, for any δ>Δ>δ̄ where δ̄≤1−γ is a data-dependent threshold. The proof tracks radial and tangential flows, proves a geometric lower bound on the projected exponentially weighted data term (Lemma 6), and gives a matching lower bound in the form of a minimal time before V(t) can drop below δ. The results are extended to the logistic loss and supported by simulations.

Significance. The paper makes a genuine contribution by giving a non-asymptotic, finite-time analysis of the early-stage implicit bias of gradient descent, complementing the slow asymptotic rate e^{O(1/log^2 t)}. The continuous-time upper bound (Theorem 8) is well structured and essentially correct: the geometric lemma (Lemma 6) properly verifies the ansatz, the temperature monotonicity lemma (Lemma 16) is proved, and the logistic-loss extension (Corollary 13) broadens the applicability. The lower-bound proofs are original, and the simulations illustrate the claimed phenomenon. The main value is the qualitative insight that weak alignment occurs in constant (problem-dependent) time rather than at the asymptotic rate. However, the discrete-time analysis contains a load-bearing gap in the risk/radial bounds under the stated step-size assumption, and the lower-bound statements need quantifier corrections before the tightness claim is justified.

major comments (2)

[Appendix E.1 (Lemma 19), Assumption 14] The discrete-time risk and radial bounds in Lemma 19 are not established under Assumption 14 as stated. The proof of the upper bound on L(w(t)) uses the Taylor expansion e^{η⟨∇L(w(t)), z_i⟩} = 1 + η⟨∇L(w(t)), z_i⟩ + O(η^2‖∇L(w(t))‖^2) and then derives the recurrence (20). This expansion is only valid with an absolute O-constant when |η⟨∇L(w(t)), z_i⟩| is bounded by a constant. Assumption 14 gives ηL(0) ≤ c0ρ, and the initialization condition ρ > γ^{-2} log L(0) allows ρ and hence ηL(0) to be arbitrarily large (e.g., ρ=10^6, γ=0.1, L(0)≈e^{10^4} satisfies the assumptions while ηL(0)≈c0·10^6). Remark 20's induction only proves ηL(w(t))/r(t) ≤ 1/5, which does not bound ηL(w(t)) in absolute terms when r(t) is large. Consequently the discrete bounds (17)–(19), the one-step recurrence (20), and the iteration bound of Theorem 23 are not proven for the full range of step sizes permitted by Assumption 14. The gap is fixable by strengthening Assumption 14 to include a uniform absolute bound such as ηL(0) ≤ C, but this restricts the admissible step size and the resulting ρ-dependence (through η^{-1} ≥ L(0)/C) should be stated explicitly.
[Theorem 10 and Theorem 24] The lower-bound statements are overbroad. Theorem 10 claims that for any δ>0, V(t) ≥ δ for all t∈[0,T1] with T1 = Ω(exp(exp(−δ))); Theorem 24 makes an analogous claim for δ>1−γ. Both fail when δ > V(0), because V(0)<δ at t=0, so no positive T1 can satisfy the conclusion. The proofs (Appendix C and Section E.3) require the exponent V(0)−C0−δ (continuous) or V(0)−C0−δ (discrete) to be positive for the derived interval to be nonempty; this condition is not stated in either theorem. Please add the explicit upper bound on δ (for example δ < V(0)−C0−c for a problem-dependent constant) and, in Theorem 24, state clearly that the claim only applies in the regime where the initialization has not already achieved the desired alignment.

minor comments (5)

[Section 1.3 (Theorem 2)] The notation eO(1/log^2 t) is used without definition; please define the tilde-notation (e.g., O-tilde) or replace it with an explicit polylogarithmic factor.
[Theorem 8 and Theorem 23] Since V(t) ∈ [0,2], the statements 'for any δ>Δ' should restrict δ to (Δ,2] (or to (Δ,V(0)] in the lower-bound statements) to avoid the trivial case δ≥2 where the conclusion holds immediately.
[Abstract and Section 4] The expression O(exp(exp(−δ))) for δ restricted to a bounded interval (δ>Δ>δ̄) is O(1); the authors should clarify that constants absorb problem-dependent quantities and that the doubly-exponential form describes the dependence on δ only through the exponent, not a decay rate in t.
[Section 4 informal summary] The lower-bound bullet states Ω(η^{-1}exp(exp(−δ))) without repeating the conditions δ>1−γ and the initialization assumption L(0)<e^{ργ^2}; please align the informal summary with the precise statement of Theorem 24.
[References] There are several typos in the reference list: [14] 'Proceedings ot', [23] 'Advanced in Neural Information Processing Systems', and [26] 'Proeedings'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central ansatz is proven geometrically, not assumed; constants are problem-dependent but not fitted; self-citation is not load-bearing.

full rationale

The paper's main derivation is self-contained. The 'ansatz' in (5), namely ⟨Zα, P⊥_u u*⟩ ≥ κ, is not an unproved input: Lemma 6 proves a strictly positive lower bound κ0(V) on the same inner product using the margin assumption, the temperature monotonicity Lemma 16, and Cauchy–Schwarz, and Section 3.1 shows by a quadratic inequality that κ0(V) > 0 for V ∈ (δ̄, 1). Thus Lemma 5's conditional statement becomes unconditional in Theorem 8. The threshold δ̄ is derived from κ0, not chosen by fitting, and the constants κ0(Δ) and γ are problem-dependent but not tuned to data. The discrete-time section (Appendix E) likewise derives its bounds rather than importing them; the Taylor-expansion step in Lemma 19 requires η∥∇L(w(t))∥ to be small relative to r(t), and Remark 20 attempts to establish this by induction. Whether that induction's O-constants are uniform for very large initialization norm ρ is a proof-tightness concern, not a circular reduction to the theorem's conclusion. The paper's own stated limitations—e.g., Section 5 says the analysis leaves larger stepsizes open and the error tolerance δ is limited by δ̄—are honest scope statements, not circularity. No fitted parameter is renamed as a prediction, and no load-bearing result is imported from the authors' earlier work; citation [3] appears only in related-work and discussion contexts.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No parameters are fitted to data. Problem-dependent constants (C0, T0, δ̄, κ0(Δ)) arise from the analysis and depend on the dataset and initialization; they are not tuned to make the theorem hold. The axioms are standard domain assumptions for linearly separable classification with bounded inputs and exponential/logistic loss.

assumptions (5)

domain assumption Dataset is linearly separable with margin γ: ⟨u*, z_i⟩ ≥ γ for all i (Assumption 1)
Defines the max-margin problem and is used in the self-bounding property γL ≤ ||∇L|| ≤ L and the geometric lemma.
domain assumption All inputs satisfy ||x_i|| ≤ 1 (Section 1.1)
Used to bound ||Zα|| ≤ 1 and ||z̄|| ≤ 1 throughout the proofs.
domain assumption Loss is exponential ℓ(m)=e^{-m} for the main theorem; logistic ℓ(m)=log(1+e^{-m}) for the extension
The self-bounding property and temperature monotonicity are proven for these losses; extension to other exponentially-tailed losses is only conjectured.
domain assumption For the discrete-time analysis, the step size satisfies η ≤ c0 min{ρ L(0)^{-1}, γ²} (Assumption 14)
Controls discretization error in the radial/tangential increments and the telescoping sums.
domain assumption Initialization satisfies ρ = ||w(0)|| > γ^{-2} log L(0) and L(0) < e^{ργ²} for the lower bound
Ensures the radial component stays positive and the refined radial lower bound is valid.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics." pith.science (2026). https://pith.science/paper/PFTOANDV

@misc{pith2026260804382,
  author       = {Pith},
  title        = {Pith review of: Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFTOANDV}},
  note         = {Machine review of arXiv:2608.04382}
}

abstract

Gradient descent has been of particular interest in modern machine learning beyond sole focus on optimization. Implicit bias emerging from optimization, though not being encoded by the learning objective, often prevents from overfitting to spurious patterns. A typical instance is the max-margin implicit bias of a linear classifier, widely established for exponentially tailed loss functions. Even after having a given dataset separated, the parameter vector continues to evolve towards the max-margin direction asymptotically along the gradient descent dynamics. This phenomenon corroborates a frequent empirical observation of "train longer, generalize better." However, the max-margin convergence is an asymptotic phenomenon, and what is worse, this asymptotic convergence rate is significantly slower than pure convex optimization. Even so, the parameter vector along gradient descent dynamics commonly correlates with the max-margin direction positively (though not exactly) within considerably fewer iterations than the asymptotic rate. By shedding another light on this classical problem, this work aims to understand the mechanism of this early-stage alignment phenomenon. Our theoretical results demonstrate that the parameter vector weakly aligns with the max-margin direction within $O(\exp(\exp(-\delta)))$ iterations, where $\delta>0$ is the permissible alignment error, which is shown to be tight. By tracking the radial and tangential flows, our proof operates on the alignment dynamics directly with dataset geometry and gets rid of the asymptotic expansion, which is a key insight to establishing faster weak alignment.

Figures

Figures reproduced from arXiv: 2608.04382 by the authors.

**Figure 1.** Simulation of gradient descent running with the exponential loss ℓexp(m) := exp(−m): (Left) a training dataset with larger margin; (Right) a training dataset with smaller margin. Each shows time evolution of the training risk L(w(t)), the radial ∥w(t)∥, the parameter alignment ⟨u(t), u∗⟩ (where u(t) := w(t)/∥w(t)∥), and the tangential trajectory (i.e., the trajectory of the normalized parameter u(t)) in the loss lan… view at source ↗

**Figure 2.** The geometry behind the proof of Lemma 6, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** The two-stage dynamics: the escape stage quickly moves away from poor initialization, followed by the weak alignment stage achieving ⟨u, u∗⟩ ≥ 1 − δ. 0 0.2 0.4 0.6 0.8 1 0 0.5 V κ κ0(V ) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Simulation results with the unnormalized parameter trajectory w(t) depicted in the loss landscapes, unlike the tangential trajectory u(t) depicted in [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 38 canonical work pages

[3]

H. Bao, S. Sakaue, and Y. Takezawa. Any-stepsize gradient descent for separable data under Fenchel– Young losses.Advances in Neural Information Processing Systems, 38, 2025

work page 2025
[1]

K. Ahn, J. Zhang, and S. Sra. Understanding the unstable convergence of gradient descent. InProceedings of the 39th International Conference on Machine Learning, pages 247–257, 2022

work page 2022
[2]

Ardeshir, D

N. Ardeshir, D. J. Hsu, and C. H. Sanford. Intrinsic dimensionality and generalization properties of the R-norm inductive bias. InProceedings of the 36th Conference on Learning Theory, pages 3264–3303, 2023. 10

work page 2023
[4]

Boursier, L

E. Boursier, L. Pillaud-Vivien, and N. Flammarion. Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs.Advances in Neural Information Processing Systems, 35:20105–20118, 2022

work page 2022
[5]

Y. Cao, Z. Chen, M. Belkin, and Q. Gu. Benign overfitting in two-layer convolutional neural networks. Advances in Neural Information Processing Systems, 35:25237–25250, 2022

work page 2022
[6]

Cohen, S

J. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. InProceedings of the 9th International Conference on Learning Representations, 2021

work page 2021
[7]

C. Fan, M. Schmidt, and C. Thrampoulidis. Implicit bias of spectral descent and Muon on multiclass separable data.Advances in Neural Information Processing Systems, 38, 2025

work page 2025
[8]

M. Glasgow. SGD finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the XOR problem. InProceedings of the 12th International Conference on Learning Representations, 2024

work page 2024

Show all 40 references

[9]

Gunasekar, J

S. Gunasekar, J. Lee, D. Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geometry. InProceedings of the 35th International Conference on Machine Learning, pages 1832–1841, 2018

2018
[10]

Gunasekar, J

S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro. Implicit bias of gradient descent on linear convolutional networks.Advances in Neural Information Processing Systems, 31:9461–9471, 2018

2018
[11]

Hoffer, I

E. Hoffer, I. Hubara, and D. Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks.Advances in Neural Information Processing Systems, 30: 1731–1741, 2017

2017
[12]

Ji and M

Z. Ji and M. Telgarsky. The implicit bias of gradient descent on nonseparable data. InProceedings of the 32nd Conference on Learning Theory, pages 1772–1798, 2019

2019
[13]

Ji and M

Z. Ji and M. Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks. InProceedings of the 8th International Conference on Learning Representations, 2020

2020
[14]

Z. Ji, M. Dudík, R. E. Schapire, and M. Telgarsky. Gradient descent follows the regularization path for general losses. InProceedings ot the 33rd Conference on Learning Theory, pages 2109–2136, 2020

2020
[15]

Koltchinskii and D

V. Koltchinskii and D. Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers.The Annals of Statistics, 30(1):1–50, 2002

2002
[16]

Liang and G

S. Liang and G. Montúfar. Implicit bias of mirror flow for shallow neural networks in univariate regression. InProceedings of the 13th International Conference on Learning Representations, 2025

2025
[17]

K. Lv, S. Jiang, and J. Li. Learning gradient descent: Better generalization and longer horizons. In Proceedings of the 34th International Conference on Machine Learning, pages 2247–2255, 2017

2017
[18]

Lyu and J

K. Lyu and J. Li. Gradient descent maximizes the margin of homogeneous neural networks. InProceedings of the 8th International Conference on Learning Representations, 2020

2020
[19]

S. Y. Meng, A. Orvieto, D. Y. Cao, and C. De Sa. Gradient descent on logistic regression with non-separable data and large step sizes.arXiv preprint arXiv:2406.05033, 2024

2024 arXiv
[20]

M. S. Nacson, J. Lee, S. Gunasekar, P. H. P. Savarese, N. Srebro, and D. Soudry. Convergence of gradient descent on separable data. InProceedings of the 22nd International Conference on Artificial Intelligence and Statistics, pages 3420–3428, 2019

2019
[21]

Onconvergenceproofsforperceptrons

A.B.J.Novikoff. Onconvergenceproofsforperceptrons. InProceedings of the Symposium on Mathematical Theory of Automata, pages 615–620, 1962. 11

1962
[22]

Ongie, R

G. Ongie, R. Willett, D. Soudry, and N. Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case. InProceedings of the 8th International Conference on Learning Representations, 2020

2020
[23]

H. Ravi, C. Scott, D. Soudry, and Y. Wang. The implicit bias of gradient descent on separable multiclass data.Advanced in Neural Information Processing Systems, 37:81324–81359, 2024

2024
[24]

Rosset, J

S. Rosset, J. Zhu, and T. Hastie. Margin maximizing loss functions.Advances in Neural Information Processing Systems, 16:1237–1244, 2003

2003
[25]

Savarese, I

P. Savarese, I. Evron, D. Soudry, and N. Srebro. How do infinite width bounded norm networks look in function space? InProceedings of the 32nd Conference on Learning Theory, pages 2667–2690, 2019

2019
[26]

Schliserman and T

M. Schliserman and T. Koren. Stability vs implicit bias of gradient methods on separable data and beyond. InProeedings of the 35th Conference on Learning Theory, pages 3380–3394, 2022

2022
[27]

O. Shamir. Gradient methods never overfit on separable data.Journal of Machine Learning Research, 22 (85):1–20, 2021

2021
[28]

Soudry, E

D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19(70):1–57, 2018

2018
[29]

D. A. Tarzanagh, Y. Li, X. Zhang, and S. Oymak. Max-margin token selection in attention mechanism. Advances in Neural Information Processing Systems, 36:48314–48362, 2023

2023
[30]

Tsilivis, G

N. Tsilivis, G. Vardi, and J. Kempe. Flavors of margin: Implicit bias of steepest descent in homogeneous neural networks. InProceedings of the 13th International Conference on Learning Representations, 2025

2025
[31]

A. Tyurin. From logistic regression to the perceptron algorithm: Exploring gradient descent with large step sizes. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 20938–20946, 2025

2025
[32]

Vasudeva, J

B. Vasudeva, J. W. Lee, V. Sharan, and M. Soltanolkotabi. The rich and the simple: On the implicit bias of Adam and SGD.Advances in Neural Information Processing Systems, 38, 2025

2025
[33]

Woodworth, S

B. Woodworth, S. Gunasekar, J. D. Lee, E. Moroshko, P. Savarese, I. Golan, D. Soudry, and N. Srebro. Kernel and rich regimes in overparametrized models. InProceedings of the 33rd Conference on Learning Theory, pages 3635–3673, 2020

2020
[34]

J. Wu, P. L. Bartlett, M. Telgarsky, and B. Yu. Large stepsize gradient descent for logistic loss: Non- monotonicity of the loss improves optimization efficiency. InProceedings of the 37th Conference on Learning Theory, pages 5019–5073, 2024

2024
[35]

Xie and Z

S. Xie and Z. Li. Implicit bias of AdamW:ℓ∞-norm constrained optimization. InProceedings of the 41st International Conference on Machine Learning, pages 54488–54510, 2024

2024
[36]

H. Xu, C. Caramanis, and S. Mannor. Robustness and regularization of support vector machines.Journal of Machine Learning Research, 10:1485–1510, 2009

2009
[37]

Zhang, S

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. InProceedings of the 5th International Conference on Learning Representations, 2017

2017
[38]

Zhang, D

C. Zhang, D. Zou, and Y. Cao. The implicit bias of Adam on separable data.Advances in Neural Information Processing Systems, 37:23988–24021, 2024. 12 A Technicalities in Section 3.1 Tangential error ¯δ≤1−γ. Recall the quantity¯δdefined in (9): ¯δ := 1− s 1 + 2γ¯γ−¯γ2 − p (1 + ...

2024
[39]

This condition was used in Lemma 18 to establish the radial/tangential increments formula

Local gradient condition: η∥∇L(w(t))∥ ≤r(t)/5for all t≥ 0. This condition was used in Lemma 18 to establish the radial/tangential increments formula. We verify this by induction. Att = 0, the self-bounding property (3) gives η∥∇L(w(0))∥ ≤ηL(0) ≤c 0ρ < ρ/5. For the inductive st...
[40]

scattered

Discretization error:The risk/radial bounds, originally shown in Lemma 4 for the gradient flow, incur discretization error in Lemma 19 for the discrete-time dynamics, which is multiplicatively up to1−O (ηγ −2) and thus can be negligible with the stepsize choiceη≤c 0γ2. Subsequ...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [3]

H. Bao, S. Sakaue, and Y. Takezawa. Any-stepsize gradient descent for separable data under Fenchel– Young losses.Advances in Neural Information Processing Systems, 38, 2025

work page 2025

[2] [1]

K. Ahn, J. Zhang, and S. Sra. Understanding the unstable convergence of gradient descent. InProceedings of the 39th International Conference on Machine Learning, pages 247–257, 2022

work page 2022

[3] [2]

Ardeshir, D

N. Ardeshir, D. J. Hsu, and C. H. Sanford. Intrinsic dimensionality and generalization properties of the R-norm inductive bias. InProceedings of the 36th Conference on Learning Theory, pages 3264–3303, 2023. 10

work page 2023

[4] [4]

Boursier, L

E. Boursier, L. Pillaud-Vivien, and N. Flammarion. Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs.Advances in Neural Information Processing Systems, 35:20105–20118, 2022

work page 2022

[5] [5]

Y. Cao, Z. Chen, M. Belkin, and Q. Gu. Benign overfitting in two-layer convolutional neural networks. Advances in Neural Information Processing Systems, 35:25237–25250, 2022

work page 2022

[6] [6]

Cohen, S

J. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. InProceedings of the 9th International Conference on Learning Representations, 2021

work page 2021

[7] [7]

C. Fan, M. Schmidt, and C. Thrampoulidis. Implicit bias of spectral descent and Muon on multiclass separable data.Advances in Neural Information Processing Systems, 38, 2025

work page 2025

[8] [8]

M. Glasgow. SGD finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the XOR problem. InProceedings of the 12th International Conference on Learning Representations, 2024

work page 2024