REVIEW 3 major objections 8 minor 28 references
Gradient Descent as a Shrinkage Operator for Spectral Bias
T0 review · 3 major / 8 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Gradient descent implicitly performs singular-value shrinkage whose learning rate and iteration count directly dial the spectral bandwidth of a shallow network.
desk verdict A useful heuristic—GD as a singular-value mask with an explicit q–K formula—but the headline formula comes from gradient flow, not exact GD, and the difference is unquantified where it matters. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the masking function $m_{\mathrm{gd}}(s;\alpha,q)=1-\exp(-\alpha q s^2)$, applied to the singular values $s$ of the design matrix $A=[\eta(\hat{x}_i-b_j)]$, alongside its companion relation $q=s_{\max}^{2}s_{K}^{-2}$. The mask converts gradient descent from an iterative procedure into a closed-form shrinkage operator: singular values above roughly $s_K$ pass nearly unchanged, values below are suppressed, and the active window has width $\rho\approx\sqrt{q}$. The second piece is the SVD of $A$ for different activations: for monotonic activations, the principal components approximate the discrete sine transform and the singular-value drop-off rate is set by $r$ in $\eta(x)=x^{r-1}[x>0]$, which the paper derives from spline smoothing; for non-monotonic activations the spectrum is controlled by the scale $\sigma$, with sinc's singular values staying near unity up to $k=\sigma$.
What would settle it
Run exact discrete gradient descent on the paper's peppers reconstruction (shallow net, $\alpha=s_{\max}^{-2}$, $\sigma=1024$) and sweep $q$; compare the $q$ at which the PSNR curve crosses the discrete-sine-transform baseline for $K$ components against $q=s_{\max}^{2}s_{K}^{-2}$ from the paper. If the crossing point is systematically off by about a factor of two or more in $q$, the exponential-mask relation fails as a quantitative predictor; the exact discrete-GD mask $1-(1-\alpha s^2)^q$ should be used to compute the predicted crossing instead.
Extended reading notes
Core claim
The central discovery is that the implicit regularization of gradient descent for least squares can be rewritten as an explicit shrinkage of the singular values $s$ of the design matrix $A$ (the Jacobian of the shallow network). For learning rate $\alpha$ and $q$ iterations, the effective inverse singular values are $\hat{s}^{-1}=[1-\exp(-\alpha q s^2)]s^{-1}$, so the mask $m_{\mathrm{gd}}$ is a smooth low-pass filter whose active window widens like $\sqrt{q}$. Setting $\alpha=s_{\max}^{-2}$ and reading the mask at threshold $\epsilon=e^{-1}$ gives $\kappa=\sqrt{-(\log\epsilon)/(\alpha q)}$ and hence $q=s_{\max}^{2}s_{K}^{-2}$: the number of iterations needed to keep the first $K$ principal components, which for monotonic activations are approximately discrete-sine-transform frequency bases, so $K$ is essentially the bandwidth. The paper further claims that activation choice determines how effective this mechanism is: monotonic activations ($\eta(x)=x^{r-1}[x>0]$) correspond to spline smoothing of order $r$, their singular spectra drop steeply, and GD can select components, whereas non-monotonic activations such as sinc and Gaussian have a flat-then-steep spectrum controlled by scaling $\sigma$, so scaling rather than iterations sets their bandwidth.
Load-bearing premise
The paper's iteration-to-bandwidth formula is derived from the continuous-time (gradient-flow) approximation of gradient descent, whose mask is $1-\exp(-\alpha q s^2)$; discrete gradient descent actually has mask $1-(1-\alpha s^2)^q$, and if the 'near perfect' approximation fails at the learning rates and iteration counts used in practice, then the predicted iteration counts will not deliver the claimed number of active frequency components.
Editorial extensions
If this is right
- For monotonic activations, choosing $q=s_{\max}^{2}s_{K}^{-2}$ (with $\alpha=s_{\max}^{-2}$) selects $K$ active principal components in advance, making spectral bias a tunable design choice rather than an accident of training.
- The spline connection sets the order of implicit smoothing by activation: Heaviside gives first-order, ReLU second-order, and ReLU$^2$ third-order derivative penalties, with higher $r$ making GD regularization increasingly iteration-expensive.
- GD regularizes monotonic activations well and non-monotonic activations poorly: for sinc and Gaussian, learning rate and iteration count barely affect bandwidth, and the activation scale $\sigma$ (with $\sigma\approx K$) is the controlling hyperparameter.
- A suitably regularized monotonic network matches a discrete-sine-transform band-limited reconstruction, and a sinc network with $\sigma=K$ and a modest fixed $q$ matches that baseline almost exactly, making sinc an iteration-efficient surrogate for spectral bias.
- For $r=1$ activations (Heaviside, tanh) the iteration counts required to keep a given number of components are orders of magnitude smaller than for ReLU-family activations, directly affecting training cost.
Reading between the lines
- [Editorial inference] The relation $q=s_{\max}^{2}s_{K}^{-2}$ can be inverted into a bandwidth curriculum: increasing $q$ over training would let a monotonic network fit low frequencies first and then admit higher frequencies, a training schedule the paper does not discuss.
- [Editorial inference] The same singular-value-mask reasoning could be applied layer by layer in deep networks; measuring per-layer Jacobian spectra during training would test whether the shallow-network picture of spectral bias survives depth.
- [Editorial inference] The paper's 'grey zone' -- the slow transition of the GD mask -- predicts leakage of higher-frequency content when singular values around $s_K$ are closely spaced; the exact discrete-GD mask $1-(1-\alpha s^2)^q$ could turn that leakage into a quantitative bound and a sharper predictor than the exponential approximation.
- [Editorial inference] For non-monotonic activations, the paper's $\sigma=K$ rule suggests that adapting $\sigma$ per sample or per epoch could act as a data-dependent low-pass filter in coordinate networks, a testable design not explored in the paper.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript studies spectral bias in one-dimensional shallow networks f(x) = Σ_m w_m η(x − b_m) trained with full-batch gradient descent (GD) on a least-squares objective. Three main claims are made. First (Proposition 3.1), minimizing an r-th order gradient penalty as in spline smoothing is asserted to be equivalent to ridge regression on the weights when the activation is the truncated power function η(x) = x^{r−1}[x > 0]. Second (Propositions 4.1 and 4.2), the implicit regularization of GD is reinterpreted as an explicit shrinkage operator on the singular values of the design matrix A, with masking function m(s; α, q) = 1 − exp(−α q s²) and an explicit relationship q = s_max² s_K^{−2} between the iteration count q, the learning rate α = s_max^{−2}, and the number K of retained principal components. Third, experiments on a row of the peppers image (N = 256, M = 1024) show that tuning q controls bandwidth for monotonic activations (tanh, ReLU) in agreement with a discrete sine transform (DST) baseline, whereas non-monotonic activations (sinc, Gaussian) are insensitive to q and are instead controlled by their scaling σ, consistent with Shannon–Whittaker sampling. The paper concludes that GD-based spectral-bias control works only for monotonic activations and promotes sinc as an iteration-efficient surrogate.
Significance. The paper's strengths are its transparent SVD-based setup, the correct derivation of the exact GD iterate in Appendix B, the falsifiable experimental design that compares activation families against a DST baseline, and a genuinely useful qualitative dichotomy: monotonic activations have a steep initial singular-value falloff and respond to (α, q) tuning, whereas sinc/Gaussian have a flat spectrum up to a scale-dependent cutoff and are regularized by σ. The validation is not circular, since the masking function is derived rather than fitted, and the sinc/bandwidth connection (Eq. 19) is a nice conceptual bridge. If the quantitative claims are repaired, the paper would provide practical guidance for bandwidth selection in coordinate MLPs. However, the headline relation in Proposition 4.2 is derived from a gradient-flow approximation rather than from the exact GD mask, no error bound is given, and the experiments run a different iteration schedule from the one derived; as stated, the explicit q–K formula is therefore not established. These are local, fixable defects in an otherwise defensible central argument, so revision is appropriate.
major comments (3)
- [§4 (Prop. 4.1–4.2); Appendix B] Proposition 4.1 (Eq. 15) states the masking function as m_gd(s; α, q) = 1 − exp(−α q s²), but the exact GD iterate derived in Appendix B, Eq. 29, yields m_exact(s; α, q) = 1 − (1 − α s²)^q. Eq. 15 is the gradient-flow mask, not the GD mask, and the assertion in Appendix B that gradient flow is a 'near perfect approximation' is unquantified and fails at the top of the spectrum: with α = s_max^{−2}, αs² = 1 at the largest singular value, where the exact mask reaches 1 after one iteration while the exponential gives 1 − e^{−q}. Because Proposition 4.2 is built on Eq. 15, the headline relation q = s_max² s_K^{−2} (Eq. 17) is itself approximate. The exact threshold condition m_exact(s_K) = 1 − e^{−1} gives q = −1/log(1 − s_K²/s_max²), which differs from Eq. 17 by about 15% at s_K/s_max = 0.5 (3.48 vs 4) and about 60% at 0.8 (0.98 vs 1.56), where a sub-unit q shows that the threshold interpretation breaks down in the small-K regime that Figure 6 actually sweeps. The approximation is accurate only when s_K² ≪ s_max², and no bound or numerical comparison is supplied. Since this is the paper's central explicit contribution, Proposition 4.2 should be re-derived from the exact mask or accompanied by a quantified error estimate over the relevant range of K; the qualitative q-monotonic control of bandwidth in Figure 6 would survive either way, but the stated formula would not.
- [§6, Fig. 6(a)] Section 6 and the caption of Figure 6(a) state that the experiments set q = s_max·s_K^{−1}, whereas Proposition 4.2/Eq. 17 states q = s_max²·s_K^{−2}; these schedules differ by a square and cannot both describe what was run. The manuscript must state which schedule was used in the reconstruction experiments and reconcile the two formulas; as it stands, the claim that Figure 6(a) validates Proposition 4.2 is not verifiable. Relatedly, the statement in Section 4 that the active-window width satisfies ρ = s_max·s_K^{−1} = √q is consistent with Eq. 17 but not with the Figure 6(a) schedule, under which the same quantity would equal q rather than √q.
- [§3 (Prop. 3.1); Appendix A] Proposition 3.1 asserts an equivalence between the spline objective in Eq. 3 and the ridge-penalized basis expansion in Eq. 4, and the text promises that 'a full derivation ... can be found in Appendix A.' Appendix A in fact contains only an empirical visualization of the columns of ∇^{−r}; no derivation of the equivalence is given. The statement is plausible and can be proved via the Green's function identity d^r/dx^r (x − t)_+^{r−1} = (r−1)!·δ(x − t), but the manuscript does not provide this argument, does not account for the (r−1)! constant that is silently absorbed into λ, and does not discuss how the boundary conditions of the discrete operator ∇ (the 'same' convolution) affect the claimed equivalence. The appendix description should be corrected and the proof supplied, or the proposition should be explicitly downgraded to an empirically supported observation.
minor comments (8)
- [§6] In the first paragraph of Section 6 the SVD spectrum is attributed to Figure 4, but the spectrum figure is Figure 3; the principal-component figures are Figures 4 and 5.
- [§6] In the same section, the falloff rate is stated for 'η(x) = x^{r−1}·[x > 1]', which should read '[x > 0]' to agree with Proposition 3.1.
- [§7] In the Key Insights list, the summary relation q ≈ α^{−2}·s²_B is dimensionally inconsistent and disagrees with Eq. 17; from Eq. 17 and α = s_max^{−2} the correct form is q ≈ α^{−1}·s_B^{−2}.
- [Appendix C] Equation (36) gives ρ(ε) = √(q/(−log ε)) with ρ defined as s(ε)/s_max, but from Eq. 35 the correct expression is ρ(ε) = √(−log ε / q); the claimed ρ ∝ √q follows only under the main-text definition ρ = s_max/s(ε), so the two definitions should be aligned.
- [§1, References] There are several naming and reference slips: 'Sitzman' should be 'Sitzmann', 'Whitaker' should be 'Whittaker', 'adhoc' should be 'ad hoc', 'explicitly effect spectral bias' should be 'explicitly affect spectral bias', reference [13] lacks a year, and reference [26] lacks its publication venue.
- [§4] The caption of Figure 2 says α = 1 'ensures the upper singular value of the mask is normalized at unity', but m(1; 1, q) = 1 − exp(−q), which equals unity only as q → ∞; the caption should be reworded.
- [§4] Proposition 4.2 is stated with the specific threshold ε = exp(−1), but any ε ∈ (0,1) is allowed in Eq. 16, so the 'explicit' q–K relation is determined only up to the constant factor −log ε; since the experiments use a different q schedule, they cannot pin down this constant, and the agreement with the DST baseline in Figure 6 should be described as qualitative.
- [§4] The derivation in Appendix B restates standard results on implicit regularization of GD and early stopping for least squares; the paper should cite that literature (e.g., Ali, Kolter, and Tibshirani (2019) and Yao, Rosasco, and Caponnetto (2007)) and soften the claim of being 'the first, to our knowledge, to make explicit such a relationship' accordingly.
Circularity Check
No significant circularity: the q–K relation is derived from the gradient-flow mask, not fitted, and the self-citations are background, not load-bearing.
full rationale
The central derivation is self-contained. Proposition 4.1's masking function m_gd(s;α,q)=1-exp(-α q s^2) is derived in Appendix B from the Neumann-series / gradient-flow solution of linear least squares, not assumed as the conclusion. Proposition 4.2 then inverts the threshold condition m_gd(s_K;α,q)=1-e^{-1} to obtain q=s_max^2 s_K^{-2}; the threshold ε=exp(-1) is explicitly labeled an arbitrary simplification, not fitted to reconstruction data. The experiments vary q or σ and compare with a DST baseline, making them consistency checks of the interpretation rather than predictions of fitted parameters. Self-citations [14,16,27,28] support background claims about Gaussian/sinc activations and do not carry the load of the GD-as-shrinkage derivation. No uniqueness theorem or ansatz is imported from prior work by the same author. Two non-circular caveats are worth flagging. First, Appendix B switches from the exact GD mask 1-(1-α s^2)^q to the gradient-flow mask 1-exp(-α q s^2) calling it a 'near perfect approximation' without a quantitative bound; this is a correctness risk, especially for small K, but not a circular reduction. Second, Fig. 6(a)'s caption writes q=s_max s_K^{-1}, which is inconsistent with Eq. 17's q=s_max^2 s_K^{-2}; this is an internal-consistency issue, not evidence that the theoretical result is defined in terms of the experimental output. No circular step meets the standard of exhibiting Eq. X = Eq. Y by construction or a fitted parameter renamed as a prediction.
Assumptions & free parameters
free parameters (4)
- mask threshold epsilon =
exp(-1) (approx 0.3679)
- learning rate alpha =
s_max^{-2}
- activation scaling sigma =
15, 30, 1024, or sigma = M / sigma = K depending on experiment
- width M and sample count N =
M=N=5000 for spectra, M=1024 and N=256 for reconstruction, M=50 for finite width
assumptions (5)
- domain assumption Gradient flow accurately approximates discrete gradient descent for the least-squares objective (Eq. 15 vs Eq. 29)
- domain assumption The activation matrix A = [eta(x_i - b_j)] captures the network's Jacobian and its SVD basis aligns with frequency for monotonic activations
- ad hoc to paper Spline smoothing equivalence in Proposition 3.1: minimizing the r-th order gradient penalty is equivalent to ridge penalty with activation x^{r-1} [x>0]
- domain assumption Squared loss and zero initialization for GD
- standard math Shannon-Whittaker sampling theorem applies as guidance for bias spacing in sinc networks
Cite this review
Pith. "Pith review of Gradient Descent as a Shrinkage Operator for Spectral Bias." pith.science (2026). https://pith.science/paper/AFQPW437
@misc{pith2026250418207,
author = {Pith},
title = {Pith review of: Gradient Descent as a Shrinkage Operator for Spectral Bias},
year = {2026},
howpublished = {\url{https://pith.science/paper/AFQPW437}},
note = {Machine review of arXiv:2504.18207}
}
read the original abstract
We generalize the connection between activation function and spline regression/smoothing and characterize how this choice may influence spectral bias within a 1D shallow network. We then demonstrate how gradient descent (GD) can be reinterpreted as a shrinkage operator that masks the singular values of a neural network's Jacobian. Viewed this way, GD implicitly selects the number of frequency components to retain, thereby controlling the spectral bias. An explicit relationship is proposed between the choice of GD hyperparameters (learning rate & number of iterations) and bandwidth (the number of active components). GD regularization is shown to be effective only with monotonic activation functions. Finally, we highlight the utility of non-monotonic activation functions (sinc, Gaussian) as iteration-efficient surrogates for spectral bias.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Stability and generalization of learning algorithms that converge to global 8 optima
Zachary Charles and Dimitris Papailiopoulos. Stability and generalization of learning algorithms that converge to global 8 optima. In International conference on machine learning , pages 745–754. PMLR, 2018. 2
work page 2018
-
[2]
Approximation by superpositions of a sig- moidal function
George Cybenko. Approximation by superpositions of a sig- moidal function. Mathematics of control, signals and sys- tems, 2(4):303–314, 1989. 5
work page 1989
-
[3]
Gecynalda S. da S. Gomes, Teresa B. Ludermir, and Lean- dro M. Almeida. Neural networks with asymmetric activa- tion function for function approximation. In 2009 Interna- tional Joint Conference on Neural Networks, pages 980–987,
work page 2009
-
[4]
Spectral bias in practice: The role of function fre- quency in generalization
Sara Fridovich-Keil, Raphael Gontijo Lopes, and Rebecca Roelofs. Spectral bias in practice: The role of function fre- quency in generalization. Advances in Neural Information Processing Systems, 35:7368–7382, 2022. 1
work page 2022
-
[5]
Optimal shrinkage of singular values
Matan Gavish and David L Donoho. Optimal shrinkage of singular values. IEEE Transactions on Information Theory, 63(4):2137–2152, 2017. 3
work page 2017
-
[6]
Stochastic training is not nec- essary for generalization
Jonas Geiping, Micah Goldblum, Phil Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not nec- essary for generalization. In International Conference on Learning Representations, 2022. 2
work page 2022
-
[7]
Train faster, generalize better: Stability of stochastic gradient descent
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pages 1225–
-
[8]
Jakob Heiss, Josef Teichmann, and Hanna Wutte. How im- plicit regularization of relu neural networks characterizes the learned function–part i: the 1-d case of two layers with ran- dom first layer. arXiv preprint arXiv:1911.02903, 2019. 2
arXiv 1911
Show all 28 references
-
[9]
Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural in- formation processing systems, 30, 2017. 2
2017
-
[10]
Neu- ral tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Cl ´ement Hongler. Neu- ral tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing sys- tems, 31, 2018. 2, 3, 4, 11
2018
-
[11]
A note on the use of principal components in regression
Ian T Jolliffe. A note on the use of principal components in regression. Journal of the Royal Statistical Society Series C: Applied Statistics, 31(3):300–303, 1982. 3, 5
1982
-
[12]
Imagenet classification with deep convolutional neural net- works
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. 25, 2012. 2
2012
-
[13]
Beyond lipschitz: Sharp generalization and excess risk bounds for full-batch gd
Konstantinos Nikolakakis, Farzin Haddadpour, Amin Kar- basi, and Dionysios Kalogerias. Beyond lipschitz: Sharp generalization and excess risk bounds for full-batch gd. In The Eleventh International Conference on Learning Repre- sentations. 2
-
[14]
Beyond periodicity: Towards a unifying framework for activations in coordinate- mlps
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for activations in coordinate- mlps. In European Conference on Computer Vision , pages 142–158. Springer, 2022. 2, 5
2022
-
[15]
Smoothing by spline functions
Christian H Reinsch. Smoothing by spline functions. Nu- merische mathematik, 10(3):177–183, 1967. 2
1967
-
[16]
A sam- pling theory perspective on activations for implicit neural representations
Hemanth Saratchandran, Sameera Ramasinghe, Violetta Shevchenko, Alexander Long, and Simon Lucey. A sam- pling theory perspective on activations for implicit neural representations. In Forty-first International Conference on Machine Learning. 2, 5
-
[17]
Communication in the presence of noise
Claude Elwood Shannon. Communication in the presence of noise. Proceedings of the IRE, 37(1):10–21, 1949. 5
1949
-
[18]
Implicit neural representa- tions with periodic activation functions
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representa- tions with periodic activation functions. Advances in neural information processing systems, 33:7462–7473, 2020. 2, 5
2020
-
[19]
The implicit bias of gradi- ent descent on separable data
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradi- ent descent on separable data. Journal of Machine Learning Research, 19(70):1–57, 2018. 2
2018
-
[20]
Fourier features let networks learn high frequency functions in low dimen- sional domains
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ra- mamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimen- sional domains. 33:7537–7547, 2020. 2
2020
-
[21]
Sampling-50 years after shannon
Michael Unser. Sampling-50 years after shannon. Proceed- ings of the IEEE, 88(4):569–587, 2000. 5
2000
-
[22]
Principles of risk minimization for learn- ing theory
Vladimir Vapnik. Principles of risk minimization for learn- ing theory. Advances in neural information processing sys- tems, 4, 1991. 2
1991
-
[23]
The “fourier” theory of the cardinal func- tion
John M Whittaker. The “fourier” theory of the cardinal func- tion. Proceedings of the Edinburgh Mathematical Society, 1 (3):169–176, 1928. 5
1928
-
[24]
Gradient dynamics of shallow univariate relu networks
Francis Williams, Matthew Trager, Daniele Panozzo, Clau- dio Silva, Denis Zorin, and Joan Bruna. Gradient dynamics of shallow univariate relu networks. Advances in neural in- formation processing systems, 32, 2019. 2
2019
-
[25]
A mathematical analysis of the motion coherence theory
Alan L Yuille and Norberto M Grzywacz. A mathematical analysis of the motion coherence theory. International Jour- nal of Computer Vision, 3:155–175, 1989. 2, 5
1989
-
[26]
Understanding deep learning re- quires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning re- quires rethinking generalization. 2017. 2, 5, 12
2017
-
[27]
Rethinking positional encoding
Jianqiao Zheng, Sameera Ramasinghe, and Simon Lucey. Rethinking positional encoding. arXiv preprint arXiv:2107.02561, 2021. 5
2021 arXiv
-
[28]
Trading positional complexity vs deepness in coordinate networks
Jianqiao Zheng, Sameera Ramasinghe, Xueqian Li, and Si- mon Lucey. Trading positional complexity vs deepness in coordinate networks. In European Conference on Computer Vision, pages 144–160. Springer, 2022. 5 9 Gradient Descent as a Shrinkage Operator for Spectral Bias Supplem...
2022
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.