REVIEW 2 major objections 4 minor 23 references
Revisiting Glorot Initialization for Long-Range Linear Recurrences
T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Standard Glorot initialization is unstable for linear RNNs: the spectral radius overshoots 1 by O(1/√n), so sequences of length Θ(√n) suffice to make hidden states explode, and a variance rescaling restores stability.
desk verdict The rescaling is practical and Theorem 6.1 is correct, but the headline Θ(√n) explosion threshold is unsupported because Eq. (20)'s asymptotic is wrong in the regime they claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument stands on three objects. First is the spectral-edge law for non-Hermitian Gaussian matrices (Rider; Rider–Sinclair): the spectral radius of a Glorot matrix satisfies √(4ρ_n n)(|λ_max| − 1 − √(ρ_n/4n)) ⇒ G, a Gumbel variate, so the radius exceeds one by Θ(1/√n). Second is a finite-width variance lower bound, E[||W^k x||²] ≥ (1/n^k)(n/(k+1))((n+k)!/n!), proved from the Ginibre eigenvalue density via Hutchinson's trace trick and the Schur decomposition. Third is the double-scaling asymptotic approximation that turns the summed bound into (n/t)exp(t²/2n) when t,n → ∞ with t/√n fixed; this is the step that converts the small spectral-radius overshoot into a concrete sequence-length threshold. The rescaling is then derived by inverting the Gumbel formula to pick a variance that sits one standard deviation below the typical radius.
What would settle it
Compute, for complex Glorot matrices of increasing width n, the normalized hidden-state variance (1/n) E||h_t||² at t = ⌊√n⌋ with i.i.d. Gaussian inputs. If this quantity saturates or grows only logarithmically rather than diverging as n grows, the Θ(√n) explosion threshold is not correct.
Extended reading notes
Core claim
The central claim is that Glorot-initialized complex Gaussian matrices, when iterated in a linear recurrence, drive hidden-state variance to explode once the sequence length reaches the square root of the hidden width. Concretely, for W with i.i.d. complex entries of variance 1/n, Theorem 6.1 proves E[||W^k x||^2] ≥ (1/n^k)(n/(k+1))((n+k)!/n!) for i.i.d. standard Gaussian inputs; in the limit n,k → ∞ with k = α√n this lower bound tends to exp(α²/2). Summing over time steps then yields a normalized variance lower bound on the order of (n/t)exp(t²/2n), which the paper argues diverges when t = Θ(√n). The paper attributes the mechanism to the spectral edge: the largest eigenvalue modulus of a Glorot matrix exceeds one by O(1/√n), and the size of that deviation is governed by Gumbel fluctuations. The proposed correction multiplies the entry variance by (1 + $\sqrt$(ρ_n/4n) + a_p/$\sqrt$(4ρ_n n))^{−2}, with a_p chosen one Gumbel standard deviation above the mean, giving roughly an 86% chance that the spectral radius lies inside the unit disk.
Load-bearing premise
The proof's explosion threshold rests on the approximation that the summed variance lower bound behaves like (n/t)exp(t²/2n) at t = Θ(√n) under uncorrelated Gaussian inputs; if that approximation is inaccurate, the claimed √n threshold does not follow.
Editorial extensions
If this is right
- Linear RNNs and LRU-style models initialized with standard Glorot should be expected to blow up on long-context tasks, so practitioners should treat Glorot as a non-starter for long sequences rather than a neutral baseline.
- The proposed rescaling gives a drop-in, theory-backed replacement that keeps the spectral radius below one with about 86% probability, and in the reported experiments it outperforms the Glorot/2 and unit-circle diagonal baselines on LRA tasks.
- The Θ(√n) threshold means instability is not a rare-event concern: it appears as soon as sequence length scales with the square root of width, which is a typical operating point for modest-width RNNs on 1K–8K token tasks.
- Because the mechanism is the spectral edge rather than the bulk spectrum, the circular-law intuition that Glorot keeps the spectrum inside the unit disk is misleading for recurrent use.
Reading between the lines
- Beyond the paper: the same edge-of-the-spectrum mechanism likely applies to any architecture that repeatedly applies a shared, randomly initialized matrix, such as deep weight-tied networks, not just linear RNNs.
- Beyond the paper: if the Θ(√n) threshold is read as t ≈ α√n, the onset of instability should depend on the Gumbel quantile α, giving a testable scaling prediction that wider networks tolerate proportionally longer sequences before exploding.
- Beyond the paper: the analysis uses i.i.d. Gaussian inputs; with real, correlated, bounded inputs the explosion may be delayed or accelerated, so an empirical check on natural-language sequences with controlled spectral radius would separate the input-statistics effect from the matrix edge effect.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Bar, Seleznova, Alexander, Kutyniok, and Giryes revisit Glorot initialization for linear recurrent networks. They prove a finite-width lower bound on the expected squared norm of W^k x for complex Gaussian (Glorot) W, and combine it with edge-of-spectrum results from random matrix theory to claim that hidden-state norms explode when the sequence length is Θ(√n) in the double-scaling limit. They then propose a variance rescaling that uses the Gumbel edge distribution to put the spectral radius below one with probability about 0.86, and they evaluate this rescaling on three Long Range Arena tasks, where standard Glorot fails to train.
Significance. The paper has two clear strengths. First, Theorem 6.1 is a clean finite-width lower bound derived from the Ginibre density, and the proof is elementary and checkable. Second, the proposed rescaling is simple, theoretically motivated by independent results of Rider and Rider–Sinclair rather than by fitting the benchmarks, and the experiments show a practical improvement over Glorot/2 and a circular-law baseline. However, the headline quantitative claim — that O(√n) sequence length induces explosion — rests on the approximate summation in Eq. (20), which is incorrect in the stated regime. The contribution is therefore significant only after that step is repaired and the advertised threshold is restated.
major comments (2)
- [§6.1.2, Eq. (20)] The step replacing the sum by (n/t) exp(t^2/2n) is not valid when t = Θ(√n), which is exactly the regime used for the Abstract's central claim. For t = α√n, set k = x√n; the left-hand side of Eq. (20) behaves as 1 + ∫_{1/√n}^{α} exp(x^2/2)/x dx = (1/2) log n + C(α) + o(1), whereas the right-hand side is (√n/α) exp(α^2/2), which is larger by a factor of order √n/log n. Theorem 6.1's lower bound therefore gives only logarithmic growth of the normalized hidden-state variance at t = α√n, not the Θ(√n) explosion claimed in the Abstract and Section 1. The paper should replace Eq. (20) with the correct asymptotic and state the explosion threshold as t/√n → ∞, or otherwise justify a different definition of 'explosion'.
- [Abstract/Section 1 vs. §4] The paper advertises two incompatible thresholds for the same phenomenon: the Abstract and Section 1 say t = O(√n) is sufficient, while Section 4 ('Connection to the double-scaling limit') says t = Θ(√{n^{1+ε}}) for any ε > 0. These scalings are not equivalent, since √{n^{1+ε}} = n^{(1+ε)/2} is much larger than √n for ε > 0. The paper must choose a single threshold and prove it; the current inconsistent presentation is part of the load-bearing quantitative claim.
minor comments (4)
- [§6.1.1, Eq. (16)] The harmonic asymptotics should be H_t = log t + γ + 1/(2t) + O(t^{-2}), not log(t−1) as written, or the index range should be adjusted; this is cosmetic but confusing.
- [§5] The choice of a_p = γ − δ_R ln 2 + π/√6 is a heuristic (mean plus one standard deviation of the Gumbel); since the paper presents this as a baseline, a sentence on sensitivity to a_p would be useful.
- [Table 1] The '✗' entries should state explicitly whether training diverged (NaN) or merely failed to reach the baseline, since that affects the interpretation of the table.
- [Figures 1 and 2] The rightmost panels show hidden-state norms ∥h_k∥ but the horizontal axis is labeled k; for clarity, label the time index consistently with Section 3.
Circularity Check
No significant circularity: the derivation rests on independent random-matrix-theory results, and the rescaling is a parameter-derived construction rather than a fitted prediction.
full rationale
The central claims are derived from external random matrix theory: the spectral edge behavior comes from Rider (2003) and Rider and Sinclair (2014), and the finite-width eigenvalue moment bound comes from the Ginibre ensemble via the density given in Theorem 6.1 and its proof. The proposed rescaling is constructed directly from the Gumbel edge law with a fixed quantile choice, ap = gamma - delta_R ln 2 + pi/sqrt(6), and is not tuned to the benchmark results or to any subset of the experimental data. The self-citations in the paper, such as Seleznova and Kutyniok (2022) and Razin et al. (2024), appear in contextual comparisons about double-scaling limits and are not load-bearing premises for the instability theorem or for the rescaling formula. The numerical experiments on LRA tasks are an external check on the proposed initialization rather than an input used to define it, and the paper's own limitation statement explicitly acknowledges that Theorem 6.1 provides only a lower bound under uncorrelated Gaussian inputs. The reader's flagged concern about Eq. (20), namely that the sum approximation fails at t = Theta(sqrt(n)), is a mathematical-validity issue rather than a circularity issue: if that asymptotic were valid, the claimed instability threshold would still be an independent derived consequence of the external random-matrix-theory inputs and the finite-width variance bound. No step in the derivation reduces by construction to its own input, no fitted parameter is renamed as a prediction, and no load-bearing uniqueness claim is imported from the authors' own prior work. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- a_p (Gumbel quantile offset) =
γ - δ_R ln 2 + π/√6, giving p≈0.86
assumptions (4)
- standard math Rider's edge scaling for spectral radius of complex/real Ginibre matrices (Theorem 4.1)
- standard math Ginibre eigenvalue density for complex Gaussian matrices
- domain assumption Inputs x_t are i.i.d. standard complex Gaussian and h_0 = 0
- domain assumption The double-scaling regime with t/√n → α is the relevant setting for recurrence
Cite this review
Pith. "Pith review of Revisiting Glorot Initialization for Long-Range Linear Recurrences." pith.science (2026). https://pith.science/paper/RS2HDDO3
@misc{pith2026250519827,
author = {Pith},
title = {Pith review of: Revisiting Glorot Initialization for Long-Range Linear Recurrences},
year = {2026},
howpublished = {\url{https://pith.science/paper/RS2HDDO3}},
note = {Machine review of arXiv:2505.19827}
}
abstract
Proper initialization is critical for Recurrent Neural Networks (RNNs), particularly in long-range reasoning tasks, where repeated application of the same weight matrix can cause vanishing or exploding signals. A common baseline for linear recurrences is Glorot initialization, designed to ensure stable signal propagation--but derived under the infinite-width, fixed-length regime--an unrealistic setting for RNNs processing long sequences. In this work, we show that Glorot initialization is in fact unstable: small positive deviations in the spectral radius are amplified through time and cause the hidden state to explode. Our theoretical analysis demonstrates that sequences of length $t = O(\sqrt{n})$, where $n$ is the hidden width, are sufficient to induce instability. To address this, we propose a simple, dimension-aware rescaling of Glorot that shifts the spectral radius slightly below one, preventing rapid signal explosion or decay. These results suggest that standard initialization schemes may break down in the long-sequence regime, motivating a separate line of theory for stable recurrent initialization.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Plugging this back in: E[|λi|2k] = 1 nk+1 n−1X s=0 (k + s)! s! = 1 nk · 1 k + 1 · (k + n)! n!
= π(k + s)! nk+s . Plugging this back in: E[|λi|2k] = 1 nk+1 n−1X s=0 (k + s)! s! = 1 nk · 1 k + 1 · (k + n)! n! . Finally, summing over all i and plugging into the lower bound, we obtain: E[Tr((Wk)∗Wk)] ≥ 1 nk · n k + 1 · (n + k)! n! , which completes the proof. 11.2 Real Glorot Initialization While our discussion of the spectral radius in Section 4 and ...
work page 2023
-
[2]
As in the real case discussed in the main text, complex Glorot initialization leads to an explosion in the hidden state norm, while our proposed rescaling effectively mitigates this. Eigenvalues Radius Matrix Powers Hidden States Glorot 0.0 0.2 0.4 0.6 0.8 1.0 Radius (magnitude of eigenvalues) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75Density 0 25 50 75 100 ...
work page 2023
-
[10]
B. Hanin and M. Nica. Finite depth and width corrections to the neural tangent kernel. arXiv preprint arXiv:1909.05989,
arXiv 1909
-
[13]
URL http://arxiv.org/abs/1504.00941. M. Li, M. Nica, and D. Roy. The future is log-gaussian: Resnets and their infinite-depth-and-width limit at initialization. Advances in Neural Information Processing Systems, 34:7852–7864,
-
[14]
T. Mikolov, A. Joulin, S. Chopra, M. Mathieu, and M. Ranzato. Learning longer memory in recurrent neural networks. arXiv preprint arXiv:1412.7753,
- [15]
-
[16]
S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep information propagation. arXiv preprint arXiv:1611.01232,
- [19]
Show all 23 references
-
[20]
URL https://arxiv.org/abs/2405.21064. N. Zucchet, R. Meier, and S. Schug. Minimal implementation of linear recurrent units,
-
[22]
Table 2: Hyperparameters used for each dataset. Hyperparameter CIFAR-10 IMDB ListOps Warmup End 18 7 5 D 512 192 256 H 384 256 192 Batch Size 50 32 32 Epochs 180 65 50 LR Factor 0.025 0.025 0.05 Dropout Probability 0.05 0.05 0 13 Compute resources and training time. All experi...
1965
-
[24]
Proposition 11.2 (Complex-eigenvalue density [Edelman, 1997, Thm
+ (√nλ)n−1e −nλ2 2 Γ( n 2 )2 n 2 γ( (n−1) 2 , nλ2 2 ) Γ( n−1 2 ) , where Γ(s, z) = R ∞ z ts−1e−t dt is the upper-incomplete gamma function, γ(s, x) = R x 0 t s−1 e−tdt is the lower incomplete gamma function, and we use the shorthand Γ(s) := Γ( s, 0), and C real n = O(√n) is a ...
1997
-
[1994]
Biegun, R
K. Biegun, R. Dolga, J. Cunningham, and D. Barber. Rotrnn: Modelling long sequences with rotations. arXiv preprint arXiv:2407.07239,
-
[1997]
doi: 10.1006/jmva.1996.1653
ISSN 0047-259X. doi: 10.1006/jmva.1996.1653. URL https://doi.org/10.1006/jmva.1996.1653. A. Edelman, E. Kostlan, and M. Shub. How many eigenvalues of a random matrix are real. Journal of the American Mathematical Society, 7:247–267,
1996
-
[2001]
Gilboa, B
D. Gilboa, B. Chang, M. Chen, G. Yang, S. S. Schoenholz, E. H. Chi, and J. Pennington. Dynamical isometry and a mean field theory of lstms and grus. arXiv preprint arXiv:1901.08987,
1901 arXiv
-
[2010]
Gu and T
10 A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,
-
[2014]
S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y . Chen, S. Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427,
-
[2015]
Henaff, A
M. Henaff, A. Szlam, and Y . LeCun. Recurrent orthogonal networks and long-memory tasks. In International Conference on Machine Learning, pages 2034–2042. PMLR,
-
[2018]
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078,
-
[2019]
Y . Tay, M. Dehghani, S. Abnar, Y . Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006,
2011 arXiv
-
[2020]
A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396,
-
[2021]
J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450,
-
[2022]
Sundermeyer, R
M. Sundermeyer, R. Schlüter, and H. Ney. Lstm neural networks for language modeling. In Interspeech, volume 2012, pages 194–197,
2012
-
[2024]
Byun and P
S.-S. Byun and P. J. Forrester. Progress on the study of the ginibre ensembles i: Ginue.arXiv preprint arXiv:2211.16223,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.