Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

On the Stability of the Jacobian Matrix in Deep Neural Networks

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper establishes a general stability theorem: for a deep ReLU network, the spectral norm of the input–output Jacobian converges, as width grows, to the norm of a product of free semicircular elements, and this universal limit is…

desk verdict A solid, clean application of recent free-probability results to Jacobian stability, with genuinely useful scaling prescriptions; the main caveat is that the central theorem is proved for exogenous ReLU signs, and the bridge to real networks is only verified for i.i.d. Gaussian weights. read the letter →

arxiv 2506.08764 v3 pith:QZTGQLTI submitted 2025-06-10 cs.LG

Benjamin Dadoun , Soufiane Hayou , Hanan Salam , Mohamed El Amine Seddik , Pierre Youssef This is my paper · ORCID
classification cs.LG MSC 60B2015B5246L5468T07
keywords Jacobianspectralnormdeepneuralnetworksexplodingandvanishinggradientsrandommatrixtheoryfreeprobabilitynetworkpruningweightcorrelationedgeofchaos
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that Jacobian stability—the spectral norm of the input–output Jacobian growing only sub-exponentially with depth—holds far beyond the textbook setting of i.i.d. Gaussian weights. Its central theorem shows that, under a fair-coin model of ReLU sign patterns, the Jacobian norm of an MLP converges as width grows to the norm of a product of free semicircular elements, a universal quantity independent of the weight distribution. From this it derives concrete scaling prescriptions for pruned networks: random pruning needs a $(1-s)^{-1/2}$ rescaling of the surviving weights, while magnitude-based pruning needs a different, threshold-dependent scaling; in both cases the pruned network's Jacobian norm matches the critically initialized i.i.d. network. For correlated Gaussian weights, the same limit holds as long as correlations decay faster than $n^{-3/2}/\log n$. If correct, this extends edge-of-chaos initialization theory to the structured and dependent weight distributions that arise in pruning, fine-tuning, and data-dependent initialization.

What carries the argument

The engine is strong asymptotic freeness for products of structured random matrices. The Jacobian is expressed through trace moments of a symmetrized matrix $H = \begin{pmatrix} 0 & M \\ M^* & 0 \end{pmatrix}$ with $M = B \odot W D$; the theorem verifies the conditions under which trace moments of powers of $H$ converge to those of a freely independent product of semicircular elements $s_k \cdots s_L$. This free-probability limit is the universal baseline against which all the pruning and correlation results are compared.

What would settle it

Take a randomly pruned MLP (e.g., width 256, sparsity 0.5, mask scaling $(1-s)^{-1/2}$) and test the joint law of the pre-activation sign diagonals $D_l$ against the weights and mask: if a chi-square test of independence between entries of $D_l$ and mask or weight values, or between pairs of entries of $D_l$, rejects at the asymptotic level as width grows, or if the empirical distribution of signs visibly departs from Bernoulli(1/2), then Approximation 2.1 fails and the universal limit in Theorem 3.1 loses its justification. A direct quantitative check is to compute $\|J^B_1\|$ for increasing widths at fixed depth and compare the values to the predicted free-probability limit $\|s_1 \cdots s_L\|$; a systematic gap that does not close as $n$ grows would indicate non-universal behavior.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 3.1: under Assumptions (A1)–(A3), which require subgaussian weights, bounded masks, and ReLU sign diagonals that are i.i.d. Bernoulli(1/2) independent of weights and masks, together with conditions on the masked weights (vanishing means, row-wise second moments converging to 1, and a vanishing mask-size-to-width ratio), the spectral norm of the Jacobian $J^B_k = \prod_{l=k}^L (B_l \odot W_l) D_{l-1}$ converges as $n \to \infty$ to $\|s_k \cdots s_L\|$, the norm of a product of freely independent semicircular elements. A Gaussian variant under Assumption (A$'$1) yields the same convergence in expectation for centered Gaussian entries with variance $2/n$ and covariances $O(n^{-5/2}/\log n)$. The point is universality: the limit does not depend on the fine structure of the weights or masks, so stability of the Jacobian is preserved whenever the masked weights are close enough to critical i.i.d. weights.

Load-bearing premise

Everything rests on the assumption that the ReLU sign pattern in each layer is an independent fair coin that is independent of the weights and the pruning mask—a fact verified empirically only for i.i.d. Gaussian weights, and potentially false for sparse or correlated weights.

Editorial extensions

If this is right

  • Randomly pruned networks: with mask entries $(1-s_n)^{-1/2} \mathrm{Bernoulli}(1-s_n)$, the Jacobian norm has the same limit as the i.i.d. critical network, provided $1-s_n \gg \log^4 n / n$; without rescaling, the norm grows exponentially in depth.
  • Magnitude-pruned networks: keeping weights above a threshold $t$ yields stability after rescaling by $\left(\sqrt{2\pi}\, e^{n t^2/2} / (t n^{3/2})\right)^{1/2}$ (Theorem 4.2), and keeping the $r = \lceil n \log^c n\rceil$ largest weights needs the random-threshold variant (Theorem 4.3) with $c > 7/4$.
  • Correlated Gaussian weights with $\mathrm{corr}(W_{ij}, W_{ml}) = O(n^{-3/2}/\log n)$ for distinct entries give the same expected Jacobian norm limit as i.i.d. weights, so mild training-induced correlations do not break stability.
  • An edge of stability in sparsity appears near $1-s_n \sim n^{-1}$ up to logarithmic factors; empirical results show the scaling restores trainability of pruned MLPs on Fashion-MNIST.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the universality theorem transfers to convolutional or residual architectures with the same sign-independence, the scaling prescriptions here give a principled way to renormalize weights after structured pruning, not just unstructured MLP pruning.
  • The $O(n^{-3/2}/\log n)$ correlation threshold gives a quantitative target for how much shared noise a fine-tuning or meta-learning procedure can inject before gradient propagation degrades; this could be tested by measuring Jacobian norms of networks fine-tuned from a common pretrained initialization with increasing shared-noise components.
  • For magnitude pruning, using the random-pruning scaling $(1-s)^{-1/2}$ is provably suboptimal: it leaves the Jacobian unstable in the infinite-width limit, as the paper's numerical comparison of scaling factors illustrates, so pruning algorithms should report their scaling rule along with the mask.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies the spectral norm of the input-output Jacobian of deep MLPs with ReLU activations, aiming to extend classical critical-initialization results beyond i.i.d. Gaussian weights. The central result (Theorem 3.1) is a stability theorem for products of masked random weight matrices, proved by invoking strong asymptotic freeness results of Brailovskaya--van Handel. The theorem assumes that the ReLU sign matrices D_1,...,D_L are exogenous i.i.d. Bernoulli(1/2) matrices independent of the weights and masks. The paper then applies this theorem to randomly pruned networks, magnitude-pruned networks with deterministic or random thresholds, and Gaussian weights with weak correlations, deriving explicit scaling rules that make the Jacobian norm asymptotically equal to that of a critically initialized i.i.d. network. The theoretical results are complemented by numerical experiments on pruning, training, and correlation strength.

Significance. If the main results were fully established, the paper would extend the initialization-theory toolbox to two practically relevant settings: pruned networks and weakly correlated weights. The proof strategy is modern and the paper makes no attempt to fit constants: all scaling factors are derived from moment conditions, and the theory is benchmarked against external random matrix theory results. The random-pruning scaling and the correlated-Gaussian threshold are crisp, falsifiable predictions. However, the central theorem is conditional on a decoupling approximation (Approximation 2.1) that is verified empirically only in the i.i.d. Gaussian case, and the magnitude-pruning scaling contains a substantial algebraic error. These issues currently prevent the paper from delivering its advertised 'rigorous guarantees' for the actual Jacobian of pruned or correlated networks.

major comments (3)
  1. [Section 3, Approximation 2.1, and Sections 4--5] The proof of Theorem 3.1 requires the sign matrices D_1,...,D_L to be independent of the weights and masks (Assumptions A2--A3). In the actual network, however, D_k is a deterministic function of W_k, B_k, and the previous activations. The paper's only bridge from the theorem's model to the real Jacobian is Approximation 2.1, but Section 6.1 verifies this approximation only for i.i.d. Gaussian weights, through scatter plots and a single pairwise chi-square test on entries of D_l; it does not test dependence between D_k and (W_k,B_k), nor does it cover sparse or correlated weights. For magnitude pruning (Theorems 4.2 and 4.3), B is a threshold function of |W|, and the surviving weights are truncated Gaussians, so the pre-activations are not Gaussian and no argument is given for the independent-coin model even in distribution. Consequently the equalities lim_n ||J^B_1|| = lim_n ||J^iid_1|| claimed in Theorems 4.1, 4.2, 4.3, and 5.1 are not established for the actual Jacobian; they hold only for a surrogate model with exogenous sign matrices. This is the central gap of the paper and should be addressed directly, either by proving the decoupling in each new setting or by clearly rephrasing the theorems as results for the surrogate model and providing explicit empirical validation for pruned and correlated networks.
  2. [Proof of Theorem 4.2] The mask scaling in Theorem 4.2 appears to be incorrect. The proof claims that E|w_{i,k}|^2 1_{|w_{i,k}|>t} ~ e^{-nt^2/2} sqrt(2nt^2)/pi, but this expression is not the correct asymptotic. With w ~ N(0,2/n) and N ~ N(0,1), we have w^2 = (2/n) N^2, so the event |w|>t is N^2 > nt^2/2, not N^2 > nt^2, and E|w|^2 1_{|w|>t} = (2/n) E[N^2 1_{N^2>nt^2/2}] ~ (2t/sqrt(n pi)) e^{-nt^2/4}. Using the mask b defined in the theorem, one obtains E|b w|^2 ~ 2 sqrt(2) e^{nt^2/4}/n^2. Under the theorem's scaling condition nt^2 <= 4 log n - c log log n, this tends to 0, not to 2/n. Condition (ii) of Theorem 3.1 therefore fails, and the claimed equality with ||J^iid_1|| is not justified. The constant in the mask must be rederived; with a corrected constant satisfying condition (ii), the proof of condition (i) will also need to be revisited, as the present c>7 may no longer suffice.
  3. [Theorem 3.1, condition (ii')] There is an inconsistency between the statement of Theorem 3.1 in the Gaussian case and its proof. The theorem states condition (ii') as |(2/n) sum_{k=1}^n |b_{i,k}|^2 - 1| -> 0, but the proof of the Gaussian case bounds ||E[H^2|D] - 1|| using the term |(1/n) sum_k |b_{i,k}|^2 - 1|, not the factor 2/n. For the unpruned case B=1, the stated condition gives |2-1| = 1, so Theorem 3.1 as stated does not apply to the setting of Theorem 5.1. This appears to be a typo, but since Theorem 5.1 depends on it, the statement must be corrected and the proof aligned with the corrected condition.
minor comments (4)
  1. [Section 3, notation] The Jacobian product J^B_k uses D_{l-1} for l=k,...,L, but Assumptions (A2) define D_1,...,D_L; the indexing of D_0 is never specified. Please clarify whether D_0 is another independent copy or D_1.
  2. [Section 4.1, page 9] The numerical claim 'log^10(256)/256 ~ 0.009' is unclear; log^4(256)/256 is about 3.69, so the stated asymptotic condition 1-s_n >> log^4(n)/n is not satisfied at the experimental width n=256. Please check the displayed threshold and its relation to the phase transition shown in Figure 2.
  3. [Section 6.1] The independence verification is limited to two randomly selected entries of D_l and to scalar aggregates T_W and T_D. This does not establish independence of the full matrix D_l from W_l, nor does it cover the sparse or correlated settings used in later theorems. The text should state this limitation explicitly.
  4. [Section 7, Conclusion] The conclusion states that 'stability at initialization implies stability throughout training (under some approximations)', but the paper only analyzes the Jacobian at initialization. This claim is not supported by the results and should be removed or substantially qualified.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular reduction: scaling factors are derived from moment conditions via external RMT; the main gap is an unverified modeling assumption (Approximation 2.1) for non-iid weights, not a self-referential derivation.

full rationale

No load-bearing circular step is exhibited. Theorem 3.1 is an application of external random-matrix universality results ([BvH24, Thm 3.29]; [BBvH23, Thm 2.10]) after checking explicit moment and boundedness conditions (i)-(iii) / (i')-(ii'), which are asymptotic conditions, not fitted targets. The pruning scalings in Theorems 4.1-4.3 are chosen analytically (e.g., the proof of Thm 4.2 computes E|w_{i,k}b_{i,k}|^2 ~ 2/n) to satisfy condition (ii); the equalities lim ||J^B_1|| = lim ||J^iid_1|| then follow from the universal theorem, so these are not fitted inputs renamed as predictions. Theorem 5.1 is likewise a direct application of the Gaussian case of Theorem 3.1. The genuine weakness is Approximation 2.1: Assumptions (A2)-(A3) take the ReLU sign matrices D_k as exogenous i.i.d. Bernoulli(1/2) independent of W and B, and Section 6.1 verifies this only for iid Gaussian weights (joint-distribution plots, a chi-square test on two D entries, and a T_W/T_D heuristic). For magnitude pruning, B is a threshold function of |W|, and for correlated weights the pre-activations are functions of W, so the independent-coin model is asserted without proof; this is a missing-support/assumption gap rather than circularity, because the conclusion is not assumed in the hypothesis and the approximation is not identical to the claimed norm equality. The paper's self-citations ([TY21], [AOSTY24], [HTDT21], [HDR19]) are contextual, confirmatory, or introductory and are not load-bearing; the central theorem rests on external work by non-overlapping authors. Score 1 reflects these minor non-load-bearing self-citations and the acknowledged approximation gap, not a circular derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central theorem uses no fitted constants; all scaling factors are derived from stated moment conditions. The main unpaid input is the Bernoulli/discrete-sign approximation for ReLU pre-activations and the black-box use of recent free probability theorems.

assumptions (4)
  • domain assumption D_l are i.i.d. Bernoulli(1/2) and independent of W_l and B_l (Assumptions A2-A3; Approximation 2.1)
    Needed to apply strong asymptotic freeness to W_l D_{l-1}. Verified for i.i.d. Gaussian weights in Section 6.1, but not established for sparse or correlated weights.
  • standard math Strong asymptotic freeness and matrix concentration results of [BvH24, Theorem 3.29] and [BBvH23, Theorem 2.10] are valid in the regime used here
    These published RMT theorems are the engine of Theorem 3.1; the paper checks conditions but does not reprove them.
  • domain assumption In the infinite-width limit, entries of pre-activations Y_k become i.i.d. Gaussian and independent across layers, justifying the Bernoulli approximation
    Invoked via Approximation 2.1 and supporting references [Nea95, LBN+18, MHR+18, HDR19, Yan19].
  • domain assumption Input x is fixed and non-zero; stability conclusions are assumed to be independent of input choice
    Stated in Section 1 after Eq. (2); only empirically checked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Stability of the Jacobian Matrix in Deep Neural Networks." pith.science (2026). https://pith.science/paper/QZTGQLTI

@misc{pith2026250608764,
  author       = {Pith},
  title        = {Pith review of: On the Stability of the Jacobian Matrix in Deep Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZTGQLTI}},
  note         = {Machine review of arXiv:2506.08764}
}
read the original abstract

Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian. Prior work has identified critical initialization schemes that ensure Jacobian stability, but these analyses are typically restricted to fully connected networks with i.i.d. weights. In this work, we go significantly beyond these limitations: we establish a general stability theorem for deep neural networks that accommodates sparsity (such as that introduced by pruning) and non-i.i.d., weakly correlated weights (e.g. induced by training). Our results rely on recent advances in random matrix theory, and provide rigorous guarantees for spectral stability in a much broader class of network models. This extends the theoretical foundation for initialization schemes in modern neural networks with structured and dependent randomness.

Figures

Figures reproduced from arXiv: 2506.08764 by the authors.

Figure 1
Figure 1. Illustration of the Jacobian norm at initialization in an MLP network of width n = 256 and varying depth. The input is randomly selected from MNIST. All results are averaged over 3 runs. (Left) Impact of depth on the Jacobian norm for different σw. (Right) Evolution of the Jacobian norm as a function of depth for critical initialization. The choice σ 2 w = 2 corresponds to the Edge of Chaos initialization; an initia… view at source ↗
Figure 2
Figure 2. Jacobian norm after pruning at initialization as depth increases in a randomly pruned MLP of width n = 256 (input sampled randomly from MNIST). (Left) Without scaling. (Right) With scaling. 4.1. Random Pruning. In the subsequent analysis, we use the notation an ≫ bn for two positive sequences an, bn, whenever bn = O(an). Recall that J iid 1 refers to the Jacobian of an iid randomly initialized MLP as in Section 2. T… view at source ↗
Figure 3
Figure 3. Jacobian norm after pruning at initialization as depth increases in a score-based pruned MLP of width n = 256 (input sampled randomly from MNIST). (Upper Left) Without scaling (scaling factor = 1). (Upper Right) scaling factor = (1 − sn) − 1 2 . (Lower Left) Scaling factor is calculated based on Theorem 4.3. (Lower Right) Comparison of the scaling factors as a function of sparsity level. Putting the two pieces toget… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Illustration of the Jacobian norm for a randomly selected input with an MLP archi￾tecture of width n = 256 and varying depths. All results are averaged over 3 runs, and confidence intervals are highlighted with shaded areas. (Left) Impact of Depth on the Jacobian norm …
Figure 5
Figure 5. Figure 5: Joint distributions of three randomly selected entries of Dl (denoted by 1, 2, and 3) for l = 10 in a depth L = 30 and width n = 100 MLP with a randomly selected input, based on N = 1000 simulations. Since the values of the entries are binary (0 or 1) we added random G…
Figure 6
Figure 6. Figure 6: Joint distribution of (TW , TD) for Dl and Wl (l = 10) in a L = 30 and n = 100 MLP with a randomly selected input, based on N = 1000 simulations. 0 2 4 6 8 10 Epoch 0 20 40 60 80 100 Val accuracy Normalization: False, Depth: 5 Sparsity 0.0 0.5 0.9 0.9175 0.935 0.9525 0…
Figure 7
Figure 7. Figure 7: Test (Validation) accuracy of a width n = 256 randomly pruned MLP of varying depths trained on Fashion-MNIST with learning rate γ = 0.01. Top figures show the results without scaling (we refer to this by normalization) while the bottom figures show the results with sca…
Figure 8
Figure 8. Figure 8: Test loss of a width n = 256 randomly pruned MLP of varying depths trained on Fashion-MNIST with learning rate γ = 0.01. Top figures show the results without normalization while the bottom figures show the results with normalization. 10 1 10 2 Depth 10 1 10 2 Test Erro…
Figure 9
Figure 9. Figure 9: (Left) Test error after convergence (70 epochs) as a function of depth in an MLP with width n = 256 trained with SGD with learning rate γ = 0.001 on MNIST. The results are shown for 3 different choices of σw, the variance of the iid weights. The critical initialization…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    Omni-DuplexEval provides a new benchmark and automatic evaluation method for real-time duplex omni-modal interaction, showing state-of-the-art models reach only 39.6% overall and 20% on proactive reminders.

Reference graph

Works this paper leans on

13 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    On spectral outliers of inhomogeneous symmetric random matrices

    [AOSTY24] D. Altschuler, P. Oliveira Santos, K. Tikhomirov, and P. Youssef,On spectral outliers of inhomogeneous symmetric random matrices, Arxiv:2401.07852 (2024). [BBvH23] A. S. Bandeira, M. T. Boedihardjo, and R. van Handel,Matrix concentration inequalities and free probability, Invent. Math.234(2023), no. 1, 419–487. MR 4635836 [BGBK19] F. Benaych-Geo...

    work page Pith review arXiv 2024

  2. [6]

    LeCun, J

    [LDS90] Y. LeCun, J. Denker, and S. Solla,Optimal brain damage, Advances in Neural Information Processing Sstems, 1990, pp. 598–605. [LFLY18] C. Li, H. Farkhoor, R. Liu, and J. Yosinski,Measuring the intrinsic dimension of objective landscapes, arXiv preprint arXiv:1804.08838 (2018). [LvHY18] R. Lata la, R. van Handel, and P. Youssef,The dimension-free st...

    arXiv 2018

  3. [7]

    Mishkin and J

    [MM15] D. Mishkin and J. Matas,All you need is a good init, arXiv preprint arXiv:1511.06422 (2015). [MS17] J. A. Mingo and R. Speicher,Free probability and random matrices, Fields Institute Monographs, vol. 35, Springer, New York; Fields Institute for Research in Mathematical Sciences, Toronto,

    arXiv 2015

  4. [110]

    Pennington, S

    MR 0083529 [PSG17] J. Pennington, S. S. Schoenholz, and S. Ganguli,Resurrecting the sigmoid in deep learning through dynamical isometry: Theory and practice, Proceedings of the 31st International Conference on Neural Information Processing Systems (Red Hook, NY, USA), NIPS’17, Curran Associates Inc., 2017, p. 4788–4798. [Qi84] L. Q. Qi,Some simple estimat...

    work page 1984

  5. [1995]

    Elementary inequalities for Mills’ ratio

    [PLR+16] B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli,Exponential expressivity in deep neural networks through transient chaos, 30th Conference on Neural Information Processing Systems (2016). [Pol56] H. O. Pollak, A remark on “Elementary inequalities for Mills’ ratio” by Yˆ usaku Komatu, Rep. Statist. Appl. Res. Un. Japan. Sci. Engrs.4(1956),

    work page 2016

  6. [2011]

    [Tro15] J. A. Tropp,An introduction to matrix concentration inequalities, Foundations and Trends®in Machine Learning 8(2015), no. 1-2, 1–230. [TY21] K. Tikhomirov and P. Youssef,Outliers in spectrum of sparse Wigner matrices, Random Structures Algorithms 58(2021), no. 3, 517–605. MR 4234995 [Ver18] R. Vershynin,High-dimensional probability, Cambridge Seri...

    work page 2015

  7. [2013]

    Brailovskaya and R

    MR 3185193 [BvH24] T. Brailovskaya and R. van Handel,Universality and sharp matrix concentration inequalities, Geom. Funct. Anal. 34(2024), no. 6, 1734–1838. MR 4823211 [Cha07] S. Chatterjee, Stein’s method for concentration inequalities, Probab. Theory Related Fields138(2007), no. 1-2, 305–321. MR 2288072 [CDK22] R. Chhaibi, T. Daouda, and E. Kahn,Free p...

    work page 2024

  8. [2016]

    [vH17] ,Structured random matrices, Convexity and concentration, IMA Vol. Math. Appl., vol. 161, Springer, New York, 2017, pp. 107–156. MR 3837269 [WQB+22] H. Wang, C. Qin, Y. Bai, Y. Zhang, and Y. Fu,Recent advances on neural network pruning at initialization, Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI, Vienna, Au...

    work page 2017

Show all 13 references
  1. [2017]

    [SMG13] A. M. Saxe, J. L. McClelland, and S. Ganguli,Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, arXiv preprint arXiv:1312.6120 (2013). [Tao11] T. Tao,Topics in random matrix theory,

    2013 arXiv

  2. [2018]

    LeCun, Y

    [LBH15] Y. LeCun, Y. Bengio, and G. Hinton,Deep learning, nature521(2015), no. 7553, 436–444. [LBN+18] J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein,Deep neural networks as Gaussian processes, International Conference on Learning Representations,

    2015

  3. [2019]

    [HDR22] ,The curse of depth in kernel regime, Proceedings on ”I (Still) Can’t Believe It’s Not Better!” at NeurIPS 2021 Workshops (M. F. Pradier, A. Schein, S. Hyland, F. J. R. Ruiz, and J. Z. Forde, eds.), Proceedings of Machine Learning Research, vol. 163, PMLR, 2022, pp. 41...

    2020

  4. [2020]

    Yang,Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are Gaussian processes, arXiv preprint arXiv:1910.12478 (2019)

    [Yan19] G. Yang,Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are Gaussian processes, arXiv preprint arXiv:1910.12478 (2019)

    2019 arXiv

  5. [2021]

    [JYZ+20] G. Jin, X. Yi, L. Zhang, L. Zhang, S. Schewe, and X. Huang,How does weight correlation affect generalisation ability of deep neural networks?, Advances in Neural Information Processing Systems33(2020), 21346–21356. [KB14] D. P. Kingma and J. Ba,Adam: A method for stoc...

    2020 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.