$\epsilon$-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics

Jiang Yang; Quanhui Zhu; Yuxiang Zhao

REVIEW 3 major objections 5 minor 1 cited by

$\epsilon$-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper establishes that as a neural network trains with standard SGD, the effective rank of its last hidden layer's features rises in steps, and it proves a lower bound making high effective rank necessary for large loss reductions.

desk verdict Useful empirical observation and a simple initialization trick, but the 'rank growth is necessary for loss reduction' claim is oversold; worth a serious referee. read the letter →

arxiv 2412.05144 v3 pith:5ZYOL75F submitted 2024-12-06 cs.LG cs.NAmath.NA

Jiang Yang , Yuxiang Zhao , Quanhui Zhu This is my paper · ORCID

classification cs.LGcs.NAmath.NA

keywords epsilon-rankstaircasephenomenonneuralnetworktrainingdynamicsGrammatrixfeaturediversitypre-traininginitializationphysics-informednetworkslossplateau

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces $\epsilon$-rank, a computable measure of how many genuinely independent features a neural network's last hidden layer represents: it counts the eigenvalues of the Gram matrix of the neuron functions that exceed a small tolerance. Across function fitting, PDE solving with physics-informed networks, and MNIST classification, the authors observe a staircase phenomenon under standard stochastic gradient descent training: the loss falls in plateaus, and each rapid drop coincides with a stepwise increase in $\epsilon$-rank. The paper proves a lower bound relating loss to $\epsilon$-rank: if the last hidden layer has effective rank $p$, the loss cannot go below a constant multiple of the distance from the true solution to the class of networks with only $p$ effective features. A sufficiently high $\epsilon$-rank is therefore necessary for a significant loss reduction, which explains why training stalls when feature diversity is low. Based on this, the paper proposes a first-layer pre-training strategy that starts the network with high $\epsilon$-rank and shows it removes the initial plateau and speeds up training.

What carries the argument

The load-bearing object is the $\epsilon$-rank $r_\epsilon(M_u)$, defined as the number of eigenvalues of the Gram matrix $M_u$ (with entries $\int_\Omega \phi_i(x;\theta)\phi_j(x;\theta)\,dx$ for the last-hidden-layer neuron functions $\phi_j$) that exceed a tolerance $\epsilon$. It converts the question of how many useful features a layer has into a linear-algebra count. The argument then runs through two steps: a matrix perturbation lemma (Lemma 3.1, based on rank-revealing QR factorizations) showing that any $n$-function set with $\epsilon$-rank $p$ has a $p$-function subset that approximates the original span to within a controlled error, and Theorem 3.2, which combines that approximation with a stability condition on the solution operator to lower-bound the loss of any low-$p$ approximator. The same machinery motivates the pre-training strategy: by constructing the first hidden layer with localized, well-separated neurons (via explicit centers in 1D and random directions on the unit sphere in higher dimensions), the $\epsilon$-rank is forced to $O(n)$ from the start, bypassing the staircase.

What would settle it

Train a wide network on a target that is exactly a linear combination of two fixed basis functions (for example, $u^*(x)=\cos x+2\sin x$). If the training loss can be driven to machine precision while the last hidden layer's Gram matrix has $\epsilon$-rank near 2 throughout the run, the necessity claim is refuted. The same test can be run on a classification dataset whose labels depend linearly on a low-dimensional projection of the input.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the evolution of a network's last-hidden-layer feature diversity is not smooth but stepwise, and that this stepwise behavior is forced by a quantitative relation between loss and effective rank. The relevant bound is $\sqrt{L(u_n)} \ge \frac{1}{C_S}\left(\operatorname{dist}(u^*, F_p) - \sqrt{C(p+1)(n-p)^2\epsilon}\right)$ for any approximator $u_n$ whose $\epsilon$-rank is $p$; here $u^*$ is the exact solution, $F_p$ is the class of networks with $p$ neurons, $C_S$ is a stability constant, and $\epsilon$ is the tolerance defining the rank. Since $\epsilon$ is small, the bound says the best possible loss is controlled by how far the true solution is from the $p$-neuron network class. Reducing the loss therefore requires reducing that distance, which in turn requires increasing $p$. The paper interprets this as the theoretical explanation of the staircase: when the $\epsilon$-rank is stuck at a low level, the loss is trapped at a plateau; when the rank jumps up, the loss drops sharply. It then shows experimentally that raising the initial $\epsilon$-rank of the first hidden layer through a tailored initialization produces the same final accuracy in fewer iterations.

Load-bearing premise

The paper's necessity conclusion rests on the assumption, stated in Section 3.1, that the exact solution $u^*$ does not belong to the network function class $F_n$; if the target can be represented exactly or very accurately by a small number of features, loss can go to zero while the $\epsilon$-rank stays low, and the claimed necessity would fail.

Editorial extensions

If this is right

If the necessity claim holds, a flat loss curve with a stagnant $\epsilon$-rank is a capacity signal: the network is not failing because of learning rate or optimizer choice, but because its last hidden layer lacks functionally independent features.
A practical corollary is that initialization and architecture choices that raise the initial $\epsilon$-rank should reduce or remove the early-training plateau, which the paper demonstrates with its first-layer pre-training strategy.
The lower bound ties optimization to approximation theory: the achievable loss after training is limited by $\operatorname{dist}(u^*, F_p)$, so progress requires effective rank to grow until the target is well approximated by the feature span.
Layer-wise experiments showing higher $\epsilon$-rank in deeper layers suggest that feature diversity is built up through the network, so shallow bottlenecks can be identified by comparing per-layer ranks.
The staircase is observed across function fitting, PDE solving, and image classification, so if universal it applies to the common deep-learning pipeline for scientific computing as much as to vision tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper stops short of prescribing rank monitoring as an adaptive training control; an immediate extension would be to trigger learning-rate changes or early stopping when $\epsilon$-rank plateaus, which the staircase observation suggests.
The necessity result may fail to transfer to classification losses: for a dataset whose classes are separated by a low-dimensional manifold, cross-entropy loss can plausibly reach near zero while the last-layer $\epsilon$-rank stays small. That is testable by measuring ranks on well-trained classifiers.
The UDI initialization samples directions uniformly on the sphere; a natural extension is to replace random directions with data-dependent or learned directions, or to apply the same rank-boosting idea to convolutional filters, which the paper does not explore.
The conjecture that the sharp constant in Lemma 3.1 is $1/\sqrt{n}$ for all $p$ is a standalone mathematical question; resolving it would tighten the loss lower bound and sharpen the claimed necessity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper has a genuinely useful empirical observation and a simple initialization trick, but the theory is oversold. The staircase phenomenon — loss plateaus punctuated by jumps in the effective rank of the last-layer Gram matrix — is a real and clearly demonstrated pattern across tanh-MLP and PINN tasks. The UDI initialization (random directions and biases on the first layer) is simple, adds no parameters, and the experiments show it breaks the early plateau and speeds convergence. That part is worth taking seriously.

The soft spot is the necessity claim. Theorem 3.2 is a static lower bound: if the network has epsilon-rank p and the target is well approximated by some p-dimensional feature subspace, the bound is near zero and no rank growth is required to reach low loss. The paper's Remark 3.2 slides from 'loss lower bound depends on dist(u*,F_p)' to 'therefore rank must increase,' which only follows if you already know the loss is below the bound for the initial p. The assumption u*∉F_n does not fix this, because u* can be outside F_n while still being within O(ε) of F_p for small p. The MNIST cross-entropy experiment doesn't even fit the theorem's L2-residual framework. So the 'universal necessity' statement is not established. The authors should either drop it or state it as a conditional observation.

Other issues are minor: epsilon is fixed at 1e-6 with no sensitivity check; there are no error bars or released code; the 'universal' claim rests on a handful of single-run curves. None of this suggests fabrication — just a paper that needs another pass.

Recommendation: send it out. A good referee can push the authors to soften the theory, add repeated runs, and release code. If the staircase observation and UDI replicate, this is a useful paper for the scientific-computing/PINN community.

Referee Report

3 major / 5 minor

Summary. The paper introduces the ε-rank of the Gram matrix of last-hidden-layer neuron functions, reports a 'staircase phenomenon' in which loss plateaus are accompanied by stepwise increases in this rank across function fitting, PINN, and MNIST experiments, and proves in Theorem 3.2 a lower bound on L2 residual loss in terms of dist(u*, F_p). It interprets this bound as showing that a sufficiently high ε-rank is a necessary condition for significant loss reduction. It then proposes a uniform-distribution initialization (UDI) that gives O(n) ε-rank at initialization and reports faster convergence and improved accuracy in 1D/2D function fitting and PINN tests.

Significance. The ε-rank is a clean, computable diagnostic, and the empirical staircase pattern across several tasks is a useful observation that connects naturally to spectral bias and feature-learning dynamics. The UDI initialization is simple, cheap, and appears to help in the tested settings, with numerical settings documented in Appendix A. The theoretical result, however, is a static approximation bound rather than a training-dynamics necessity theorem; the paper's central claim overstates what Eq. (9) actually establishes. With a weaker, properly qualified statement and additional numerical checks of dist(u*, F_p), the contribution would be valuable to the deep-learning theory community.

major comments (3)

[§3.1, Eq. (9), Remark 3.2] The central claim that a sufficiently high ε-rank is essential for significant loss reduction is not a consequence of Theorem 3.2. For a fixed p, Eq. (9) is a static lower bound; the loss can decrease toward dist(u*,F_p)/C_s − O(√ε) while the ε-rank remains p. Necessity of rank growth follows only if the final loss is below the bound computed at the initial p and dist(u*,F_p)>0 for all smaller p. The manuscript never estimates dist(u*,F_p) for its problems, and the assumption u*∉F_n is too weak to imply it: a target fδ = tanh(a1·x)+tanh(a2·x)+δg with g∉F_n and δ small is not in F_n yet has dist(fδ,F_2)=O(δ), so Eq. (9) does not rule out reaching O(δ²) loss with ε-rank O(2). The abstract, Section 1, and Remark 3.2 should be revised to state this conditional form of the claim.
[§2.2, Example 2.3 and §3] Theorem 3.2 is stated for losses L(u)=||G(u)−f||² with a stable solution operator G^{-1}, but Figure 5 (MNIST) uses cross-entropy loss, for which the theorem does not apply. Since Fig. 5 is presented as evidence of universality and Section 3 as its theoretical explanation, this is a gap. Either extend the theory to classification losses or explicitly present the MNIST observation as an empirical phenomenon outside the scope of the theorem.
[§4.2, Eq. (11), Fig. 8] The pre-training strategy is motivated by the layer-wise observation in Fig. 8, but that observation is reported for a single network (L=4, n=50) on one task. Since Section 4.2 states a general 'depth-dependent growth in ε-rank', supporting evidence across widths, depths, and tasks, or an explicit disclaimer that this is a single-run observation, is needed before the motivation for UDI is fully established.

minor comments (5)

[Example 4.3] The example title contains a typo: 'Possion's equation' should be 'Poisson's equation'.
[Figures 2, 3, 10] Several axis labels are malformed (for example, 'L oss' in Figures 2 and 3, and broken superscripts in Figure 10); please regenerate the figures with correct typography.
[§4] The term 'pre-training' is used for a one-time initialization scheme rather than a separate training phase; consider renaming it 'specialized initialization' to avoid confusion with standard pretraining.
[Definition 2.4 and Eq. (12)] The ε-rank is computed in practice from a quadrature or Monte Carlo approximation of the Gram matrix, but the paper does not state an error bound connecting the discrete and continuous ranks; a short remark on how m and the quadrature weights affect the reported ranks would help.
[§3.1] The assumption u*∉F_n is not verified for any of the numerical examples; the paper should at least acknowledge that this is an assumption rather than a checked condition.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the epsilon-rank definitions, Theorem 3.2 bound, and UDI experiments do not reduce to fitted inputs or self-citations.

full rationale

The paper's central theoretical result (Theorem 3.2) is derived from Definition 2.4 and Theorem 3.1 using an external rank-revealing lemma [13]. The lower bound sqrt(L(u_n)) >= (dist(u*,F_p) - O(sqrt(eps)))/C_s is a mathematical consequence of the fact that an epsilon-rank-p Gram matrix leaves a tail of eigenvalues below epsilon; it does not fit any parameter to the loss data and does not assume the staircase conclusion. This is a definitional consequence rather than a circular prediction, and the paper does not use the theorem's bound as the source of the training curves it reports. The staircase phenomenon is an empirical observation recovered independently from loss and rank measurements, and the UDI pretraining method is evaluated by loss/accuracy, not solely by the epsilon-rank it was designed to raise. No load-bearing self-citations appear; reference [13] is external and [38] is used for inspiration, not as justification of the necessity claim. The main weakness is logical, not circular: Remark 3.2 moves from a static lower bound to the dynamic conclusion that loss reduction forces rank growth, and the standing assumption u* not in F_n does not guarantee dist(u*,F_p)>0 for p<n. That is an overclaim or missing support, but it is not an input-output equivalence. Also, the near-tautological status of the bound does not make the derivation circular, because Definition 2.4 does not mention loss or training dynamics. Score 0.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on assumptions about the target being outside the network class, a stable solution operator, a bounded output coefficient norm, and a quadrature approximation of the Gram matrix. These are reasonable in parts of the theory but are not verified experimentally. The free parameters are hand-set tolerances and hyperparameters such as epsilon, gamma, PINN weights, and quadrature size. No new physical or independently evidenced entities are postulated; epsilon-rank is a mathematical definition rather than an empirical entity.

free parameters (4)

epsilon tolerance for rank = 1e-6 for most examples, 1e-8 for Example 2.2, 1e-12 for Example 3.2
The definition of epsilon-rank and the observed staircase pattern depend on this hand-set threshold; no sensitivity analysis is provided.
gamma (UDI shape parameter) = gamma = 2 for Example 4.2, gamma = n/50 for Example 4.3
Hand-tuned per experiment; affects the scale of the initial neuron features and the reported speed-up.
PINN loss weights mu_bc and mu_ic = 1 or 20 depending on example
Chosen per problem in Table 2; influences the reported loss curves for PDE examples.
Gram matrix quadrature size m = 129, 100, 65, or 50 depending on example
The computed epsilon-rank changes with the number of sample points; the paper fixes m per example without a convergence study.

assumptions (6)

domain assumption The exact solution u* does not belong to the network function class F_n
Stated in Section 3.1; without this assumption low-rank networks could achieve zero loss and the claimed necessity of rank growth fails.
domain assumption Stability condition (8): G^{-1} is Lipschitz with constant C_S
Used in the proof of Theorem 3.2 to convert output loss into solution error; not verified for the PDE examples.
domain assumption Coefficient norm bound ||beta||_2 <= C
Assumed in Theorems 3.1 and 3.2; not enforced or tracked during training.
standard math Hong-Pan rank-revealing QR bound (Lemma 3.1 of [13])
Imported from the literature to bound ||V22^{-T}||; the paper does not reprove it and applies it in a somewhat loose form.
standard math Universal approximation theorem for the activation functions used
Invoked in Section 3.1 to justify the approximation capacity of the network function class F_n.
domain assumption Discrete quadrature or Monte Carlo samples approximate the L2 Gram matrix
Appendix A replaces the L2 Gram matrix with a weighted sum over m points; computed rank values depend on this approximation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of $\epsilon$-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics." pith.science (2026). https://pith.science/paper/5ZYOL75F

@misc{pith2026241205144,
  author       = {Pith},
  title        = {Pith review of: $\epsilon$-rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ZYOL75F}},
  note         = {Machine review of arXiv:2412.05144}
}

abstract

Understanding the training dynamics of deep neural networks (DNNs), particularly how they evolve low-dimensional features from high-dimensional data, remains a central challenge in deep learning theory. In this work, we introduce the concept of $\epsilon$-rank, a novel metric quantifying the effective feature of neuron functions in the terminal hidden layer. Through extensive experiments across diverse tasks, we observe a universal staircase phenomenon: during training process implemented by the standard stochastic gradient descent methods, the decline of the loss function is accompanied by an increase in the $\epsilon$-rank and exhibits a staircase pattern. Theoretically, we rigorously prove a negative correlation between the loss lower bound and $\epsilon$-rank, demonstrating that a high $\epsilon$-rank is essential for significant loss reduction. Moreover, numerical evidences show that within the same deep neural network, the $\epsilon$-rank of the subsequent hidden layer is higher than that of the previous hidden layer. Based on these observations, to eliminate the staircase phenomenon, we propose a novel pre-training strategy on the initial hidden layer that elevates the $\epsilon$-rank of the terminal hidden layer. Numerical experiments validate its effectiveness in reducing training time and improving accuracy across various tasks. Therefore, the newly introduced concept of $\epsilon$-rank is a computable quantity that serves as an intrinsic effective metric characteristic for deep neural networks, providing a novel perspective for understanding the training dynamics of neural networks and offering a theoretical foundation for designing efficient training strategies in practical applications.

Figures

Figures reproduced from arXiv: 2412.05144 by the authors.

**Figure 2.** (example 2.1) Staircase phenomenon under varying network width ( [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** (example 2.1) Staircase phenomenon under different activation functions. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: (example 2.2) Staircase phenomenon in solving Allen–Cahn equation. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: (example 2.3) Staircase phenomenon in handwriting recognition. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: (example 3.1) The training process under different sampling settings. The left subfigure [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: (example 3.2) (a)-(e) are the solutions and point-wise errors of random feature method [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

**Figure 8.** Figure 8: (example 2.1) Layer-wise ϵ-rank over training process Building on these observations, we propose a targeted pre-training strategy to enhance the ϵrank of the first hidden layer. Inspired by basis functions in classical finite element methods, for x ∈ [−1, 1], we defin…

**Figure 9.** Figure 9: (example 4.1) Training dynamics for one-dimensional function fitting under different [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

**Figure 10.** Figure 10: (example 4.2) Training dynamics of 2-D function approximation using Xavier initial [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

**Figure 11.** Figure 11: (example 4.3)Training dynamics of PINN for the 2D Poisson equation with Xavier [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Structured First-Layer Initialization Pre-Training Techniques to Accelerate Training Process Based on $\varepsilon$-Rank
math.NA 2025-07 conditional novelty 5.0 of 10

A structured first-layer initialization raises the ε-rank of neuron functions and accelerates training of PINNs and function approximators.

Reference graph

Works this paper leans on

38 extracted references · 33 canonical work pages · cited by 1 Pith paper

[1]

On the Inductive Bias of Neural Tangent Kernels

Alberto Bietti and Julien Mairal. On the Inductive Bias of Neural Tangent Kernels. In Advances in Neural Information Processing Systems , volume 32, pages 12893–12904. Curran Associates, Inc., 2019

work page 2019
[2]

The Random Feature Method for Time-Dependent Problems

Jingrun Chen, Weinan E, and Yixin Luo. The Random Feature Method for Time-Dependent Problems. East Asian Journal on Applied Mathematics , 13(3):435–463, January 2023. 22 Table 2: Numerical experiment settings in function fitting and solving PDEs Example L n m N ϵ Other Parameters F unction Fitting example 2.1(a) 2/4 25/50 129 250 10 −6 - example 2.1(b) 3 ...

work page 2023
[3]

Optimization of Random Feature Method in the High- Precision Regime

Jingrun Chen, Weinan E, and Yifei Sun. Optimization of Random Feature Method in the High- Precision Regime. Communications on Applied Mathematics and Computation , 6(2):1490– 1517, June 2024

work page 2024
[4]

A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

Zixiang Chen, Yuan Cao, Quanquan Gu, and Tong Zhang. A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks. In Advances in Neural Information Processing Systems, volume 33, pages 13363–13373. Curran Associates, Inc., 2020

work page 2020
[5]

Sharp Minima Can Gener- alize For Deep Nets

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp Minima Can Gener- alize For Deep Nets. In Proceedings of the 34th International Conference on Machine Learning, pages 1019–1028. PMLR, July 2017

work page 2017
[6]

Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations

Suchuan Dong and Zongwei Li. Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations. Computer Methods in Applied Mechanics and Engineering , 387:114129, December 2021

work page 2021
[7]

Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations

Weinan E, Jiequn Han, and Arnulf Jentzen. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics , 5(4):349–380, 2017

work page 2017
[8]

The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems

Weinan E and Bing Yu. The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics , 6(1):1–12, 2018. 23

work page 2018

Show all 38 references

[9]

Understanding the difficulty of training deep feedforward neural networks

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial In- telligence and Statistics , pages 249–256, 2010

2010
[10]

ReLU deep neural networks from the hierarchical basis perspective

Juncai He, Lin Li, and Jinchao Xu. ReLU deep neural networks from the hierarchical basis perspective. Computers & Mathematics with Applications , 120:105–114, August 2022

2022
[11]

Relu Deep Neural Networks and Linear Finite Elements

Juncai He, Lin Li, Jinchao Xu, and Chunyue Zheng. Relu Deep Neural Networks and Linear Finite Elements. Journal of Computational Mathematics , 38(3):502–527, 2020

2020
[12]

Flat Minima

Sepp Hochreiter and J¨ urgen Schmidhuber. Flat Minima. Neural Computation , 9(1):1–42, January 1997

1997
[13]

Y. P. Hong and C.-T. Pan. Rank-revealing QR factorizations and the singular value decom- position. Mathematics of Computation , 58(197):213–232, 1992

1992
[14]

Approximation capabilities of multilayer feedforward networks

Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251–257, 1991

1991
[15]

Universal Approximation using Incre- mental Constructive Feedforward Networks with Random Hidden Nodes

Guang-Bin Huang, Lei Chen, and Chee-Kheong Siew. Universal Approximation using Incre- mental Constructive Feedforward Networks with Random Hidden Nodes. IEEE Transactions on Neural Networks , 17(4):879–892, July 2006

2006
[16]

Extreme learning machine: Theory and applications

Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Siew. Extreme learning machine: Theory and applications. Neurocomputing, 70(1):489–501, December 2006

2006
[17]

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In Advances in Neural Information Processing Systems , volume 31, pages 8580–8589. Curran Associates, Inc., 2018

2018
[18]

On large-batch training for deep learning: Generalization gap and sharp minima

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations , 2017

2017
[19]

Wight and Jia Zhao

Colby L. Wight and Jia Zhao. Solving Allen-Cahn and Cahn-Hilliard Equations Using the Adaptive Physics Informed Neural Networks. Communications in Computational Physics , 29(3):930–954, 2021

2021
[20]

Visualizing the Loss Landscape of Neural Nets

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Landscape of Neural Nets. In Advances in Neural Information Processing Systems, volume 31, pages 6391–6401. Curran Associates, Inc., 2018

2018
[21]

Multi-Scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains.Communications in Computational Physics, 28(5):1970–2001, 2020

Ziqi Liu, Wei Cai, and Zhi-Qin John Xu. Multi-Scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains.Communications in Computational Physics, 28(5):1970–2001, 2020

1970
[22]

Meethal, Anoop Kodakkal, Mohamed Khalil, Aditya Ghantasala, Birgit Obst, Kai- Uwe Bletzinger, and Roland W¨ uchner

Rishith E. Meethal, Anoop Kodakkal, Mohamed Khalil, Aditya Ghantasala, Birgit Obst, Kai- Uwe Bletzinger, and Roland W¨ uchner. Finite element method-enhanced neural network for forward and inverse problems. Advanced Modeling and Simulation in Engineering Sciences , 10(1):6, Ma...

2023
[23]

Mitusch, Simon W

Sebastian K. Mitusch, Simon W. Funke, and Miroslav Kuchta. Hybrid FEM-NN models: Combining artificial neural networks with the finite element method.Journal of Computational Physics, 446:110651, December 2021

2021
[24]

Nelsen and Andrew M

Nicholas H. Nelsen and Andrew M. Stuart. The Random Feature Model for Input-Output Maps between Banach Spaces. SIAM Journal on Scientific Computing , 43(5):A3212–A3243, January 2021

2021
[25]

On the Spectral Bias of Neural Networks

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the Spectral Bias of Neural Networks. In Proceedings of the 36th International Conference on Machine Learning , pages 5301–5310. PMLR, May 2019
[26]

Random features for large-scale kernel machines

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20:1177–1184, 2007

2007
[27]

Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686–707, 2019

2019
[28]

Ramuhalli, L

P. Ramuhalli, L. Udpa, and S.S. Udpa. Finite-Element Neural Networks for Solving Differential Equations. IEEE Transactions on Neural Networks , 16(6):1381–1392, November 2005

2005
[29]

DGM: A deep learning algorithm for solving partial differential equations

Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics , 375:1339–1364, 2018

2018
[30]

When and why PINNs fail to train: A neural tangent kernel perspective

Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computational Physics , 449:110768, January 2022

2022
[31]

Multi-scale deep neural network (MscaleDNN) methods for oscillatory stokes flows in complex domains

Wenzhong Wang, BoZhang and Wei Cai. Multi-scale deep neural network (MscaleDNN) methods for oscillatory stokes flows in complex domains. Communications in Computational Physics, 28(5):2139–2157, 2020

2020
[32]

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Zhi-Qin John Xu. Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks. Communications in Computational Physics , 28(5):1746–1767, June 2020

2020
[33]

Overview Frequency Principle/Spectral Bias in Deep Learning

Zhi-Qin John Xu, Yaoyu Zhang, and Tao Luo. Overview Frequency Principle/Spectral Bias in Deep Learning. Communications on Applied Mathematics and Computation , September 2024

2024
[34]

Training Behavior of Deep Neural Network in Frequency Domain

Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training Behavior of Deep Neural Network in Frequency Domain. In Tom Gedeon, Kok Wai Wong, and Minho Lee, editors, Neural Information Processing, pages 264–274, Cham, 2019. Springer International Publishing

2019
[35]

Weak adversarial networks for high-dimensional partial differential equations

Yaohua Zang, Gang Bao, Xiaojing Ye, and Haomin Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics , 411:109409, 2020

2020
[36]

Zeiler and Rob Fergus

Matthew D. Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014 , pages 818–833, Cham, 2014. Springer International Publishing. 25

2014
[37]

Why shallow networks struggle with approximating and learning high frequency: A numerical study, 2023

Shijun Zhang, Hongkai Zhao, Yimin Zhong, and Haomin Zhou. Why shallow networks struggle with approximating and learning high frequency: A numerical study, 2023

2023
[38]

Transferable Neural Networks for Partial Differential Equations

Zezhong Zhang, Feng Bao, Lili Ju, and Guannan Zhang. Transferable Neural Networks for Partial Differential Equations. Journal of Scientific Computing , 99(1):2, February 2024. 26

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

On the Inductive Bias of Neural Tangent Kernels

Alberto Bietti and Julien Mairal. On the Inductive Bias of Neural Tangent Kernels. In Advances in Neural Information Processing Systems , volume 32, pages 12893–12904. Curran Associates, Inc., 2019

work page 2019

[2] [2]

The Random Feature Method for Time-Dependent Problems

Jingrun Chen, Weinan E, and Yixin Luo. The Random Feature Method for Time-Dependent Problems. East Asian Journal on Applied Mathematics , 13(3):435–463, January 2023. 22 Table 2: Numerical experiment settings in function fitting and solving PDEs Example L n m N ϵ Other Parameters F unction Fitting example 2.1(a) 2/4 25/50 129 250 10 −6 - example 2.1(b) 3 ...

work page 2023

[3] [3]

Optimization of Random Feature Method in the High- Precision Regime

Jingrun Chen, Weinan E, and Yifei Sun. Optimization of Random Feature Method in the High- Precision Regime. Communications on Applied Mathematics and Computation , 6(2):1490– 1517, June 2024

work page 2024

[4] [4]

A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

Zixiang Chen, Yuan Cao, Quanquan Gu, and Tong Zhang. A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks. In Advances in Neural Information Processing Systems, volume 33, pages 13363–13373. Curran Associates, Inc., 2020

work page 2020

[5] [5]

Sharp Minima Can Gener- alize For Deep Nets

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp Minima Can Gener- alize For Deep Nets. In Proceedings of the 34th International Conference on Machine Learning, pages 1019–1028. PMLR, July 2017

work page 2017

[6] [6]

Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations

Suchuan Dong and Zongwei Li. Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations. Computer Methods in Applied Mechanics and Engineering , 387:114129, December 2021

work page 2021

[7] [7]

Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations

Weinan E, Jiequn Han, and Arnulf Jentzen. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics , 5(4):349–380, 2017

work page 2017

[8] [8]

The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems

Weinan E and Bing Yu. The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics , 6(1):1–12, 2018. 23

work page 2018