Structured First-Layer Initialization Pre-Training Techniques to Accelerate Training Process Based on $\varepsilon$-Rank

Jiang Yang; Quanhui Zhu; Tao Tang; Yuxiang Zhao

REVIEW 4 major objections 5 minor 38 references

Structured First-Layer Initialization Pre-Training Techniques to Accelerate Training Process Based on $\varepsilon$-Rank

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper establishes that structured first-layer initialization—spreading first-layer neurons to be ε-linearly independent—raises initial ε-rank, accelerates convergence, mitigates spectral bias, and improves prediction accuracy in…

desk verdict A useful, clearly reported initialization trick with real empirical gains, but the claimed mechanism via first-layer ε-rank is unproven and the reporting gaps (no code, no ε threshold, weak cavity baseline) need fixing before it can be trusted. read the letter →

arxiv 2507.11962 v1 pith:U2BFM5WM submitted 2025-07-16 math.NA cs.NA

Tao Tang , Jiang Yang , Yuxiang Zhao , Quanhui Zhu This is my paper · ORCID

classification math.NAcs.NA MSC 68T0765Z0535Q68

keywords ε-rankstaircasephenomenonstructuredfirst-layerinitializationspectralbiasphysics-informedneuralnetworkstrainingaccelerationfeaturediversityscientificcomputing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a one-line change to the first hidden layer's initial weights can substantially speed up and improve neural network training for scientific computing. It builds on the staircase phenomenon, in which training loss falls mainly when the ε-rank of the neuron functions—the number of effectively independent features—grows. The proposed structured first-layer initialization (SFLI) places first-layer neurons so that their pre-activations are approximately ε-linearly independent from the start, with recipes for cosine, tanh, hat, and Gaussian activations. Across function-approximation and physics-informed PDE benchmarks, the method is reported to raise initial ε-rank, accelerate convergence, mitigate spectral bias, and improve accuracy without adding optimization cost. If the assumed transfer from first-layer diversity to network-wide expressiveness holds, this is a lightweight and broadly compatible speed-up for scientific-computing networks.

What carries the argument

The load-bearing object is the ε-rank $r_\varepsilon(M_u)$: the number of eigenvalues of the Gram matrix of neuron functions in a hidden layer that exceed a threshold $\varepsilon$, which the staircase phenomenon links to loss decay. The second central object is the structured first-layer initialization, which reparameterizes each first-layer neuron as $F_i(x)=\sigma(\gamma(\alpha_i\cdot x+c_i))$, samples the orientation $\alpha_i$ according to the activation function, spreads the shifts $c_i$ uniformly over the domain, and scales localization by $\gamma=C(n^{1/d}-1)/|\Omega|^{1/d}$. This decomposition is used only to set the initial weights; after initialization, training optimizes the original weights and biases directly. The mechanism is that spatially separated, directionally diverse pre-activations are nearly linearly independent at the start, lifting the initial ε-rank and skipping the plateau that standard Xavier-like initialization produces.

What would settle it

Train two otherwise identical networks whose first layers start at the same high ε-rank, one built by SFLI and one by random weights selected to match that ε-rank; if the loss curves track each other, initial ε-rank is the operative variable, while if SFLI pulls ahead, some structural property beyond ε-rank is responsible. A second check is to track first- and final-layer ε-ranks in parallel during training, since a long interval with high first-layer rank but stagnant final-layer rank would break the assumed transfer.

Watch

Extended reading notes

Core claim

The paper's central assertion is that slow early training is a diversity bottleneck: standard initializations leave the first hidden layer with low ε-rank, so the network must spend iterations growing independent features before the loss can drop. SFLI breaks that bottleneck by constructing first-layer weights and biases so each neuron behaves like a localized, directionally distinct feature, written $F_i(x)=\sigma(\gamma(\alpha_i\cdot x+c_i))$ with shifts spread across the input domain. With the localization scale set to $\gamma=C\,(n^{1/d}-1)/|\Omega|^{1/d}$, $C\in[0.5,2]$, the scheme is designed to push the initial ε-rank toward the layer width $n$ and trigger the staircase descent immediately. The paper argues the method is activation-agnostic and architecture-compatible, and supports it with numerical results on high/low-frequency functions, discontinuous multiscale functions, Allen–Cahn, lid-driven cavity flow at Re=3200, torus Navier–Stokes, and high-dimensional parabolic equations.

Load-bearing premise

The whole acceleration argument rests on the assumption that raising the first hidden layer's ε-rank at initialization transfers to the final hidden layer's ε-rank during training, because the staircase lower bound that justifies the method is stated for the final hidden layer's neuron functions.

Editorial extensions

If this is right

Adding SFLI to an existing network requires no change to architecture, optimizer, or loss; the paper reports it as a one-line modification to the first-layer initialization.
In the tested 2D multiscale function, every SFLI activation variant (Gaussian, tanh, cosine, hat) beats the baseline's initial ε-rank and convergence speed, with cosine best matching the target's spectral structure.
For high-dimensional approximation and PDE problems from d=5 up to d=50, SFLI with the fixed default C=1 keeps relative errors well below the baseline and avoids the baseline's breakdown at d=50.
On the lid-driven cavity benchmark at Re=3200, SFLI combined with advanced training techniques reaches 3.75% relative L2 error against 88.3% without SFLI, and outperforms the reported prior baselines.
Because the method changes only initialization, it composes with other acceleration tools such as random weight factorization, loss balancing, causal training, and curriculum training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An inference beyond the paper: if the initial ε-rank is the true driver, random sampling with rejection to reach the same first-layer ε-rank should produce similar gains, so an experiment comparing SFLI against rank-matched random initialization would separate the structured placement from the achieved rank.
Another inference: since the paper's own layer-wise picture shows lower layers lagging in ε-rank, a dynamic regularizer that keeps all layers' ε-ranks high during training—mentioned only as future work—might extend the benefit beyond the initialization stage.
A further inference: the strong performance of cosine SFLI on a target with known frequency content suggests that for PDEs with known spectra, the activation recipe could be chosen as a cheap spectral prior, effectively folding frequency knowledge into initialization.
If the staircase lower bound transfers across layers, SFLI's benefit should grow with network depth; a shallow-versus-deep comparison would quantify how much of the gain is due to the first layer alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper gives activation-specific first-layer initialization recipes (Gaussian, Tanh, Cosine, Hat) and shows they accelerate PINN training across several benchmarks. That part is genuinely useful. The fixed C=1 high-dimensional results are encouraging because they do not require per-problem tuning, and the improvements on Allen-Cahn and the high-dimensional parabolic equation are nontrivial. Credit where it is due: the experiments are broad, the ablations with and without SFLI are the right comparison, and the spectral bias analysis on the discontinuous 1D function is a nice touch.

Now the soft spots, in proportion to how soft they are. The central mechanistic claim, that raising first-layer ε-rank at initialization accelerates whole-network training, is not proven. The lower bound in Section 2 constrains the loss via the final hidden layer's ε-rank; the paper simply assumes that first-layer ε-rank controls final-layer ε-rank throughout training. Figure 2 is correlational, and no control isolates rank from other effects of the initialization, such as the high-frequency feature maps that SFLI also produces. That is a real gap, but it is a missing-experiment problem rather than a flaw in the reported results; the measured loss and error improvements stand on their own.

More fixable issues: the ε threshold in Definition 2.1 is never specified, so the reported rank curves cannot be independently checked. No code is released. The Gram matrix integration rule is also unspecified. These are reporting failures that a careful referee should catch.

The cavity flow comparison worries me more. The paper claims 3.75% relative L2 error versus 88.3% for the baseline, and compares favorably to JAXPI (15.8%) and PirateNet with RFF (4.21%). But the baseline in this paper uses the same smoothed boundary condition and curriculum as the SFLI run, so an 88.3% error suggests the baseline is unusually weak, likely missing standard training practices that even the AT rows in Figure 11 do not include. That comparison makes the headline gain look bigger than it is. The paper should either present a stronger baseline or be explicit about why the baseline is so far behind published numbers.

The paper is a straightforward extension of the authors' prior ε-rank work, but the extension is systematic and the empirical study is new and substantial. It deserves a serious referee, though the referee should push for code, the ε threshold, a proper control, and a fairer cavity baseline. I would not cite it in its current form, but I might after revision.

Referee Report

4 major / 5 minor

Summary. The paper proposes Structured First-Layer Initialization (SFLI), a deterministic initialization scheme for the first hidden layer of fully connected networks used in scientific computing. The method reparameterizes first-layer weights as w_i = γ α_i and biases as b_i = γ c_i, with γ chosen via Eq. (3.3), so that the first-layer neuron functions are approximately ε-linearly independent at initialization. SFLI is integrated into modified MLPs and PirateNets, and the authors report experiments on 2D function fitting, a discontinuous 1D function with spectral-error tracking, high-dimensional smooth approximation, and PINNs for the Allen-Cahn equation, lid-driven cavity flow, torus Navier-Stokes, and high-dimensional parabolic PDEs. The reported results show that SFLI raises the initial ε-rank, accelerates loss decay, reduces spectral bias, and improves prediction errors compared with the paper's baselines.

Significance. If the empirical gains are robust and the claimed mechanism holds, SFLI would be a valuable, almost-zero-cost contribution: it is a one-line change to existing code, is activation-agnostic, and requires no additional training cost. The benchmark set is reasonably diverse, and the paper reports consistent improvements, including high-dimensional tests with fixed C=1 and comparisons inside an advanced PINN pipeline. The use of standard techniques (Adam, NTK weighting, causal training) lends practical credibility. However, the central interpretative claim—that gains occur specifically through an increase in first-layer ε-rank—is not established by the current experiments, because the metric is under-specified and no control isolates rank from other effects of the initialization.

major comments (4)

[Section 2, Definition 2.1] The paper never specifies the threshold ε used to compute the ε-rank, nor the rule used to approximate the Gram matrix integral over Ω. Section 2, Definition 2.1 defines ε-rank abstractly, and Figures 6, 7, 9, and 11 report rank curves without stating ε or the quadrature scheme. This makes the central metric uncheckable and the reported curves non-reproducible. Please state, for every experiment, the value of ε (or a scale-dependent rule such as a fraction of the largest eigenvalue) and how the integral in the Gram matrix is evaluated.
[Section 4.1, Figure 6 and surrounding text] The text claims that SFLI 'accelerates convergence and enhancing accuracy through ε-rank.' This causal claim is not supported by the experiments as reported. SFLI changes several first-layer properties simultaneously: the weight scale γ, the directional distribution of α_i, and the bias placement. The observed speedups could plausibly arise from the high-frequency feature maps induced by large γ, as in random Fourier feature methods, rather than from ε-linear independence per se. Please add a control initialization that matches the statistics of the SFLI first layer (e.g., same γ, same bias distribution) but uses randomly oriented or clustered α_i so that the initial ε-rank is low, and compare rank, loss, and final error against SFLI.
[Section 2 vs. Section 3.1] The theoretical anchor, inequality (2.2), constrains the loss through the ε-rank of the final hidden layer, while SFLI modifies only the first layer. Section 3.1 assumes, without proof or experimental tracking, that a high initial first-layer ε-rank propagates to a high final-layer ε-rank during training and that the associated necessary condition is sufficient to accelerate convergence. Please either track the final-layer ε-rank in the SFLI runs, provide a transfer argument, or explicitly weaken the stated mechanism to 'SFLI improves initial feature diversity and empirically accelerates convergence.'
[Section 4.2, lid-driven cavity] The comparison with 'JAXPI (15.8%)' and 'original PirateNet with random Fourier features (4.21%)' is not apples-to-apples, because the SFLI result is obtained with a full advanced pipeline (RWF, loss balancing, curriculum, and other tricks), whereas the cited baselines may not use those techniques. The within-paper comparison AT vs. AT+SFLI in Figure 11 is more useful. Please either compare with matching advanced techniques in the cited baselines or rephrase the claim to avoid implying a direct method-only comparison.

minor comments (5)

[Abstract] 'With the help of SILP' should read 'With the help of SFLI.'
[Section 4.1, Figure 7] Several labels in Figure 7 are corrupted: 'Bas lin', 'F r qu nc0', 'Magni−ud', 'SFLI_T anh', and the subplot titles for e_low/e_high are missing letters. Please regenerate the figure with clean text.
[Section 4.2, Example 4.7] 'n contrast' should be 'In contrast'.
[Section 2, Figure 2] The layer-wise rank curves in Figure 2 are presented without specifying the dataset, network depth, or the training time at which they were recorded; please add the missing experimental details.
[Reference [6]] The title of reference [6] is duplicated in the bibliography; please correct it.

Circularity Check

2 steps flagged · score 4.0 of 10

SFLI's headline ε-rank benefit is its own construction goal, and the causal 'through ε-rank' mechanism is imported from the authors' prior [35]; loss and PDE accuracy gains are external and therefore keep the paper from being fully circular.

self definitional [Abstract; Section 3.1, 'Pre-Activation Method', goals bullet list]
"The objective of structured first-layer initialization is to construct a set of neuron functions that are approximately ε-linearly independent at initialization. ... The goals of the SFLI pre-training strategy can be summarized as follows: ... Ensure that the ε-rank of the initial feature set {Fi} approaches n with high probability."

SFLI is defined as the deliberate construction of ε-linearly independent first-layer neurons; the bullet 'Ensure that the ε-rank of the initial feature set {Fi} approaches n with high probability' is one of the method's stated goals. The Abstract's claimed benefit that SFLI 'significantly improves the initial ε-rank' is therefore the design target restated as an empirical outcome, not an independent consequence. The loss and PDE-error results are measured externally, so only the rank component of the claim is self-definitional; the probabilistic 'with high probability' is asserted rather than proved, which is why the circularity is partial.
self citation load bearing [Section 2, 'Staircase Phenomenon'; Section 3.1, first paragraph]
"Recent work [35] has identified staircase phenomenon: In training dynamics, the loss function often decreases rapidly along with a significant growth of linear independence of neuron functions. ... This behavior has been observed consistently across different tasks and architectures in [35]. ... Following the findings in [35], the first hidden layer plays a critical role in learning meaningful feature representations."

The paper's causal story—loss decay requires ε-rank growth, so raising first-layer ε-rank at initialization accelerates training—is taken from [35], a preprint by the same author group (Yang, Zhao, Zhu). The lower bound (2.2) that 'theoretically justifies' the staircase is cited from [35] and constrains the final hidden layer; the paper does not prove that first-layer ε-rank controls final-layer ε-rank throughout training. Thus the load-bearing mechanism rests on a self-citation chain rather than on a derivation in this paper. The numerical accuracy gains in Section 4 are independent of ε-rank, so this step affects the explanatory mechanism, not the existence of the observed improvements.

full rationale

Most of what SFLI claims is externally testable and is in fact tested: training loss, relative L2 errors on PDEs, and spectral errors are measured against the problem data, not against ε-rank, so the main accuracy and speed results are not circular. The circular elements are narrower. First, the abstract's claim that SFLI 'significantly improves the initial ε-rank' is essentially the method's stated design goal ('Ensure that the ε-rank of the initial feature set {Fi} approaches n with high probability'); reporting a high initial rank is a sanity check of the construction, not an independent prediction. Second, the mechanism that links first-layer ε-rank to faster whole-network training is imported from the authors' own prior work [35]. The lower bound (2.2) in Section 2 is quoted from [35] and is about the final hidden layer; the paper simply asserts that the first hidden layer 'plays a critical role' and that raising its ε-rank bypasses the rank bottleneck. The present numerical experiments show correlation but do not isolate ε-rank from the other changes SFLI makes (γ scale, directional distribution, bias placement), and the threshold ε in Definition 2.1 is never specified, so the rank curves cannot be independently checked. These are underdetermination and self-citation-load-bearing concerns; they do not reduce the loss/PDE-error results to the method's definition. Hence score 4.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on the staircase phenomenon from the authors' previous work, a heuristic shape-parameter formula, and an unspecified ε-rank threshold. The central empirical gains are measured on external benchmarks, but the theoretical motivation is self-contained to [35].

free parameters (3)

γ (SFLI shape parameter) = varies: 10, 420, 3.16, 10, 1.22, etc.
Controls localization of first-layer activations. Chosen per benchmark; for Example 4.2 Gaussian γ=420 is far outside the recommended interval from (3.3). The formula (3.3) itself is a heuristic.
C (scaling factor in γ) = 0.6 to 1.2 from grid search in Table 1
Grid-searched over [0.5,2] to report best results; fixed C=1 also works but with somewhat larger errors.
ε (threshold for ε-rank) = unspecified
Definition 2.1 requires a threshold ε; the paper never states its value, yet the reported ε-rank values and the claim of initial ε-rank near n depend on it.

assumptions (4)

domain assumption Staircase phenomenon: loss decreases are closely correlated with increases in ε-rank of the final hidden layer.
Adopted from [35]; not re-derived here. It motivates the whole SFLI design.
domain assumption Raising the first hidden layer's ε-rank at initialization transfers to faster loss decay.
Assumed, not proven. The experiments support it indirectly, but no theoretical link between first-layer and final-layer ε-rank is given.
domain assumption Gram matrix of neuron functions can be computed numerically with some integration rule and threshold ε.
The exact quadrature and ε are not specified; all ε-rank plots depend on this.
ad hoc to paper The shape parameter formula γ = C(n^{1/d}-1)/|Ω|^{1/d} produces well-separated basis functions.
Presented as a recommended practical choice without derivation; it is not a physical or mathematical necessity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Structured First-Layer Initialization Pre-Training Techniques to Accelerate Training Process Based on $\varepsilon$-Rank." pith.science (2026). https://pith.science/paper/U2BFM5WM

@misc{pith2026250711962,
  author       = {Pith},
  title        = {Pith review of: Structured First-Layer Initialization Pre-Training Techniques to Accelerate Training Process Based on $\varepsilon$-Rank},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2BFM5WM}},
  note         = {Machine review of arXiv:2507.11962}
}

abstract

Training deep neural networks for scientific computing remains computationally expensive due to the slow formation of diverse feature representations in early training stages. Recent studies identify a staircase phenomenon in training dynamics, where loss decreases are closely correlated with increases in $\varepsilon$-rank, reflecting the effective number of linearly independent neuron functions. Motivated by this observation, this work proposes a structured first-layer initialization (SFLI) pre-training method to enhance the diversity of neural features at initialization by constructing $\varepsilon$-linearly independent neurons in the input layer. We present systematic initialization schemes compatible with various activation functions and integrate the strategy into multiple neural architectures, including modified multi-layer perceptrons and physics-informed residual adaptive networks. Extensive numerical experiments on function approximation and PDE benchmarks, demonstrate that SFLI significantly improves the initial $\varepsilon$-rank, accelerates convergence, mitigates spectral bias, and enhances prediction accuracy. With the help of SILP, we only need to add one line of code to conventional existing algorithms.

Figures

Figures reproduced from arXiv: 2507.11962 by the authors.

**Figure 2.** Layer-wise staircase phenomenon within the same neural network. The width [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Visualization of the first-layer outputs using different activation functions un [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Architecture of the neural network with Structured First-Layer Initialization [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Architecture of PirateNet with Structured First-Layer Initialization (SFLI). [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Comparison of ε-rank dynamics (left) and training loss (right) across different first-layer initializations: baseline (no SFLI) vs. SFLI with Gaussian, Tanh, Cosine, and Hat pre-activations. The detailed hyper-parameter settings are presented in [PITH_FULL_IMAGE:figur…

**Figure 7.** Figure 7: Spectral analysis on Example 4.2 using different SFLI strategies. The target func [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: Allen–Cahn equation: Comparison between the solutions predicted by a trained [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: Allen–Cahn equation: Comparison of [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

**Figure 10.** Figure 10: Lid-driven cavity flow at Re=3200: Prediction of speed [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

**Figure 11.** Figure 11: Lid-driven cavity flow at Re=3200: Comparison of [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

**Figure 12.** Figure 12: Navier-Stokes flow in a torus: Comparison of the predictions of [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 31 canonical work pages

[1]

Emergence of Invariance and Disentanglement in Deep Representations

Alessandro Achille and Stefano Soatto. Emergence of Invariance and Disentanglement in Deep Representations. Journal of Machine Learning Research, 19(50):1–34, 2018

work page 2018
[2]

Mart ´ın

Santiago Badia, Wei Li, and Alberto F. Mart ´ın. Adaptive finite element interpolated neural networks. Computer Methods in Applied Mechanics and Engineering, 437:117806, 2025

work page 2025
[3]

Physically guided neural network based on transfer learning (TL-PGNN) for hypersonic heat flux prediction

Biao Chen, Jifa Zhang, Shuai Zhang, Song Xiaoxiao, and Yao Zheng. Physically guided neural network based on transfer learning (TL-PGNN) for hypersonic heat flux prediction. Communications in Computational Physics, 36(3):651–672, 2024

work page 2024
[4]

A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks

Suchuan Dong and Naxian Ni. A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks. Journal of Computational Physics, 435:110242, 2021

2021
[5]

Es’kin, Alexey O

Vasiliy A. Es’kin, Alexey O. Malkhanov, and Mikhail E. Smorkalov. Are two hidden layers still enough for the physics-informed neural networks? Journal of Computational Physics , 537:114085, 2025

work page 2025
[6]

Landscape and training regimes in deep learning

Mario Geiger, Leonardo Petrini, and Matthieu Wyart. Landscape and training regimes in deep learning. Landscape and training regimes in deep learning, 924:1–18, 2021. 21

work page 2021
[7]

N Ghia, and C

U Ghia, K. N Ghia, and C. T Shin. High-Re solutions for incompressible flow using the Navier-Stokes equations and a multigrid method.Journal of Computational Physics, 48(3):387– 411, 1982

work page 1982
[8]

Understanding the difficulty of training deep feedfor- ward neural networks

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010

work page 2010

Show all 38 references

[9]

Makridakis

Georgios Grekas and Charalambos G. Makridakis. Deep Ritz - Finite element methods: Neu- ral network methods trained with finite elements.Computer Methods in Applied Mechanics and Engineering, 437:117798, 2025

2025
[10]

How to start training: The effect of initialization and archi- tecture

Boris Hanin and David Rolnick. How to start training: The effect of initialization and archi- tecture. In Advances in Neural Information Processing Systems, volume 2018-December, pages 571–581. Neural information processing systems foundation, 2018

2018
[11]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016

2016
[12]

and Em Karniadakis, George

Ameya Jagtap, D. and Em Karniadakis, George. Extended physics-informed neural net- works (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. Communications in Computational Physics, 28(5):2002–2041, 2020

2002
[13]

Jagtap, Ehsan Kharazmi, and George Em Karniadakis

Ameya D. Jagtap, Ehsan Kharazmi, and George Em Karniadakis. Conservative physics- informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering , 365:113028, 2020

2020
[14]

Characterizing possible failure modes in physics-informed neural networks

Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. In Advances in Neural Information Processing Systems , volume 34, pages 26548–26560. Curran Associates, Inc., 2021

2021
[15]

Visualizing the Loss Landscape of Neural Nets

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Landscape of Neural Nets. In Advances in Neural Information Processing Systems, volume 31, pages 6391–6401. Curran Associates, Inc., 2018

2018
[16]

A Cyclical Learning Rate Method in Deep Learning Training

Jiaqi Li and Xiaodong Yang. A Cyclical Learning Rate Method in Deep Learning Training. In 2020 International Conference on Computer, Information and Telecommunication Systems (CITS) , pages 1–5, 2020

2020
[17]

Solving a class of multi- scale elliptic PDEs by Fourier-based mixed physics informed neural networks

Xi’an Li, Jinran Wu, Xin Tai, Jianhua Xu, and You-Gan Wang. Solving a class of multi- scale elliptic PDEs by Fourier-based mixed physics informed neural networks. Journal of Computational Physics, 508:113012, 2024

2024
[18]

Multi-Scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains

Ziqi Liu, Wei Cai, and Zhi-Qin John Xu. Multi-Scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains. Communications in Computa- tional Physics, 28(5):1970–2001, 2020

1970
[19]

Con- vergence analysis of pinns with over-parameterization

Chen Mo, Ding Zhao, Jiao Yuling, Lu Xiliang, Peiying Wu, and Jerry Zhijian, Yang. Con- vergence analysis of pinns with over-parameterization. Communications in Computational Physics, 37(4):942–974, 2025

2025
[20]

M. S. Nakhodnov, M. S. Kodryan, E. M. Lobacheva, and D. S. Vetrov. Loss Function Dy- namics and Landscape for Deep Neural Networks Trained with Quadratic Loss. Doklady Mathematics, 106(1):S43–S62, 2022

2022
[21]

Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 22 117(40):24652–24663, 2020

2020
[22]

On the Spectral Bias of Neural Networks

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the Spectral Bias of Neural Networks. In Proceed- ings of the 36th International Conference on Machine Learning, pages 5301–5310. PMLR, 2019
[23]

Physics-informed neural net- works: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural net- works: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019

2019
[24]

DGM: A deep learning algorithm for solv- ing partial differential equations

Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solv- ing partial differential equations. Journal of Computational Physics, 375:1339–1364, 2018

2018
[25]

PirateNets: Physics-informed deep learning with residual adaptive networks

Sifan Wang, Bowen Li, Yuhan Chen, and Paris Perdikaris. PirateNets: Physics-informed deep learning with residual adaptive networks. Journal of Machine Learning Research , 25(402):1–51, 2024

2024
[26]

Respecting causality for training physics-informed neural networks

Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality for training physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering , 421:116813, 2024

2024
[27]

An Expert’s Guide to Training Physics-informed Neural Networks, 2023

Sifan Wang, Shyam Sankaran, Hanwen Wang, and Paris Perdikaris. An Expert’s Guide to Training Physics-informed Neural Networks, 2023

2023
[28]

Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks

Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks. SIAM Journal on Scientific Computing , 43(5):A3055–A3081, 2021

2021
[29]

When and why PINNs fail to train: A neural tangent kernel perspective

Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768, 2022

2022
[30]

Bordas, and Chao Jiang

Wei Xiong, Xiangyun Long, St ´ephane P .A. Bordas, and Chao Jiang. The deep finite element method: A deep learning framework integrating the physics-informed neural networks with the finite element method. Computer Methods in Applied Mechanics and Engineering , 436:117681, 2025

2025
[31]

Finite neuron method and convergence analysis

Jinchao Xu. Finite neuron method and convergence analysis. Communications in Computa- tional Physics, 28(5):1707–1745, 2020

2020
[32]

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Net- works

Zhi-Qin John Xu. Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Net- works. Communications in Computational Physics, 28(5):1746–1767, 2020

2020
[33]

Overview Frequency Principle/Spectral Bias in Deep Learning

Zhi-Qin John Xu, Yaoyu Zhang, and Tao Luo. Overview Frequency Principle/Spectral Bias in Deep Learning. Communications on Applied Mathematics and Computation, 2024

2024
[34]

Training Behavior of Deep Neural Net- work in Frequency Domain

Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training Behavior of Deep Neural Net- work in Frequency Domain. In Tom Gedeon, Kok Wai Wong, and Minho Lee, editors,Neural Information Processing, pages 264–274, Cham, 2019. Springer International Publishing

2019
[35]

Effective Rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics

Jiang Yang, Yuxiang Zhao, and Quanhui Zhu. Effective Rank and the Staircase Phenomenon: New Insights into Neural Network Training Dynamics. arXiv preprint arXiv:2412.05144 , 2024

2024 arXiv
[36]

Accurate adaptive deep learning method for solving elliptic problems

Jinyong Ying, Xie Yaqi, Jiao Li, and Hongqiao Wang. Accurate adaptive deep learning method for solving elliptic problems. Communications in Computational Physics , 37(3):849– 876, 2025

2025
[37]

Why shallow networks struggle with approximating and learning high frequency: A numerical study.arXiv preprint arXiv:2306.17301, 2023

Shijun Zhang, Hongkai Zhao, Yimin Zhong, and Haomin Zhou. Why shallow networks struggle with approximating and learning high frequency: A numerical study.arXiv preprint arXiv:2306.17301, 2023

2023 arXiv
[38]

Loss jump during loss switch in solving pdes with neural networks

Wang Zhiwei, Lulu Zhang, Zhang Zhongwang, and John Zhi-Qin, Xu. Loss jump during loss switch in solving pdes with neural networks. Communications in Computational Physics, 36(4):1090–1112, 2024. 23 A Hyper-parameter Configurations This appendix provides the detailed hyper-para...

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Emergence of Invariance and Disentanglement in Deep Representations

Alessandro Achille and Stefano Soatto. Emergence of Invariance and Disentanglement in Deep Representations. Journal of Machine Learning Research, 19(50):1–34, 2018

work page 2018

[2] [2]

Mart ´ın

Santiago Badia, Wei Li, and Alberto F. Mart ´ın. Adaptive finite element interpolated neural networks. Computer Methods in Applied Mechanics and Engineering, 437:117806, 2025

work page 2025

[3] [3]

Physically guided neural network based on transfer learning (TL-PGNN) for hypersonic heat flux prediction

Biao Chen, Jifa Zhang, Shuai Zhang, Song Xiaoxiao, and Yao Zheng. Physically guided neural network based on transfer learning (TL-PGNN) for hypersonic heat flux prediction. Communications in Computational Physics, 36(3):651–672, 2024

work page 2024

[4] [4]

A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks

Suchuan Dong and Naxian Ni. A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks. Journal of Computational Physics, 435:110242, 2021

2021

[5] [5]

Es’kin, Alexey O

Vasiliy A. Es’kin, Alexey O. Malkhanov, and Mikhail E. Smorkalov. Are two hidden layers still enough for the physics-informed neural networks? Journal of Computational Physics , 537:114085, 2025

work page 2025

[6] [6]

Landscape and training regimes in deep learning

Mario Geiger, Leonardo Petrini, and Matthieu Wyart. Landscape and training regimes in deep learning. Landscape and training regimes in deep learning, 924:1–18, 2021. 21

work page 2021

[7] [7]

N Ghia, and C

U Ghia, K. N Ghia, and C. T Shin. High-Re solutions for incompressible flow using the Navier-Stokes equations and a multigrid method.Journal of Computational Physics, 48(3):387– 411, 1982

work page 1982

[8] [8]

Understanding the difficulty of training deep feedfor- ward neural networks

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010

work page 2010