Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Rank Inspired Neural Network for solving linear partial differential equations

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read The paper claims that decorrelating hidden-layer outputs before a least-squares solve, and stopping early on the PDE residual, makes the physics-informed extreme learning machine robust to random initialization.

desk verdict A plausible incremental improvement to PIELM whose abstract overclaims: only the early-stopping variant RINN-es is supported by the paper's own Table 7, and the evidence is single-run. read the letter →

arxiv 2506.17654 v1 pith:UT4CSWN2 submitted 2025-06-21 math.NA cs.NA

classification math.NAcs.NA MSC 65N3568T07
keywords physicsinformedextremelearningmachinerankinspiredneuralnetworkcovariance-drivenorthogonalizationinitializationsensitivitylinearpartialdifferentialequationsleast-squaressolverearlystoppingPDEcollocation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to fix the main weakness of the physics-informed extreme learning machine (PIELM) for linear partial differential equations: the outcome depends heavily on random hidden-layer initialization. It proposes a two-stage method, RINN, that first trains the hidden weights to make the hidden-layer basis approximately orthogonal at the collocation points, then solves for the output weights by least squares; an early-stopping variant, RINN-es, monitors the PDE residual and keeps the parameters where it is smallest. The paper's headline stability claim is carried by RINN-es, since the fixed-epoch RINN is itself reported to diverge under two of the three initialization ranges in Table 7. Across Poisson, advection, heat, and wave examples, RINN-es reports errors several orders of magnitude below PIELM and far less spread across initializations, which matters because it makes a fast least-squares neural PDE solver usable without hand-tuned initialization.

What carries the argument

The central object is the sample covariance matrix of the last hidden layer's outputs, $C = \Phi^\top\Phi/(K-1)$, whose off-diagonal entries approximate the $L^2$ inner products between neural basis functions at the collocation points. The machinery is the covariance-driven loss $L_{\text{total}} = \varepsilon L_{\text{diag}} + L_{\text{ortho}}$, where $L_{\text{ortho}} = \|C \odot (J - E)\|_F$ penalizes off-diagonal correlation and $L_{\text{diag}} = \sum_i |\log_{10}(C_{ii}^2)|$ pulls each basis function toward unit norm; training this loss is the preconditioning stage. The second stage is the classic PIELM least-squares solve for $\beta = H^\dagger S$. The early-stopping mechanism is the PDE residual $L_{\text{pde}}$ from equation (25), which the paper shows tracks the evaluation error during training and is therefore used to select the stopping epoch.

What would settle it

A direct check would be to compute the condition number of $H$ at the RINN-es optimal epoch and compare it with the condition number from a random initialization on the same PDE; if RINN-es improves accuracy while $\mathrm{cond}(H)$ does not improve, the claimed conditioning mechanism is not the cause of the improvement.

Watch

Extended reading notes

Core claim

On the paper's own terms: PIELM is accurate but fragile because randomly initialized hidden weights produce correlated neural basis functions. RINN adds a pretraining stage that minimizes the off-diagonal entries of the sample covariance matrix $C = \Phi^\top\Phi/(K-1)$ of the hidden-layer outputs, enforcing pairwise orthogonality of the basis at the collocation points, and also penalizes diagonal entries that deviate from unit norm. After this decorrelation stage, the hidden parameters are frozen and the output weights are obtained from the linear least-squares problem $H(\theta,X)\beta = S$. Because the orthogonalization loss carries no PDE information, overtraining can degrade the solution, and the paper's early-stopping variant RINN-es computes the PDE residual $L_{\text{pde}} = L_{\text{res}} + L_{\text{bcs}} + L_{\text{ics}}$ after each epoch and restores the parameters at which that residual was minimal. The claim is that this combination turns PIELM from an initialization-sensitive solver into one that maintains high accuracy across wide initialization ranges.

Load-bearing premise

The load-bearing premise is that making the raw hidden-layer outputs $\Phi$ orthogonal at the collocation points is enough to make the PDE collocation matrix $H = [A\Phi; B\Phi; \Phi]$ well-conditioned and accurate, even though $H$ applies differential operators to $\Phi$ and the paper gives no analysis linking orthogonality of $\Phi$ to conditioning of $H$.

Editorial extensions

If this is right

  • For the linear PDEs tested, RINN-es reports relative $L^2$ errors several orders of magnitude below PIELM on the same collocation points and network architecture.
  • The optimal early-stopping epochs differ strongly across initializations (epochs 6, 672, and 999 in Table 7), implying that fixed-epoch training is not a reliable protocol for this class of neural least-squares solvers.
  • The covariance-driven orthogonalization can be inserted as a preconditioning stage into the existing two-step ELM pipeline without changing the least-squares solve, so it is a drop-in modification for PIELM-type methods.
  • The experiments cover elliptic, advection, heat, and wave equations, so the claimed stabilization is not limited to a single equation type or regime.
  • Because the early-stopping criterion uses only the PDE residual, it requires no extra data beyond the collocation points already used by the solver.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is to apply the same decorrelation loss to the operator-applied matrix $H = [A\Phi; B\Phi; \Phi]$ rather than to $\Phi$ alone, since the conditioning that actually determines the least-squares error is the conditioning of $H$.
  • The reported failure of plain RINN under small initializations suggests that decorrelation alone can be harmful, and that early stopping may be acting as a regularizer rather than simply locating an optimum of the orthogonalization loss; the paper leaves this distinction open.
  • The two-stage structure is general enough to serve as an initialization scheme for other random-feature PDE solvers, such as randomized Petrov-Galerkin or discontinuous Galerkin variants, though that transfer is not demonstrated here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript introduces Rank Inspired Neural Networks (RINN) as a two-stage alternative to PIELM for linear PDEs. In the first stage, hidden-layer parameters are trained using a loss that penalizes off-diagonal entries of the sample covariance matrix C = Φ^T Φ/(K−1) and deviations of diagonal entries from unity; in the second stage, the PDE collocation system H(θ,X)β = S is solved by least squares. A variant, RINN-es, adds an early-stopping rule based on the PDE residual loss Lpde defined in Eq. (25). Experiments on 1D and 2D Poisson equations, advection, heat, and wave equations report relative L2 and L1 errors and compare with PIELM under several initialization ranges. The paper claims that RINN significantly reduces performance variability due to parameter initialization compared to PIELM.

Significance. If borne out, the proposed covariance-driven pretraining would be a practically useful, inexpensive way to stabilize PIELM, and the residual-based early-stopping mechanism is a sensible addition. The paper also documents the known initialization sensitivity of PIELM in Table 1. The central claim, however, is not currently supported by the evidence: plain RINN fails badly in part of Table 7, and no repeated-seed statistics are reported. The contribution therefore needs substantial strengthening before it can be assessed.

major comments (4)
  1. [Sec. 3.2, Table 7] The abstract's claim that 'RINN significantly reduces performance variability due to parameter initialization compared to PIELM' is contradicted by the plain RINN rows. Under U(−0.5, 0.5), RINN has EL2 = 1.16×10^1 versus PIELM's 7.80×10^−2, and under U(−2, 2), RINN has EL2 = 3.46×10^0 whereas PIELM has 8.36×10^0. Only the RINN-es variant is uniformly competitive. The paper should either restrict the central claim to RINN-es and explain why plain RINN is not covered, or report fixed-epoch RINN results for all tested problems and show that the failure in Table 7 is atypical.
  2. [Tables 1–7] Each table reports a single run per configuration. Since the paper's central claim is about variability across random initializations, single-run numbers cannot establish it. Please report means and standard deviations (or medians with quartiles) over multiple independent initializations, and clarify whether the reported errors are the best, median, or a single realization. This is essential for Table 7, where the conclusion is described in terms of robustness.
  3. [Sec. 2.2, Eqs. (10)–(15) and Eq. (6)] The orthogonalization objective acts on the raw hidden-output matrix Φ, whereas the least-squares system (6) is governed by H = [AΦ; BΦ; Φ], which contains derivatives. The manuscript asserts that decorrelating Φ improves representational ability and numerical stability, but no analysis, estimate, or numerical diagnostic connects orthogonality of Φ to the conditioning of H or to the resulting error. In addition, the text states that the 'primary objective' is to maximize the rank of Φ, but Algorithm 1 never maximizes a rank; it minimizes off-diagonal covariance. Either provide a supporting argument for the transfer from Φ-orthogonality to H-conditioning, or soften the claim and present evidence that the objective is the right one.
  4. [Sec. 3.1 and Algorithm 2] The early-stopping criterion selects the epoch that minimizes Lpde on the training points. Figure 7 shows this residual tracks the test error in one example, but no robustness evidence is given for other problems, and the procedure is a form of model selection on the training set. Moreover, in Table 7 under U(−2,2), the best epoch is 999 with patience 250 and maximum 1000 epochs, so no early stopping actually occurs; the algorithm simply returns the final iterate. Please report for each case whether termination was due to patience or to reaching E, and consider a small validation split or a separate stopping rule.
minor comments (5)
  1. [Sec. 2.2, page 6] 'potimize' should be 'optimize'; also 'PyTorch 3.8.20' is not a valid PyTorch version and should be corrected.
  2. [Example 2.5, Eq. (23)] The spatial domain is written as Ω=[0,1] but the boundary condition is u(−1,t)=u(1,t)=0; these are inconsistent and the intended interval or boundary indices should be fixed.
  3. [Sec. 2.3] The section promises a computational efficiency analysis, but no runtime, iteration-count, or floating-point operation data are reported; please add the promised comparison or remove the claim.
  4. [Figures 7–8 and Table 7] The panels would benefit from explicit labels, and the caption or text should state whether the reported errors are for a single seed; see Major Comment 2.
  5. [Eq. (14)] The definition of Ldiag uses log10(C_ii^2), which can be unstable if a diagonal entry is near zero; a small floor or a different normalization should be specified.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: RINN's covariance orthogonalization and early stopping are tested against analytic solutions, and no load-bearing derivation reduces to its own inputs.

full rationale

RINN's derivation chain does not exhibit circular reasoning. The covariance-driven loss L_total = epsilon * L_diag + L_ortho in Eq. (15) is minimized over hidden parameters theta before the PDE system H * beta = S in Eq. (6) is solved by least squares; the PDE constraints enter only in stage 2, so the orthogonalization objective is not defined in terms of the PDE solution or the reported errors. All accuracy claims are measured against analytic solutions (Tables 2-6) or a fixed reference solution (Table 7); no parameter is fitted to the target error. The early-stopping epoch in Algorithm 2 is selected by monitoring the PDE residual L_pde on the training collocation points, which is a standard model-selection heuristic and not a circular prediction: the reported EL2 and EL1 are evaluated on the analytic solution independently of the selection criterion, and the stopping rule does not use the test error. The only self-citation with author overlap, Ref. [17] cited in the introduction for a generalization error bound on PINNs, is contextual and not load-bearing for RINN's construction. The paper's internal inconsistency, where plain RINN performs worse than PIELM under U(-0.5,0.5) in Table 7 while the abstract claims reduced variability, is a correctness and reproducibility concern, not a circularity. No equation in the paper reduces by construction to its own input, and no prior-work uniqueness theorem or ansatz is invoked to force the proposed form.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The method rests on heuristic assumptions rather than proven guarantees. The key free parameters, including epsilon, epochs, patience, architecture, and initialization width, are selected per experiment. The link between orthogonality of raw basis functions and accuracy of the differential operator collocation system is asserted, not derived. No new physical entities are introduced.

free parameters (7)
  • regularization coefficient epsilon = 0.1 or 0.01 per example
    Weights the diagonal penalty against the off-diagonal loss in L_total = epsilon L_diag + L_ortho; chosen per experiment.
  • learning rate eta = 1e-3 throughout
    Adam learning rate used for the stage-1 orthogonalization optimization.
  • max epochs E = 500, 1000, or 2000 depending on example
    Stopping horizon for orthogonalization training; results depend on it.
  • early stopping patience P = 250 (RINN-es)
    Number of no-improvement epochs before restoring best parameters in Algorithm 2.
  • network architecture = [1,128,1], [2,512,1024,1], etc.
    Basis function dimension and representational capacity; varied per experiment.
  • initialization distribution width a = U(-0.5,0.5), U(-1,1), U(-2,2), U(-20,20)
    Random parameter range; the robustness claim is about performance across this choice.
  • collocation point counts = Kres=1024/2048, Kbcs=2/2048/4096, Kics=1024
    Number of constraints in the covariance and least-squares systems.
assumptions (4)
  • domain assumption Neural basis orthogonality improves the conditioning and approximation power of the collocation system
    Invoked in Section 2.2 as the motivation for L_total; no theorem or numerical conditioning analysis is provided.
  • domain assumption Discrete sample covariance on collocation points is a valid proxy for L2 orthogonality, and minimizing its off-diagonal entries improves PDE solves
    Equations (9)-(12) assume the collocation sum approximates the L2 inner product; this is used as the training objective.
  • domain assumption PDE residual loss minimum during orthogonalization is a safe early-stopping signal
    Section 3.1 argues from a single empirical example that L_pde tracks generalization error; no statistical justification is given.
  • standard math Universal approximation capability of neural networks justifies the ELM ansatz
    Cybenko's theorem is cited in the Introduction to support using neural networks as global collocation approximators.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rank Inspired Neural Network for solving linear partial differential equations." pith.science (2026). https://pith.science/paper/UT4CSWN2

@misc{pith2026250617654,
  author       = {Pith},
  title        = {Pith review of: Rank Inspired Neural Network for solving linear partial differential equations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UT4CSWN2}},
  note         = {Machine review of arXiv:2506.17654}
}
read the original abstract

This paper proposes a rank inspired neural network (RINN) to tackle the initialization sensitivity issue of physics informed extreme learning machines (PIELM) when numerically solving partial differential equations (PDEs). Unlike PIELM which randomly initializes the parameters of its hidden layers, RINN incorporates a preconditioning stage. In this stage, covariance-driven regularization is employed to optimize the orthogonality of the basis functions generated by the last hidden layer. The key innovation lies in minimizing the off-diagonal elements of the covariance matrix derived from the hidden-layer output. By doing so, pairwise orthogonality constraints across collocation points are enforced which effectively enhances both the numerical stability and the approximation ability of the optimized function space.The RINN algorithm unfolds in two sequential stages. First, it conducts a non-linear optimization process to orthogonalize the basis functions. Subsequently, it solves the PDE constraints using linear least-squares method. Extensive numerical experiments demonstrate that RINN significantly reduces performance variability due to parameter initialization compared to PIELM. Incorporating an early stopping mechanism based on PDE loss further improves stability, ensuring consistently high accuracy across diverse initialization settings.

Figures

Figures reproduced from arXiv: 2506.17654 by the authors.

Figure 1
Figure 1. Structure of ELM. The input layer contains d + 1 neurons which correspond to spatial coordinates x = (x1, . . . , xd) and temporal coordinate t. Each hidden layer, indexed by ℓ ∈ {1, . . . , L}, receives the output from the previous layer and processes it using a linear transformation followed by a nonlinear activation function. This operation is defined as: zℓ = σ  W(ℓ) zℓ−1 + b (ℓ)  , W(ℓ) ∈ R Nℓ×Nℓ−1 , b (ℓ) ∈ … view at source ↗
Figure 2
Figure 2. RINN results for Example 2.1. Left column shows exact and predicted solution; right column shows pointwise absolute errors. Rows from top to bottom correspond to the solution (17a) and the solution (17b) [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. RINN results for Example 2.2. Each column shows: (left) analytical solution, (middle) predicted solution, (right) pointwise absolute error. Rows from top to bottom correspond to the high-frequency solution (18a) and the multi-scale solution (18b). Example 2.3. (1D advection equations) Consider the following 1D advection equation defined on Ω = [−1, 1], I = [0, 1],    ut(x, t) = −cux(x, t), in Ω × I, u(−1, t) = … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: RINN results for Example 2.3. Each column shows: (left) analytical solution, (middle) predicted solution, (right) pointwise absolute error. Rows from top to bottom correspond to initial condition (20a) and (20b) [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: RINN results for Example 2.4. Each column shows: (left) analytical solution, (middle) predicted solution, (right) pointwise absolute error. Rows from top to bottom correspond to (22) with k = 2 and k = 6 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: RINN results Example 2.5. Each column shows: (left) analytical solu￾tion, (middle) predicted solution, (right) pointwise absolute error. Rows from top to bottom correspond to initial condition (24a) and (24b) [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Evolution of the relative L 2 error and mean absolute error on evaluation points, alongside the PDE residual loss on training points. The PDE residual loss Lpde is defined by Lpde = Lres + Lbcs where Lres = vuut 1 Kres X Kres i=1 [PITH_FULL_IMAGE:figures/full_fig_p014…
Figure 8
Figure 8. Figure 8: Numerical solutions and pointwise absolute error distributions for the 2D Poisson equation with initialization U(−1, 1): PIELM (top panel), RINN (middle panel), and RINN-es (bottom panel). 4. Conclusion The paper introduces the Rank Inspired Neural Network (RINN) frame…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 26 canonical work pages

  1. [1]

    Calabr` o, G

    F. Calabr` o, G. Fabiani and C. Siettos. Extreme learning machine collocation for the numerical solution of elliptic PDEs with sharp gradients. Computer Methods in Applied Mechanics and Engineering, 387:114188, 2021

  2. [2]

    G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303–314, 1989

  3. [3]

    Dwivedi and B

    V. Dwivedi and B. Srinivasan. Physics-informed extreme learning machine (PIELM)—a rapid method for the numerical solution of partial differential equations. Neurocomputing, 391:96–118, 2020

  4. [4]

    Dong and Z

    S. Dong and Z. Li. Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations. Computer Methods in Applied Mechanics and Engineering, 387:114129, 2021

  5. [5]

    Dong and J

    S. Dong and J. Yang. On computing the hyperparameter of extreme learning machines: Algorithm and ap- plication to computational PDEs, and comparison with classical and high-order finite elements. Journal of Computational Physics, 463:111290, 2022

  6. [6]

    G. B. Huang, Q. Y. Zhu and C. K. Siew. Extreme learning machine: theory and applications. Neurocomputing, 70(1–3):489–501, 2006

  7. [7]

    G. B. Huang, L. Chen and C. K. Siew. Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Transactions on Neural Networks, 17(4):879–892, 2006

  8. [8]

    A. D. Jagtap, E. Kharazmi and G. E. Karniadakis. Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365:113028, 2020

Show all 28 references
  1. [9]

    A. D. Jagtap and G. E. Karniadakis. Extended physics-informed neural networks (XPINNs): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. Communications in Computational Physics, 28(5):2002–2041, 2020

  2. [10]

    Kharazmi, Z

    E. Kharazmi, Z. Zhang and G. E. Karniadakis. Variational physics-informed neural networks for solving partial differential equations. arXiv preprint arXiv:1912.00873, 2019

  3. [11]

    Kharazmi, Z

    E. Kharazmi, Z. Zhang and G. E. Karniadakis. hp-VPINNs: Variational physics-informed neural networks with domain decomposition. Computer Methods in Applied Mechanics and Engineering, 374:113547, 2021

  4. [12]

    I. E. Lagaris, A. Likas and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5):987–1000, 1998

  5. [13]

    I. E. Lagaris, A. C. Likas and D. G. Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11(5):1041–1049, 2000

  6. [14]

    R. J. LeVeque. Finite difference methods for ordinary and partial differential equations: steady-state and time-dependent problems. SIAM, 2007

  7. [15]

    Y. Liu, W. Liu, X. Yan, S. Guo and C. Zhang. Adaptive transfer learning for PINN. Journal of Computational Physics, 490:112291, 2023

  8. [16]

    Ming and Y

    P. Ming and Y. Liao. Deep Nitsche Method: Deep Ritz Method with Essential Boundary Conditions. Commu- nications in Computational Physics, 29(5):1365–1384, 2021

  9. [17]

    Y. Qian, Y. Zhang, Y. Huang and S. Dong. Physics-informed neural networks for approximating dynamic (hyperbolic) PDEs of second order in time: Error analysis and algorithms. Journal of Computational Physics, 495:112527, 2023

  10. [18]

    Raissi, P

    M. Raissi, P. Perdikaris and G. E. Karniadakis. Physics-informed neural networks: A deep learning frame- work for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019

  11. [19]

    Rahaman, A

    N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio and A. Courville. On the spectral bias of neural networks. In Proceedings of the International Conference on Machine Learning, pages 5301–5310, 2019

  12. [20]

    S. S. Rao. The finite element method in engineering. Elsevier, 2010

  13. [21]

    Shang, F

    Y. Shang, F. Wang and J. Sun. Deep Petrov-Galerkin method for solving partial differential equations. arXiv preprint arXiv:2201.12995, 2022

  14. [22]

    Sirignano and K

    J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339–1364, 2018

  15. [23]

    J. Sun, S. Dong and F. Wang. Local randomized neural networks with discontinuous Galerkin methods for partial differential equations. Journal of Computational and Applied Mathematics, 445:115830, 2024

  16. [24]

    H. K. Versteeg. An introduction to computational fluid dynamics: the finite volume method. Pearson Education India, 2007. RINN FOR PDES 19

  17. [25]

    Wang and S

    Y. Wang and S. Dong. An extreme learning machine-based method for computational PDEs in higher dimen- sions. Computer Methods in Applied Mechanics and Engineering, 418:116578, 2024

  18. [26]

    C. Wu, M. Zhu, Q. Tan, Y. Kartha and L. Lu. A comprehensive study of non-adaptive and residual-based adap- tive sampling for physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 403:115671, 2023

  19. [27]

    W. E and B. Yu. The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems. Communications in Mathematics and Statistics, 6(1):1–12, 2018

  20. [28]

    Y. Zang, G. Bao, X. Ye and H. Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411:109409, 2020. 1 School of Mathematics and Computational Science, Xiangtan University, Xiangtan 411105, P.R.China 2 National ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.