REVIEW 4 major objections 5 minor 1 cited by
A note on the physical interpretation of neural PDE's
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper shows that the forward pass of a constant-width network is exactly an Euler step of a relaxation dynamics, so weights can be read as physical information-propagation parameters.
desk verdict A clean but largely definitional re-framing of a constant-width MLP as a relaxation PDE; the advertised physical weight interpretation does not survive the paper's own discrete delta-basis, though the formal analogy is sound. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the relaxation-form update $z(t+\Delta t)=(1-\omega)z+\omega f[Z]$ with $Z=Wz-b$ and $\omega=\gamma\Delta t$, obtained by Euler-marching the neural PDE $\partial_t z=-\gamma(z-f[Wz-b])$. The work it does is to fix the identification of the activation output with the local equilibrium $z_{\mathrm{eq}}$, turning the weight matrix into an integral kernel. The companion identity is the moment expansion $Z(q)=W_0(q)z+W_1(q)\partial_q z+\frac{W_2(q)}{2}\partial_{qq}z+\cdots$, which converts the nonlocal operator into local transport coefficients; this Taylor expansion is what allows the physical reading of the weights and what supports the reduced-parameter training strategy.
What would settle it
Take a trained transformer and test the moment reconstruction: compare the actual $Z_i=\sum_j W_{ij}z_j-b_i$ with the truncated moment series $W_0 z_i+W_1\partial z_i+\frac{1}{2}W_2\partial^2 z_i$ on the training data; if the residual is large, the PDE-generator interpretation fails for that network. A second decisive test is to train only the three local fields $U(q),D(q),R(q)$ on a task where full-matrix training succeeds; failure to approach the target would refute the reduced-parameter claim.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the ML forward update is not merely similar to a dynamical system but is one: marching $\partial_t z=-\gamma(z-f[Wz-b])$ forward with a forward-Euler step gives $z(t+\Delta t)=(1-\omega)z+\omega f[Wz-b]$ with $\omega=\gamma\Delta t$, which becomes the standard layer update for $\omega=1$. The paper identifies $f[Wz-b]$ with the local equilibrium $z_{\mathrm{eq}}$ of the corresponding Liouville dynamics, so the self-consistent fixed point $z^*=f[W^*z^*-b]$ is the attractor toward which the forward pass relaxes. Expanding the kernel-transformed signal in a Taylor series defines the moments $W_k(q)=\int r^k W(q,q+r)\,dr$, with $W_0$, $W_1$, and $W_2$ interpreted as local amplitude rescaling, propagation speed, and diffusion coefficient; even higher moments act as generalized diffusion and odd ones as dispersion. The weight matrix therefore acts as a PDE generator, and the nonlinear activation scrambles the resulting linear transport terms, producing a nonlinear neural PDE.
Load-bearing premise
The physical reading depends on the signal varying smoothly enough to expand locally, and on the weight matrix having well-defined low-order spatial moments; when the standard update treats weights as point samples, neither condition is guaranteed.
Editorial extensions
If this is right
- A trained constant-width network can be read as a discrete information-transport process, so its weight blocks have concrete names: amplitude, speed, diffusion, and dispersion.
- Training can target the low-order moments of the weight kernel instead of every entry; for a local advection-diffusion-reaction process this means three fields $U(q),D(q),R(q)$, cutting $O(N^2)$ parameters to $O(N)$ (or $O(Nd^2)$ in $d$ dimensions).
- The relaxation parameter $\omega=\gamma\Delta t$ becomes an optimization parameter with a physical meaning (dissipation), and tuning it during training is a plausible new control knob.
- The self-consistent attractor equation $z^*=f[W^*z^*-b]$ gives a criterion for the optimal depth of the circuit: there is no reason to keep running layers once the layer-by-layer loss has converged.
- Smooth functional-basis discretizations generalize the ML update to the quadrature form of Eq. (24), offering a path to smoother weights at modest extra cost.
Reading between the lines
- An implicit diagnostic: compute the empirical moments $W_0,W_1,W_2$ of trained transformer weight matrices and check whether they reconstruct the $W$-transformed signal on the data manifold; if this fails on real architectures, the physical reading is not automatically valid for them.
- The paper leaves open a testable prediction: if the reduced-parameter ADR network reaches a comparable target distribution on standard benchmarks, the moment-based proposal is a genuine alternative to full-matrix training; the paper does not provide such benchmarks.
- The relaxation parameter suggests an annealing schedule: slowly decreasing $\omega$ across training would mimic a dissipative cooling process, potentially suppressing overfitting; this is not explored in the paper.
- Because the same relaxation form underlies kinetic transport solvers, existing computational machinery for those solvers could be reused as network architectures or as initialization schemes—an implication the paper gestures toward without developing.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This note argues that the forward pass of a constant-width neural network (which the author calls a "transformer") with the layer update z_{l+1} = f(W z_l - b) is exactly a forward-Euler discretization of the relaxation dynamics ∂_t z = -γ(z - f[Wz - b]) when ω = γΔt = 1. The paper identifies the activation output f[Wz - b] with the local equilibrium z_eq, expands the kernel transform z̄(q) = ∫ W(q,q+r)z(q+r)dr in a Taylor series to obtain moments W0, W1, and W2, and interprets these moments as local amplitude rescaling, propagation speed, and diffusion. It then proposes that training can be restricted to these moments rather than to all matrix entries, and illustrates the idea with constant-coefficient advection-diffusion-reaction weights. Special cases (identity, local ODE, linear PDE, full neural PDE) and the delta-basis discretization are discussed.
Significance. The formal Euler-map equivalence in Eq. (19) is correct under the stated idealizations and provides a clean dictionary between a simple feedforward network and a discrete relaxation system; the connection to lattice Boltzmann methods is a useful and physically motivated observation. The moment expansion in Eq. (16) is a standard and correctly applied identity, and the idea of parameterizing weights by a few kernel moments is intellectually appealing for explainability and parameter reduction. However, the paper does not validate any of the claimed advantages, and the central interpretive step — reading actual discrete weight matrices as smooth local advection/diffusion fields — is not justified for the standard ML setting. The identification z_eq = f[Z] is by construction rather than a derived physical fact, and the delta-basis discretization in Sec. 5.2 destroys the pointwise smoothness needed for the moment interpretation. The scope is also narrower than claimed, since the architecture analyzed is a constant-width feedforward network, not the transformer architecture usually meant in current ML.
major comments (4)
- [Sec. 3, Eqs. (10)–(13)] The identification of the model function with the local equilibrium is by construction, not a discovery. Eq. (8) defines the relaxation form with z_eq = z - L(z), and Eq. (10) then posits ∂_t z = -γ(z - f[Z]), which forces L(z) = z - f[Z] and hence z_eq = f[Z] by Eq. (12). The statement that the local equilibrium is the ML target is therefore a definitional dictionary, not an empirically established physical equivalence. The paper should explicitly present this as a formal analogy and avoid the word "substantial" without additional support.
- [Sec. 5.2, Eqs. (16)–(17) with delta-basis choice] The moment interpretation does not survive the standard discrete representation. With φ_i(q) = δ(q - q_i), the kernel becomes W(q,q') = Σ_ij δ(q - q_i) W_ij δ(q' - q_j), so the moments W_k(q) = ∫ r^k W(q,q+r) dr are Dirac combs supported at the grid points, not continuous local fields. Consequently Eq. (16) does not define a pointwise PDE with local amplitude rescaling, advection speed, and diffusion coefficient. The assertion in Sec. 5.2 that this is "no serious problem" because only the signal is differentiated is insufficient: the coefficients of the Taylor expansion are themselves distributions, so the advertised physical interpretation is not well defined for the discrete ML update. The quadrature formulation in Eqs. (23)–(24) could rescue the interpretation by providing smooth kernels, but the paper does not implement or test it.
- [Sec. 3 and Sec. 7.1 (definition of "transformer")] The paper defines "transformers" as networks with a constant number of weights across layers (Sec. 3), which is a fully connected feedforward chain with equal layer widths. This excludes the attention mechanism, residual connections, and layer normalization that constitute the transformer architecture of Vaswani et al. (Ref. [18]). As a result, the claim that the analogy bears on "current ML applications" and "LLM-based transformers" is not supported. The analysis applies to the specific chain in Eqs. (1)–(3), and the paper should either restrict its claims to that architecture or extend the analogy to attention and residual updates.
- [Secs. 7–8 (proposed moment-based reduction)] The claimed advantages — training on moments rather than individual weights, and using ω as a useful optimization parameter — are not tested. The advection-diffusion-reaction example in Sec. 7 uses a constant-coefficient tridiagonal weight matrix and explicitly notes that "there is no guarantee that a generic target can be reached" at a given time; the subsequent leap to local and time-dependent coefficients, culminating in the O(N d²) parameter-count estimate and the statement that the approach "should be well positioned" to reach many targets, is speculation. The manuscript itself ends Sec. 7 with "Future simulation work will tell," which is a candid admission that the central proposal remains unvalidated. A proof-of-concept on a simple learning task, even one-dimensional, would be needed to make the parameter-reduction claim load-bearing.
minor comments (5)
- [Sec. 5.1, Eq. (18)] Eq. (18) is missing the current state z(t) on the right-hand side. The Euler update should read z(t + Δt) = z(t) - γΔt(z - f[Z]); as written, the equation does not reduce to Eq. (19).
- [Sec. 5.2, Eq. (21)] The projection in Eq. (21) is written with repeated indices but no explicit summation, and the mass matrix definition M_ij = ∫ φ_i(q)φ_j(q')dq' has an inconsistent integration measure. It should presumably be M_ij = ∫ φ_i(q)φ_j(q)dq.
- [Sec. 6, cluster integration paragraph] The word "iformally" is a typo for "informally". Additionally, the sentence claiming that the factor V_c/N_c "has been incorporated in the weights and biases" is not derived; a short clarification would help.
- [Sec. 7, moments of the tridiagonal kernel] The moments W1 = B - A and W2 = A + B are grid-scaled quantities (U/Δ and 2D/Δ², respectively), not the physical advection speed and diffusion coefficient. The text should state the conversion factor explicitly to avoid implying that the discrete weights directly equal the continuum coefficients.
- [Introduction and Secs. 3–4] The paper repeatedly contrasts "insight" and "control" and uses informal language (e.g., "golden nuggets," "to them 'too much is not enough'"). In a journal report these stylistic choices are acceptable for a perspective note, but a few equations for the path-counting claim (N^P = N^L) would make the quantitative assertions transparent.
Circularity Check
Core 'model function = local equilibrium' claim is definitional; PDE moment interpretation is independent.
-
self definitional
[Section 3, Eqs. (8)-(13); echoed in Section 8 summary]
"Under such an assumption, we can always recast the Liouville dynamics in relaxation form: ∂tz = −γ(z − zeq) where, by construction, the local equilbrium is given by zeq = z − L(z). ... Direct identification of (10) with (8) delivers the expression of the local equilibrium in terms of the activation function, namely zeq(q,t) = z − L(z) = f[Z(q,t)]."
The paper's central identification—that the ML model function f[Wz−b] is the local equilibrium of the discrete dynamics—is not derived from independent dynamical facts. The local equilibrium is introduced by construction as zeq = z − L(z), and then the Liouville operator is chosen as L(z) = z − f[Z] (Eq. 13). Hence the statement 'zeq = f[Z]' is true by definition, and the abstract/summary's claim that the analogy 'identifies the model function of the forward ML step with the local attractor' is a restatement of that definition rather than a substantive result. The forward-Euler equivalence (Eq. 19 with ω=1) is a separate exact reformulation, and the moment expansion (Eqs.
full rationale
The forward-Euler equivalence in Eq. (19) with ω = 1 is not circular: it is an exact algebraic rearrangement of the standard layer update z_{l+1} = f[W z_l − b]. The moment interpretation in Section 3.1 is also not circular; identifying W0, W1, W2 as amplitude rescaling, advection speed, and diffusion coefficient is a standard Taylor expansion of the convolution-like kernel action. The genuinely circular element is the paper's banner claim that the ML model function equals the local equilibrium: that equality is manufactured by defining zeq = z − L(z) and then setting L(z) = z − f[Z]. No load-bearing self-citation chain is present: the author's own references on lattice Boltzmann ([13], [15]) are used only as background for relaxation parameters, not to justify the central analogy. The objection that the δ-basis discretization in Section 5.2 makes the kernel moments distribution-valued is a validity concern about the physical interpretation in the discrete setting, not a circularity, so it is not scored here. Overall the paper contains a definitional framing presented as a novel identification, but it also contains independent, verifiable mathematical observations; hence a moderate score of 4 is appropriate.
Assumptions & free parameters
assumptions (4)
- ad hoc to paper The neural layer index can be treated as a continuous time coordinate, so the discrete update is a forward Euler discretization of a PDE with a fixed kernel W(q,q').
- domain assumption The signal z(q,t) is smooth enough for Taylor expansion in the displacement r.
- domain assumption The kernel moments W_k(q) = integral r^k W(q,q+r) dr exist.
- domain assumption A local attractor z* exists, is unique, and is a meaningful target for the ML procedure.
Cite this review
Pith. "Pith review of A note on the physical interpretation of neural PDE's." pith.science (2026). https://pith.science/paper/PWAOWFGF
@misc{pith2026250206739,
author = {Pith},
title = {Pith review of: A note on the physical interpretation of neural PDE's},
year = {2026},
howpublished = {\url{https://pith.science/paper/PWAOWFGF}},
note = {Machine review of arXiv:2502.06739}
}
read the original abstract
We highlight a formal and substantial analogy between Machine Learning (ML) algorithms and discrete dynamical systems (DDS) in relaxation form. The analogy offers a transparent interpretation of the weights in terms of physical information-propagation processes and identifies the model function of the forward ML step with the local attractor of the corresponding discrete dynamics. Besides improving the explainability of current ML applications, this analogy may also facilitate the development of a new class ML algorithms with a reduced number of weights.
Forward citations
Cited by 1 Pith paper
-
The wall confronting large language models
LLM scaling exponents near 0.1 imply that reducing loss tenfold would need 10^10 more compute, making scientific-grade reliability unreachable by brute-force scaling.
Reference graph
Works this paper leans on
-
[18]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A . N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advances in Neural Infor- mation Processing Systems, 30. Curran Associates, Inc. arXiv:17 06.03762 (2017)
work page 2017
-
[1]
Anderson, The end of theory: the data deluge makes the scie ntific method obsolete
C. Anderson, The end of theory: the data deluge makes the scie ntific method obsolete. Wired; 16. https://www .wired.com/2008/06/pb- theory/ (2008)
work page 2008
-
[2]
Bellman, Dynamic Programming, Princeton University Press, Pr ince- ton, NJ, (1957)
R. Bellman, Dynamic Programming, Princeton University Press, Pr ince- ton, NJ, (1957)
work page 1957
-
[3]
F. Bouchut, Y. Jobic , R. Natalini, R. Occelli, V. Pavan, Second-or der en- tropy satisfying BGK-FVS schemes for incompressible Navier-Stok es equa- tions, SMAI Journal of Computational Mathematics, Vol. 4, 1-56 ( 2018)
work page 2018
- [4]
-
[5]
R. Car, M. Parrinello, Unified Approach for Molecular Dynamics and Density-Functional Theory, Phys. Rev. Lett. 55, 2471 (1985)
work page 1985
-
[6]
T Chotibut, DR Nelson, Population genetics with fluctuating popula tion sizes, Journal of Statistical Physics 167 (3), 777-791, 2017
work page 2017
-
[7]
PV Coveney, ER Dougherty, RR Highfield, Big data need big theory too, Philosophical Transactions of the Royal Society A: Mathematical, P hysical and Engineering Sciences; 374:20160153 (2016)
work page 2016
Show all 22 references
-
[8]
Domingos, The master algorithm: How the quest for the ultimat e learn- ing machine will remake our world
P. Domingos, The master algorithm: How the quest for the ultimat e learn- ing machine will remake our world. New York: Basic Books (2015)
2015
-
[9]
Frenkel, B
D. Frenkel, B. Smit, Understanding Molecular Simulation, Elsevier, New York, (2001)
2001
-
[10]
J Jumper, R Evans, A Pritzel, T Green, M Figurnov, O Ronneberg er, K Tunyasuvunakool, R Bates, A ˇZ ´ ıdek, A Potapenko, A Bridgland, C Meyer, S A. A. Kohl, AJ. Ballard, A Cowie, B Romera-Paredes, S Nikolov, R Jain , J Adler, T Back, S Petersen, D Reiman, E Clancy, M Zielinski...
2021
-
[11]
LeCun, J
Y. LeCun, J. Bengio and G. Hilton, Deep Learning, Nature 521 (7 553), 436-444 (2015)
2015
-
[12]
Y LeCun, Y Bengio, Convolutional networks for images, speech , and time series, The handbook of brain theory and neural networks 3361 ( 10), 1995
1995
-
[13]
Rasin, S
I. Rasin, S. Succi and W. Miller, Phase-field lattice kinetic scheme for the numerical simulation of dendritic growth, Physical Review E—Statist ical, Nonlinear, and Soft Matter Physics, 72, 6, 066705 (2005) 11
2005
-
[14]
Rumelhart, G
D.E. Rumelhart, G. Hinton and R.J. Williams, Learning representat ions by back-propagating errors, Nature, 323 (9) 533, (1986)
1986
-
[15]
Succi, The Lattice Boltzmann Equation: Theory and Applicatio ns Ox- ford: Oxford University Press (2001)
S. Succi, The Lattice Boltzmann Equation: Theory and Applicatio ns Ox- ford: Oxford University Press (2001)
2001
-
[16]
S.Succi, PV Coveney, Big data: the end of the scientific method? Philo- sophical Transactions of the Royal Society A; 377:201801 (2019)
2019
-
[17]
Succi, Chatbots and Zero Sales Resistence, Frontiers in Phy sics 12, 1484701 (2024)
S. Succi, Chatbots and Zero Sales Resistence, Frontiers in Phy sics 12, 1484701 (2024)
2024
-
[19]
Weinan E, A proposal on machine learning via dynamical systems. Comm. Math. Stat., 5(1), 1-11, 2017
2017
-
[20]
Qianxiao Li Weinan E, Machine Learning and Dynamical Sys- tems, November 01, 2021 https://www.siam.org/publications/siam- news/articles/machine-learning-and-dynamical-systems/
2021
-
[21]
Weinan, E; Ma, C; (...); Wojtowytsch, S, Towards a Mathematica l Under- standing of Neural Network-Based Machine Learning: What We Kno w and What We Don’t CSIAM TRANSACTIONS ON APPLIED MATHEMAT- ICS, 1 (4) , pp.561-615, 2020
2020
-
[22]
Hui Yin, Amir Aryani, Stephen Petrie, Aishwarya Nambissan, Alan d Astudillo, Shengyuan Cao, A Rapid Review of Clustering Algorithms, cs.arXiv:2401.07389 12
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.