Pith. sign in

REVIEW 4 major objections 6 minor 60 references

Unsupervised Adaptation of PDE Foundation Models

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A PDE foundation model can be adapted to unseen equations using only the equation and boundary data.

desk verdict Useful, honest empirical paper on residual-based adaptation of PDE foundation models, but the machine-precision residual claim in B.4 contradicts the finite-difference stencils in A.5 and the 'first attempt' framing overstates novelty. read the letter →

arxiv 2608.07053 v1 pith:IQGQ6A3S submitted 2026-08-07 cs.AI

Ziye Song , Zhao Wei , Xin Yu , Ivor Tsang , Yueming Lyu This is my paper · ORCID
classification cs.AI
keywords unsupervisedPDEadaptationfoundationmodelsphysics-informedfine-tuninglow-rankNewton-Schulzorthogonalizationfinite-differenceresidualneuraloperatorsneighborhoodattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a pretrained PDE foundation model can be adapted to a new, unseen equation without a single interior ground-truth solution, using only the governing PDE and boundary observations. The authors pretrain a neighborhood attention Transformer on a broad set of time-dependent PDEs, then fine-tune low-rank adapters against an objective that penalizes the discrete PDE residual in the interior and the mismatch on a boundary band. They report that this physics-only adaptation stays within a factor of 2.5 of supervised LoRA fine-tuning on seven of eight 2D benchmarks and beats at least one supervised neural-operator baseline on nine of eleven datasets. If true, this matters because it removes the main practical bottleneck of PDE foundation models: dense, expensive solution data for every new equation.

What carries the argument

The load-bearing mechanism is the UPAO objective, a two-term loss $L = \lambda_{\mathrm{BC}} L_{\mathrm{BC}} + L_{\mathrm{PDE}}$ in which the PDE-residual term is evaluated by finite-difference stencils on interior points and the boundary term supervises a one-grid-point peripheral band. Carrying the optimization is NSLoRA, a low-rank adapter whose two factors are orthogonalized in the forward pass by Newton-Schulz iteration and rescaled by their Frobenius norms, which the paper claims counteracts the collapse of the effective rank of $\Delta W$ and rebalances learning of weaker physical channels. A frozen pretrained neighborhood attention Transformer supplies the transferable representation that the adapters specialize, and its localized attention kernel is what lets the model accept inputs at different spatial resolutions.

What would settle it

Take the analytical 2D advection-diffusion benchmark the paper uses, evaluate the exact same finite-difference residual stencils on the highest-wavenumber modes ($|m|,|n|\le 3$) at the 256$\times$256 resolution, and compare the measured residual with the true PDE residual; if the truncation error from the second-order stencils is of the same order as the reported VRMSE improvements, roughly 0.002 to 0.02, then UPAO is minimizing a biased objective and its success should not transfer to other equations or resolutions.

Watch

Extended reading notes

Core claim

The paper's central claim is that unsupervised adaptation of a pretrained PDE foundation model to unseen equations works by enforcing the equations themselves: optimize only the PDE residual on the interior and the boundary condition on a narrow band. The paper introduces UPAO as this two-term loss, with finite-difference residuals computed using central differences for diffusive terms and conservative upwind discretization for convective terms, together with per-sub-equation normalization so that no physical quantity dominates the gradient signal. It further claims that standard LoRA suffers rank collapse during this adaptation, with measured stable ranks between 1.31 and 2.20 against a nominal rank budget of 16, and that orthogonalizing the low-rank factors with a Newton-Schulz iteration, called NSLoRA, rebalances learning across physical quantities and improves VRMSE on all eight 2D benchmarks. With the pretrained neighborhood attention backbone frozen except for the adapters, the paper presents this as the first adaptation of a PDE foundation model strictly through governing PDEs and boundary conditions.

Load-bearing premise

The paper's results rest on the premise that the discrete finite-difference residual is a faithful measure of how wrong the prediction is on the given data grids, and that the pretrained backbone already carries enough transferable structure that minimizing that residual yields accurate solutions.

Editorial extensions

If this is right

  • A PDE foundation model can be specialized to a new equation using only the equation and boundary observations, removing the dense solution data requirement from the adaptation stage.
  • The residual-based objective is nearly as informative as supervised labels on these benchmarks: the gap to supervised LoRA is bounded by 2.5x on seven of eight 2D datasets.
  • Orthogonalized low-rank adapters (NSLoRA) are a drop-in replacement for standard LoRA that improve accuracy by about 4 percent on average with the same rank budget, with gains concentrated on channels such as pressure, tracer, and one velocity component that standard LoRA under-learns.
  • The success of UPAO depends on the pretrained initialization: optimizing from random weights under the same objective degrades VRMSE by more than an order of magnitude on the exact-solution benchmarks and by factors of 1.2 to 5.3 on The Well benchmarks.
  • UPAO's benefit is backbone-dependent: it consistently helps the pretrained backbone and TFNO, but is marginal or negative on CNextU-Net, so the transferable representation, not the objective alone, carries the result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the paper leaves implicit is treating partially unknown physics: if the residual is computed with a surrogate for the missing term, UPAO could in principle estimate that term by minimizing the same objective, turning unsupervised adaptation into a parameter-identification scheme.
  • Because UPAO's residual is finite-difference based, its practical ceiling depends on data fidelity; a natural test is to add controlled discretization error or sensor noise to the exact-solution datasets and measure the VRMSE degradation curve, which would quantify how much residual fidelity buys the reported gains.
  • The reported single-step evaluation leaves autoregressive rollouts untested; one could enforce the same residual loss on predicted trajectories, which would likely require a different balance between the boundary and residual terms and is a direct testable extension of the framework.
  • If the factor-2.5 bound generalizes, physics-only adaptation could apply to operational settings where boundary data are available and interior observations are sparse or absent, with the pretrained backbone replaced by any adequate pretrained spatiotemporal model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an unsupervised adaptation framework for pretrained PDE foundation models. The method, UPAO, replaces dense interior supervision with a PDE residual term computed by finite differences and a boundary-condition term, then finetunes a frozen neighborhood-attention Transformer backbone through a low-rank adapter, NSLoRA, that orthogonalizes LoRA factors with Newton-Schulz iterations. The authors pretrain the backbone on six PDEBench subsets and evaluate on eleven downstream datasets, reporting that without interior labels the method stays within a factor of 2.5 of supervised LoRA finetuning on seven of eight 2D benchmarks and outperforms at least one supervised neural-operator baseline on nine of eleven datasets. The paper also reports ablations on pretraining, LoRA versus NSLoRA, and residual-normalization strategies, plus per-channel analyses and a speed comparison of Newton-Schulz versus SVD orthogonalization.

Significance. If the central result holds, the paper makes a useful empirical contribution: it demonstrates a plausible recipe for adapting a pretrained PDE foundation model to unseen equations using only the governing PDE and boundary observations, which is valuable in settings where interior solution data are unavailable. The controlled comparison between standard LoRA and NSLoRA, with shared initialization and identical objective, is a good experimental practice, and the per-channel breakdown helps localize where the orthogonalized adapter helps. The paper also provides detailed dataset-generation formulas and hyperparameters, which supports reproducibility. However, the significance is tempered by the fact that the central mechanism, the discrete PDE residual, is asserted to be near machine precision for the exact-solution datasets without direct numerical evidence, and the performance comparisons rest on single-seed runs and a relatively short training budget for the neural-operator baselines.

major comments (4)
  1. [Section B.4 / Appendix A.5] The claim in Section B.4 that the seven exact-solution datasets yield residuals near machine precision is not supported and is in tension with the second-order finite-difference stencils described in Appendix A.5. For the Fourier modes used in these datasets (for example |m|,|n|<=3 for Wave and Advection-Diffusion, and kx,ky in {1,2,3} for Burgers), central and upwind differences carry O((kh)^2) truncation error on nonzero wavenumbers, so the residual of the true solution under the implemented stencil is a systematic nonzero bias, not machine zero. Since LPDE in Eq. (4) is the only interior supervision and the boundary band Sb is excluded from residual evaluation, the fidelity of this residual is load-bearing for the method. Please report the actual maximum and mean absolute residual of the analytical solutions under the exact residual implementation used in training, and likewise report the residual magnitude for the four Well datasets, which contain solver-induced discretization error. If the residual is substantially biased on the Well datasets, the UPAO gradient is biased there and the interpretation of the results changes.
  2. [Section 4.3 / Table 3] The scratch baseline in Table 3 shows that training from random initialization under UPAO degrades VRMSE by more than an order of magnitude on the exact-solution datasets, so the method's success depends strongly on the pretrained backbone. To quantify how much of the reported accuracy is actually contributed by UPAO beyond the frozen initialization, please add a zero-shot row to Table 3 that reports the VRMSE of the frozen pretrained backbone without any finetuning. Table 2 already provides zero-shot values for the seven 2D targets, but Table 3 does not include them; adding the frozen baseline would make explicit the improvement attributable to UPAO and would strengthen the claim that the residual objective is doing useful work on top of the initialization.
  3. [Appendix E / Tables 1, 4, 5] All results are reported from a single fixed split with seed 42, with no error bars or multiple-seed statistics. This is a concern for the paper's quantitative claims. For example, the NSLoRA advantage over standard LoRA in Table 4 is small (about 4% average VRMSE improvement), and the per-channel changes in Table 14 include values as small as -0.2% and a degradation on Shear Flow Vx of +5.8%; Table 5 reports stable-rank and participation-ratio point estimates without variance. Without at least three seeds with means and standard deviations on the eight 2D benchmarks, the claimed consistent improvements and the 'rebalancing' interpretation are not statistically established. Please add multi-seed results for the main tables.
  4. [Section 4.1 / Table 1] The neural-operator baselines (FNO, TFNO, U-Net, CNextU-Net) are trained from random initialization for only 30 epochs with the Well release configurations. For neural operators with this parameter count and dataset size, 30 epochs is a small budget and may not correspond to convergence, so the claim of outperforming at least one supervised neural-operator baseline on nine of eleven datasets could reflect undertrained baselines rather than a genuine advantage. Please provide training curves, report the baseline VRMSE at a converged training length, or justify concretely why 30 epochs is sufficient for these baselines. The same budget is used for the supervised LoRA upper bound, which is less concerning, but the baseline comparison needs justification.
minor comments (6)
  1. [Section 1 / Related Work] The claim that the method is 'the first attempt to adapt a pretrained PDE foundation model to unseen equations strictly by enforcing the governing PDEs and boundary conditions' appears to conflict with the description of PI-MFM [59] in Section 2, which enforces residual losses during adaptation across 1D time-dependent PDEs. Please qualify the novelty claim to account for this related work, noting the difference in scope or input representation.
  2. [Table 1] The table groups rows under 'sup' and 'unsup' but does not explicitly distinguish the supervised LoRA upper bound on the same backbone from the UPAO row within the 'Ours' entries. Please rename the rows (e.g., 'Ours (supervised LoRA)' and 'Ours (UPAO)') and state in the caption which row is used for the 2.5x factor claim.
  3. [Appendix F.4 / Table 2] The zero-shot panels take the minimum VRMSE over all channel subsets of the source checkpoint when the source has more channels than the target. This minimum-selection procedure can favorably bias the zero-shot comparison and should be stated in the main text or in the Table 2 caption, not only in the appendix.
  4. [Section 5 / Limitations] The Limitations paragraph correctly notes that evaluation is restricted to single-step prediction and that autoregressive stability is untested. Since the adaptation objective is designed for next-step prediction, please also reflect this restriction in the abstract and in the contributions, where the phrasing 'adapts a pretrained PDE foundation model to unseen equations' could be read as covering multi-step forecasting.
  5. [Section 3.4 / Eq. (8)] The Newton-Schulz coefficients (a,b,c) are said to be 'predefined coefficients inherited from [21]' but are not given in the main text. Please add the numerical values at first use (or a pointer to Appendix A.4, where they appear), so that Eq. (8) is self-contained.
  6. [Table 2 caption] The caption says bold marks the lowest off-diagonal value per column, but in the two 'Ours' columns bold marks the better of zero-shot and UPAO for each target, which is a different rule. Please clarify the caption or use separate markers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the UPAO objective is an externally defined PDE-residual plus boundary loss, NSLoRA is evaluated by controlled comparison, and the main results are benchmarked against external neural operators and foundation models.

full rationale

The paper is an empirical study rather than a derivational one, and its central claims do not reduce to their inputs by construction. UPAO (Eqs. 3-5) minimizes a PDE residual computed by finite-difference stencils and a boundary-band MSE; both terms are defined from the governing equation and boundary data, not from the VRMSE values that are later reported. The supervised-LoRA comparison is an internal upper bound on the same backbone, but the method's output is not derived from that bound. NSLoRA is an architectural modification whose effect is measured in controlled ablations (Table 4) where standard LoRA and NSLoRA share identical warm-up matrices and differ only in the forward path; the stable-rank measurements in Table 5 are diagnostics, not inputs to the loss. The pretraining dependence shown in Table 3 is an acknowledged design property, not a circular step. Self-citations to [52] and [53] appear only in background statements about PINN training and meta-learning and are not load-bearing; the Newton-Schulz coefficients are inherited from external work [21], and the upwind discretization follows external CAN-PINN [9]. The skeptic's concern about Section B.4 (machine-precision residuals versus second-order stencils) and the use of the validation split as the evaluation set are verification and experimental-design risks, not circular reductions: even if the residual were biased, the objective would be wrong but not self-referential. The claim of being the first unsupervised PDE-foundation-model adaptation is a novelty assertion, not a circular step. Accordingly, no circular step meets the quoting standard, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities. The load-bearing assumptions are about the fidelity of the finite-difference residual, the sufficiency of the pretrained initialization, and the availability of boundary observations. The only fitted per-dataset hyperparameter is the boundary weight lambda_BC.

free parameters (3)
  • lambda_BC (boundary loss weight) = 1e4 default; dataset-specific values in released code
    Balances the boundary loss against the PDE residual loss. Table 12 shows per-dataset sensitivity (Wave 2D improves at 1e5), and Appendix D states that dataset-specific values are used in the released code, so the main results may embed per-dataset tuning.
  • LoRA rank r and scaling alpha = r=16, alpha=32
    Standard LoRA hyperparameters, fixed across datasets and not fitted per dataset.
  • Newton-Schulz iterations n = 5
    Chosen so that nontrivial singular values land near one, following the Muon reference; not tuned per dataset.
assumptions (5)
  • domain assumption The finite-difference stencils (central differences for diffusion and gradients, conservative upwind for convection) faithfully represent the governing PDE on the target grids.
    Sections 3.3 and A.5 define the residual loss via these stencils; if the discrete residual is biased, UPAO minimizes the wrong objective.
  • ad hoc to paper Analytical ground-truth datasets have machine-precision residuals under the chosen finite-difference discretization.
    Section B.4 asserts residuals near machine precision for the exact-solution datasets, but second-order stencils on nonzero-wavenumber Fourier modes have O((kh)^2) error; the claim is load-bearing for the exact-solution benchmarks.
  • domain assumption The pretrained backbone places the optimization in a basin where residual minimization recovers the true solution.
    Table 3 shows UPAO from random initialization fails by more than 10x on the exact-solution datasets, so the transferable representation is load-bearing, not the residual objective alone.
  • domain assumption Boundary observations on a one-grid-point-wide peripheral band are available at all times and suffice to anchor the predicted boundary.
    The boundary term in Eq. 3 uses the boundary reference on the width-1 band; the paper states this band is intrinsically accessible, but its sufficiency for the next-step mapping is assumed.
  • standard math Newton-Schulz iteration with the coefficients of the Muon reference approximately orthogonalizes low-rank matrices and preserves gradients.
    Appendix A.4 inherits correctness from the Muon reference; the paper does not re-derive convergence, which is standard for this iteration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unsupervised Adaptation of PDE Foundation Models." pith.science (2026). https://pith.science/paper/IQGQ6A3S

@misc{pith2026260807053,
  author       = {Pith},
  title        = {Pith review of: Unsupervised Adaptation of PDE Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IQGQ6A3S}},
  note         = {Machine review of arXiv:2608.07053}
}
read the original abstract

Pretrained partial differential equation (PDE) foundation models can generalize across different equations, but adapting them to unseen PDE systems typically requires dense solution data, which is often expensive or unavailable. To address this limitation, we propose an unsupervised PDE-based finetuning framework that eliminates the need for ground-truth solutions. We first pretrain a neighborhood attention Transformer on diverse time-dependent PDEs spanning varying spatial scales, yielding transferable representations across heterogeneous equations. In the adaptation stage, we construct a physics-based objective using the PDE residual and boundary conditions, and finetune the model on unseen equations via low-rank adaptation (LoRA). To address the uneven learning across physical quantities in standard LoRA, we introduce NSLoRA, a Newton-Schulz orthogonalized variant that rebalances adaptation. Our method achieves performance comparable to supervised LoRA finetuning without requiring any ground-truth solutions, while consistently outperforming competitive neural operator baselines and recent PDE foundation models across heterogeneous PDE benchmarks spanning multiple spatial dimensions.

Figures

Figures reproduced from arXiv: 2608.07053 by the authors.

Figure 1
Figure 1. Our unsupervised PDE-based finetuning framework predicts unseen PDEs strictly through [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of our framework. The shared backbone [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the discrete PDE residual computed for each downstream dataset. Exact [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Active Matter velocity components Vx and Vy. Top row: ground-truth fields at two representative time steps. Bottom row: predictions from our framework after unsupervised finetuning under UPAO [PITH_FULL_IMAGE:figures/full_fig_p024_4.png]
Figure 5
Figure 5. Figure 5: Gray–Scott reaction-diffusion species A and B. Top row: ground-truth fields at two representative time steps. Bottom row: predictions from our framework after unsupervised finetuning under UPAO. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Rayleigh–Bénard horizontal velocity Vx. Top row: ground-truth field at two representative time steps. Bottom row: prediction from our framework after unsupervised finetuning under UPAO [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: Rayleigh–Bénard vertical velocity Vy. Top row: ground-truth field at two representative time steps. Bottom row: prediction from our framework after unsupervised finetuning under UPAO. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Rayleigh–Bénard buoyancy field. Top row: ground-truth field at two representative time [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Rayleigh–Bénard pressure p. Top row: ground-truth field at two representative time steps. Bottom row: prediction from our framework after unsupervised finetuning under UPAO. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Shear Flow horizontal velocity Vx. Top row: ground-truth field at two representative time steps. Bottom row: prediction from our framework after unsupervised finetuning under UPAO [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Shear Flow vertical velocity Vy. Top row: ground-truth field at two representative time steps. Bottom row: prediction from our framework after unsupervised finetuning under UPAO [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: Shear Flow passive tracer. Top row: ground-truth field at two representative time steps. [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Shear Flow pressure p. Top row: ground-truth field at two representative time steps. Bottom row: prediction from our framework after unsupervised finetuning under UPAO. C Baseline Configurations C.1 Foundation-Model Baselines We compare against two pretrained PDE foun…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 24 canonical work pages

  1. [1]

    One-shot transfer learning for nonlinear pdes with perturbative pinns.arXiv preprint arXiv:2511.11137, 2025

    Samuel Auroy and Pavlos Protopapas. One-shot transfer learning for nonlinear pdes with perturbative pinns.arXiv preprint arXiv:2511.11137, 2025

    arXiv 2025

  2. [2]

    A table of solutions of the one-dimensional burgers equation.Quarterly of Applied Mathematics, 30(2):195–212, 1972

    Edward R Benton and George W Platzman. A table of solutions of the one-dimensional burgers equation.Quarterly of Applied Mathematics, 30(2):195–212, 1972

    work page 1972

  3. [3]

    Hypino: Multi-physics neural operators via hyperpinns and the method of manufactured solu- tions.arXiv preprint arXiv:2509.05117, 2025

    Rafael Bischof, Michal Piovar ˇci, Michael A Kraus, Siddhartha Mishra, and Bernd Bickel. Hypino: Multi-physics neural operators via hyperpinns and the method of manufactured solu- tions.arXiv preprint arXiv:2509.05117, 2025

    work page arXiv 2025

  4. [4]

    Johannes Brandstetter, Max Welling, and Daniel E. Worrall. Lie point symmetry data aug- mentation for neural PDE solvers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors,International Conference on Machine Learn- ing, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, Proceedings of Machine Learni...

    work page 2022

  5. [5]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  6. [6]

    Olora: Orthonormal low-rank adaptation of large language models.arXiv preprint arXiv:2406.01775, 2024

    Kerim Büyükakyüz. Olora: Orthonormal low-rank adaptation of large language models.arXiv preprint arXiv:2406.01775, 2024

    arXiv 2024

  7. [7]

    Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems.IEEE Trans

    Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems.IEEE Trans. Neural Networks, 6(4):911–917, 1995. doi: 10.1109/72.392253. URL https://doi. org/10.1109/72.392253. 10

    doi:10.1109/72.392253 1995

  8. [8]

    Omniarch: Building foundation model for scientific computing.arXiv preprint arXiv:2402.16014, 2024

    Tianyu Chen, Haoyi Zhou, Ying Li, Hao Wang, Chonghan Gao, Rongye Shi, Shanghang Zhang, and Jianxin Li. Omniarch: Building foundation model for scientific computing.arXiv preprint arXiv:2402.16014, 2024

    arXiv 2024

Show all 60 references
  1. [9]

    Can-pinn: A fast physics-informed neural network based on coupled-automatic–numerical differentiation method.Computer Methods in Applied Mechanics and Engineering, 395:114909, 2022

    Pao-Hsiung Chiu, Jian Cheng Wong, Chinchun Ooi, My Ha Dao, and Yew-Soon Ong. Can-pinn: A fast physics-informed neural network based on coupled-automatic–numerical differentiation method.Computer Methods in Applied Mechanics and Engineering, 395:114909, 2022

    2022

  2. [10]

    On a quasi-linear parabolic equation occurring in aerodynamics.Quarterly of applied mathematics, 9(3):225–236, 1951

    Julian D Cole. On a quasi-linear parabolic equation occurring in aerodynamics.Quarterly of applied mathematics, 9(3):225–236, 1951

    1951

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...

    2019

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  5. [13]

    Evans.Partial differential equations, volume 19 ofGrad

    Lawrence C. Evans.Partial differential equations, volume 19 ofGrad. Stud. Math.Providence, RI: American Mathematical Society (AMS), 2nd ed. edition, 2010. ISBN 978-0-8218-4974-3; 978-1-4704-6942-9; 978-1-4704-1144-2

    2010

  6. [14]

    Towards multi-spatiotemporal-scale generalized pde modeling.arXiv preprint arXiv:2209.15616, 2022

    Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporal-scale generalized pde modeling.arXiv preprint arXiv:2209.15616, 2022

    2022 arXiv

  7. [15]

    DPOT: auto-regressive denoising operator transformer for large-scale PDE pre-training

    Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu. DPOT: auto-regressive denoising operator transformer for large-scale PDE pre-training. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Well...

    2024

  8. [16]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 6185–6194. IEEE, 2023. doi: 10.1109/ CVPR52729.202...

    2023

  9. [17]

    Poseidon: Efficient founda- tion models for pdes

    Maximilian Herde, Bogdan Raonic, Tobias Rohner, Roger Käppeli, Roberto Moli- naro, Emmanuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient founda- tion models for pdes. In Amir Globersons, Lester Mackey, Danielle Belgrave, An- gela Fan, Ulrich Paquet, Jakub M. Tomczak,...

    2024

  10. [18]

    Holzschuh, Qiang Liu, Georg Kohl, and Nils Thuerey

    Benjamin J. Holzschuh, Qiang Liu, Georg Kohl, and Nils Thuerey. Pde-transformer: Efficient and versatile transformers for physics simulations. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors...

    2025

  11. [19]

    The partial differential equation ut +uu x =µu xx.Communications on Pure and Applied Mathematics, 3(3):201–230, 1950

    Eberhard Hopf. The partial differential equation ut +uu x =µu xx.Communications on Pure and Applied Mathematics, 3(3):201–230, 1950. doi: https://doi.org/10.1002/cpa.3160030302. URLhttps://onlinelibrary.wiley.com/doi/abs/10.1002/cpa.3160030302. 11

    1950 doi

  12. [20]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,

    2022

  13. [21]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/

    2024

  14. [22]

    Uniform spec- tral growth and convergence of muon in lora-style matrix factorization.arXiv preprint arXiv:2602.06385, 2026

    Changmin Kang, Jihun Yun, Baekrok Shin, Yeseul Cho, and Chulhee Yun. Uniform spec- tral growth and convergence of muon in lora-style matrix factorization.arXiv preprint arXiv:2602.06385, 2026

    2026

  15. [23]

    Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang

    George Em Karniadakis, Ioannis G. Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning.Nature Reviews Physics, 3(6):422–440, 2021

    2021

  16. [24]

    Apebench: A benchmark for autoregressive neural emulators of pdes

    Felix Koehler, Simon Niedermayr, Rüdiger Westermann, and Nils Thuerey. Apebench: A benchmark for autoregressive neural emulators of pdes. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Ne...

    2024

  17. [25]

    Multi-grid tensorized fourier neural operator for high-resolution pdes.arXiv preprint arXiv:2310.00120, 2023

    Jean Kossaifi, Nikola Kovachki, Kamyar Azizzadenesheli, and Anima Anandkumar. Multi-grid tensorized fourier neural operator for high-resolution pdes.arXiv preprint arXiv:2310.00120, 2023

    2023 arXiv

  18. [26]

    Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M

    Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes.J. Mach. Learn. Res., 24:89:1–89:97, 2023. URL https: //jmlr.org/...

    2023

  19. [27]

    LeVeque.Finite Difference Methods for Ordinary and Partial Differential Equations: Steady-State and Time-Dependent Problems

    Randall J. LeVeque.Finite Difference Methods for Ordinary and Partial Differential Equations: Steady-State and Time-Dependent Problems. SIAM, 2007

    2007

  20. [28]

    Stuart, and Anima Anandkumar

    Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In9th International Conference on Learning Representations, ICLR 2021, Virtual...

    2021

  21. [29]

    Fourier neural operator with learned deformations for pdes on general geometries.Journal of Machine Learning Research, 24(388):1–26, 2023

    Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries.Journal of Machine Learning Research, 24(388):1–26, 2023

    2023

  22. [30]

    Physics-informed neural operator for learning partial differential equations.ACM/IMS Journal of Data Science, 1(3):1–27, 2024

    Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations.ACM/IMS Journal of Data Science, 1(3):1–27, 2024

    2024

  23. [31]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022

    2022

  24. [32]

    Learning nonlinear operators via deeponet based on the universal approximation theorem of operators

    Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nat. Mach. Intell., 3(3):218–229, 2021. doi: 10.1038/S42256-021-00302-5. URL https: //doi.org/10....

    2021 doi

  25. [33]

    Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Géraud Krawezik, François Lanusse, Mariel Pettee, Tiberiu Tesileanu, Kyunghyun Cho, and Shirley Ho

    Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles D. Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Géraud Krawezik, François Lanusse, Mariel Pettee, Tiberiu Tesileanu, Kyunghyun Cho, and Shirley Ho. Multiple 12 physics pretrain...

    2024

  26. [34]

    Physix: A foundation model for physics simulations.CoRR, abs/2506.17774, 2025

    Tung Nguyen, Arsh Koneru, Shufan Li, and Aditya Grover. Physix: A foundation model for physics simulations.CoRR, abs/2506.17774, 2025. doi: 10.48550/ARXIV .2506.17774. URL https://doi.org/10.48550/arXiv.2506.17774

    2025 doi arXiv

  27. [35]

    The well: a large-scale collection of diverse physics simulations for machine learning.Advances in Neural Information Processing Systems, 37:44989–45037, 2024

    Ruben Ohana, Michael McCabe, Lucas Meyer, Rudy Morel, Fruzsina J Agocs, Miguel Beneitez, Marsha Berger, Blakesley Burkhart, Stuart B Dalziel, Drummond B Fielding, et al. The well: a large-scale collection of diverse physics simulations for machine learning.Advances in Neural I...

    2024

  28. [36]

    Karniadakis

    Maziar Raissi, Paris Perdikaris, and George E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.J. Comput. Phys., 378:686–707, 2019. doi: 10.1016/J.JCP.2018.10.0...

    2019 doi

  29. [37]

    Convolutional neural operators for robust and accurate learning of PDEs

    Bogdan Raonic, Roberto Molinaro, Tim De Ryck, Tobias Rohner, Francesca Bartolucci, Rima Alaifari, Siddhartha Mishra, and Emmanuel De Bézenac. Convolutional neural operators for robust and accurate learning of PDEs. InAdvances in Neural Information Processing Systems, volume 36...

    2023

  30. [38]

    Morph: Pde foundation models with arbitrary data modality.arXiv preprint arXiv:2509.21670, 2025

    Mahindra Singh Rautela, Alexander Most, Siddharth Mansingh, Bradley C Love, Alexander Scheinker, Diane Oyen, Nathan Debardeleben, Earl Lawrence, and Ayan Biswas. Morph: Pde foundation models with arbitrary data modality.arXiv preprint arXiv:2509.21670, 2025

    2025

  31. [39]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InInternational Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015

    2015

  32. [40]

    Test-time gen- eralization for physics through neural operator splitting.arXiv preprint arXiv:2602.00884, 2026

    Louis Serrano, Jiequn Han, Edouard Oyallon, Shirley Ho, and Rudy Morel. Test-time gen- eralization for physics through neural operator splitting.arXiv preprint arXiv:2602.00884, 2026

    2026

  33. [41]

    GLU variants improve transformer.CoRR, abs/2002.05202, 2020

    Noam Shazeer. GLU variants improve transformer.CoRR, abs/2002.05202, 2020. URL https://arxiv.org/abs/2002.05202

    2002 arXiv

  34. [42]

    Lemon: Learning to learn multi-operator networks.arXiv preprint arXiv:2408.16168, 2024

    Jingmin Sun, Zecheng Zhang, and Hayden Schaeffer. Lemon: Learning to learn multi-operator networks.arXiv preprint arXiv:2408.16168, 2024

    2024 arXiv

  35. [43]

    Pdebench: An extensive benchmark for scientific machine learning

    Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, edito...

    2022

  36. [44]

    Mechanism of the production of small eddies from large ones.Proceedings of the Royal Society of London

    Geoffrey Ingram Taylor and Albert Edward Green. Mechanism of the production of small eddies from large ones.Proceedings of the Royal Society of London. Series A-Mathematical and Physical Sciences, 158(895):499–521, 1937

    1937

  37. [45]

    Factorized fourier neural operators

    Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. InInternational Conference on Learning Representations (ICLR), 2023. arXiv:2111.13802

    2023 arXiv

  38. [46]

    Learning the solution operator of parametric partial differential equations with physics-informed DeepONets.Science Advances, 2021

    Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets.Science Advances, 2021. arXiv:2103.10974. 13

    2021 arXiv

  39. [47]

    Respecting causality is all you need for training physics-informed neural networks.CoRR, abs/2203.07404, 2022

    Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks.CoRR, abs/2203.07404, 2022. doi: 10.48550/ ARXIV .2203.07404. URLhttps://doi.org/10.48550/arXiv.2203.07404

    2022 doi arXiv

  40. [48]

    When and why pinns fail to train: A neural tangent kernel perspective.J

    Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective.J. Comput. Phys., 449:110768, 2022. doi: 10.1016/J.JCP.2021.110768. URLhttps://doi.org/10.1016/j.jcp.2021.110768

    2022

  41. [49]

    Gradient alignment in physics-informed neural networks: A second-order optimization perspective

    Sifan Wang, Ananyae Kumar Bhartari, Bowen Li, and Paris Perdikaris. Gradient alignment in physics-informed neural networks: A second-order optimization perspective. InAdvances in Neural Information Processing Systems, volume 38, 2025

    2025

  42. [50]

    Orthogonal subspace learning for language model continual learning

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10658–10671, 2023

    2023

  43. [51]

    Orthogeolora: Geometric parameter-efficient fine-tuning for structured social science concept retrieval on theweb.arXiv preprint arXiv:2601.09185, 2026

    Zeqiang Wang, Xinyue Wu, Chenxi Li, Zixi Chen, Nishanth Sastry, Jon Johnson, and Suparna De. Orthogeolora: Geometric parameter-efficient fine-tuning for structured social science concept retrieval on theweb.arXiv preprint arXiv:2601.09185, 2026

    2026

  44. [52]

    Evolutionary neural architecture search for physics-informed neural networks with variable-length designs.IEEE Transactions on Evolutionary Computation, pages 1–1, 2026

    Zhao Wei and Jiao Liu. Evolutionary neural architecture search for physics-informed neural networks with variable-length designs.IEEE Transactions on Evolutionary Computation, pages 1–1, 2026

    2026

  45. [53]

    Out-of-distribution generalization for neural physics solvers.arXiv preprint arXiv:2601.19091, 2026

    Zhao Wei, Chin Chun Ooi, Jian Cheng Wong, Abhishek Gupta, Pao-Hsiung Chiu, and Yew- Soon Ong. Out-of-distribution generalization for neural physics solvers.arXiv preprint arXiv:2601.19091, 2026

    2026

  46. [54]

    Geometry aware operator transformer as an efficient and accurate neural surrogate for PDEs on arbitrary domains

    Shizheng Wen, Arsh Kumbhat, Levi Lingsch, Sepehr Mousavi, Yizhou Zhao, Praveen Chan- drashekar, and Siddhartha Mishra. Geometry aware operator transformer as an efficient and accurate neural surrogate for PDEs on arbitrary domains. InThe Thirty-ninth Annual Conference on Neura...

    2025

  47. [55]

    Transolver: A fast transformer solver for pdes on general geometries

    Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long. Transolver: A fast transformer solver for pdes on general geometries. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,F...

    2024

  48. [56]

    Oplora: Orthogonal projection lora prevents catastrophic forgetting during parameter-efficient fine-tuning

    Yifeng Xiong and Xiaohui Xie. Oplora: Orthogonal projection lora prevents catastrophic forgetting during parameter-efficient fine-tuning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34088–34096, 2026

    2026

  49. [57]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors,Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa...

    2019

  50. [58]

    Physics-informed temporal alignment for auto-regressive PDE foundation models

    Congcong Zhu, Xiaoyan Xu, Jiayue Han, and Jingrun Chen. Physics-informed temporal alignment for auto-regressive PDE foundation models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,Forty-s...

    2025

  51. [59]

    Width” denotes hidden_channels for FNO and TFNO and init_features for the two U-Nets. “Depth

    Min Zhu, Jingmin Sun, Zecheng Zhang, Hayden Schaeffer, and Lu Lu. Pi-mfm: Physics- informed multimodal foundation model for solving partial differential equations.arXiv preprint arXiv:2512.23056, 2025. 14 A Architecture and Implementation Details A.1 Architecture Hyperparamete...

    2025

  52. [2022]

    URLhttps://openreview.net/forum?id=nZeVKeeFYf9

    OpenReview.net, 2022. URLhttps://openreview.net/forum?id=nZeVKeeFYf9

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.