REVIEW 3 major objections 7 minor 60 references
LLT: Local Linear Transformer for PDE Operator Learning
T0 review · 3 major / 7 minor · reviewed 2026-07-11 · grok-4.5
Pith's one-line read A transformer for PDE operators that keeps long-range coupling cheap and adds an explicit local path beats or matches published baselines while running faster.
desk verdict Solid engineering of linear global + local mixing for PDE operators; speed claims are controlled, accuracy ranking is not. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Local Linear Transformer (LLT): residual encoder blocks that mix a global kernelized linear-attention path (feature map ϕ = elu + 1, cost O(N d_h²) per head) with a local path (depthwise-separable convolution on structured grids or masked radius-neighborhood attention on unstructured meshes), gated by a fixed weight α = 0.7, plus Fourier and distance-to-reference encodings and a skip-connected decoder.
What would settle it
Retrain the published Transolver / Transolver++ / LNO baselines and LLT under identical optimizers, schedules, data splits and hyperparameter budgets on the same five benchmarks; if LLT no longer matches or beats their relative L2 errors, or if its wall-clock advantage disappears under the same controlled protocol, the central claim fails.
Extended reading notes
Core claim
Combining linear global attention with an explicit local spatial-mixing branch and geometry-aware embeddings yields a transformer-style PDE operator that is competitive with or more accurate than prior neural operators and transformers on standard benchmarks while scaling nearly linearly in mesh size and running faster than a leading slice-attention baseline under matched conditions.
Load-bearing premise
That single-run relative L2 numbers taken from the literature are a fair accuracy yardstick even though those baselines were tuned by their original authors under different training protocols, while only the timing comparison is controlled.
Editorial extensions
If this is right
- Operator-learning pipelines can keep transformer-style long-range coupling without quadratic cost by using linear attention plus an explicit local branch.
- The same architecture can be applied to both structured body-fitted meshes and large unstructured 3D node sets without redesigning the global path.
- Training-step wall-clock savings of 1.8–2.5× versus a leading slice-attention baseline become available on matched structured grids, improving the feasibility of high-resolution operator training.
- Geometry and coordinate encodings plus a skip-connected decoder preserve positional information that repeated global mixing would otherwise dilute.
Reading between the lines
- The local branch is likely most valuable on problems with sharp gradients or characteristic-limited dependence (airfoil, plasticity); smoother elliptic or pipe-flow maps may gain less, which is consistent with Pipe being the one case where Transolver++ still leads.
- Because the local radius is set from training-set nearest-neighbor statistics, performance on highly adaptive or multi-scale meshes may degrade unless the neighborhood construction is made resolution-aware.
- Extending the same local–global split to time-dependent rollouts or physics-informed residual losses is a direct next test of whether the architecture’s inductive bias transfers beyond pure supervised operator maps.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Local Linear Transformer (LLT), a supervised neural operator for PDE solution maps that pairs kernelized linear (global) attention with an explicit local spatial-mixing path (depthwise-separable convolution on structured grids; radius-neighbor masked attention on unstructured meshes), plus Fourier and distance-to-reference-grid encodings and a skip-connected decoder. The design is motivated by the quadratic cost and lack of local bias of standard attention. LLT is evaluated on five standard operator-learning benchmarks (elasticity, plasticity, airfoil, pipe, Darcy) spanning FEM/FVM/FD and structured/unstructured meshes, and is scaled to a 32,186-point 3D car aerodynamics mesh. Relative L2 errors are reported against published neural-operator and transformer baselines (Table 4); matched wall-clock and memory comparisons versus Transolver are given on structured grids (Tables 5–6, Figures 9–10). The authors claim competitive or lower error and 1.8–2.5× lower training-step time than Transolver under matched settings.
Significance. If the accuracy and efficiency claims hold under fair comparison, LLT is a useful practical contribution: a single encoder–decoder formulation that handles both structured and unstructured meshes, keeps attention near-linear in N, and adds an explicit local bias that classical PDE discretizations exploit. The controlled timing study against Transolver (Section 5.1) is a genuine strength and is more carefully matched than is common in this literature. Scaling to a 32k-point 3D car mesh without a convolutional local path further supports the design. The work sits squarely in the active line of scalable Transformer operators (Transolver, OFormer, GNOT, etc.) and would be of interest to the scientific ML and neural-operator communities. The main limitation is that the accuracy half of the central claim rests on literature single-run L2 numbers rather than a controlled re-evaluation, so the ranking in Table 4 is less solid than the speedup numbers.
major comments (3)
- Table 4 and §4.3/§5: The central accuracy claim (lowest or competitive relative L2 on Elasticity, Plasticity, Airfoil, Darcy; competitive on Pipe) is supported only by single-run LLT point estimates compared to published best-reported baselines taken chiefly from the Transolver comparison table after those authors’ own hyperparameter searches. The manuscript itself states that the table is “not a fully controlled retraining study.” Without seed variance, error bars, or at least a controlled re-run of the strongest competitors (Transolver / Transolver++) under the same loss, normalization, and training protocol used for LLT, the ranking can shift under modest protocol differences. This is load-bearing for the abstract’s “competitive or lower relative L2 error” claim; the timing comparison in §5.1 is already controlled and does not suffer from the same issue. Either re-evaluate the main ba
- §3.3, Eq. (14) and Table 1: The global/local mixing weight α is fixed at 0.7 for all reported runs, with no ablation of α, of the local branch alone, or of the global linear path alone. The paper’s architectural thesis is that combining linear global attention with explicit local mixing is what addresses the two stated limitations of standard attention. Without even a minimal ablation (e.g., α ∈ {0, 0.5, 0.7, 1} on one structured and one unstructured problem), it is hard to attribute the reported gains to the local–global design rather than to width, depth, encodings, or training schedule. A short ablation would make the design claim falsifiable and would strengthen the paper substantially.
- Car Design results (§4.1, §5, Figure 8): The 3D car experiment is presented as evidence that LLT “scales” to large unstructured meshes, but only qualitative surface-pressure plots are shown; no relative L2 (or other quantitative) error is reported for velocity or pressure, and no comparison to Transolver or other baselines on this dataset is given (despite Transolver having been evaluated on ShapeNet Car in prior work). Without a number, the scaling claim remains visual only and does not support the same accuracy language used for the five 2D benchmarks. Report at least test relative L2 for the car case, ideally with a baseline.
minor comments (7)
- Table 4 caption and body: errors are stated as “values ×10−1”; several entries (e.g. LLT Plasticity 0.0058) then look like 5.8e-4 absolute, which is easy to misread. State the scaling once more clearly or report raw relative L2.
- Figure 1 and §3.2: the convolutional stem is described as “structured only,” but the figure layout makes it look always present. Clarify in the caption that the Conv Stem branch is omitted for unstructured cases.
- §3.2, Eq. (2): the reference set {x̃_m} is fixed with R = 8 points per axis, but how the reference grid is placed for irregular domains (Elasticity, Car) is not specified. A sentence on construction would help reproducibility.
- §4.2: “adjoint (backward) pass” is nonstandard terminology for reverse-mode autodiff; “backward pass” is clearer.
- Figure 10 includes a “Navier Stokes (N=4,096)” bar that is not one of the five main problems or the car case; either define it or remove it to avoid confusion with Pipe.
- Related work: a brief comparison to other linear-attention PDE operators (e.g. Performer-style or Flowformer-style uses already cited) would better situate the local+linear combination.
- Typos / polish: “tim e (optional)” in Figure 1; occasional spacing issues around citations; “OneCycleLR (Pipe: cosine)” in Table 1 could note the schedule difference more explicitly in the text.
Circularity Check
No significant circularity: empirical architecture study with held-out evaluation; no prediction reduces to its inputs by construction.
full rationale
LLT is a supervised neural-operator architecture paper. Its load-bearing claims are (i) relative L2 accuracy on five PDE benchmarks plus a 3D car mesh and (ii) matched wall-clock speedups versus Transolver. Neither claim is a first-principles derivation that collapses into its own inputs. The global/local mix (Eq. 14 with fixed α=0.7), linear-attention feature map, RMSNorm/SwiGLU blocks, and geometry encodings are design choices whose quality is measured on held-out test splits (1000/200 or analogous), not quantities fitted from the same targets and then re-reported as predictions. Hyperparameters in Tables 1–2 are stated and then evaluated; they are not inverted from the reported L2 numbers. Self-citations (ViTO, TC-UNet, wave schemes) appear only in Related Work as background and do not underwrite uniqueness, forbid alternatives, or force the accuracy/speed results. Baseline L2 entries in Table 4 are taken from external literature after those authors’ own tuning—this is a fairness/protocol limitation, not circularity under the stated patterns. Timing comparisons (Tables 5–6) are controlled re-measurements, not tautologies. No self-definitional identity, fitted-input-as-prediction, uniqueness import, or ansatz-via-self-citation chain is present. Score 0 with empty steps is therefore the correct finding.
Assumptions & free parameters
free parameters (5)
- α (global/local attention weight) =
0.7
- model width D and depth L (per problem) =
problem-specific (e.g. Darcy D=128 L=8)
- max neighbors K / neighborhood radius =
K=96/32; 2× median NN
- Fourier bands B and reference-grid size R =
B=4, R=8
- learning-rate schedule and weight decay =
1e-3 / 1e-5 / 500
assumptions (4)
- domain assumption Kernelized linear attention with feature map ϕ(z)=elu(z)+1 yields O(N d_h^{2}) cost and is a valid global mixer for PDE fields.
- domain assumption PDE solution fields possess both local differential structure and nonlocal dependence induced by geometry/BCs, so an explicit local path plus global path is beneficial.
- domain assumption Relative L2 error on the held-out splits of the Geo-FNO/Transolver benchmarks is a sufficient accuracy metric for operator quality.
- ad hoc to paper Published best-reported baseline numbers (after original authors’ hyperparameter searches) may be compared directly to a newly trained model.
invented entities (2)
-
Local Linear Transformer (LLT) / Locality-Aware Attention
-
Distance-to-reference-grid encoding d(x_i)
Cite this review
Pith. "Pith review of LLT: Local Linear Transformer for PDE Operator Learning." pith.science (2026). https://pith.science/paper/WAOGAYKQ
@misc{pith2026260707718,
author = {Pith},
title = {Pith review of: LLT: Local Linear Transformer for PDE Operator Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/WAOGAYKQ}},
note = {Machine review of arXiv:2607.07718}
}
abstract
Neural operators have become a common approach for learning PDE solution maps and accelerating numerical simulations. Transformer-based neural operators are of particular interest, since attention can learn long-range dependencies in the computational domain. However, standard attention has two major limitations when applied to PDEs: it scales quadratically with the number of computational nodes, and it lacks an explicit bias toward local interactions. To address these issues, we introduce Local Linear Transformer (LLT) for PDE operator learning. The architecture combines linear global attention with local spatial mixing, and incorporates coordinate and geometry information. We evaluate LLT on several PDE problems, including elasticity, plasticity, airfoil flow, pipe flow, and Darcy flow. The reference data for these problems span finite-element, finite-volume, and finite-difference discretizations on structured and unstructured meshes. Compared with other neural-operator and transformer baselines from prior studies, LLT achieves competitive or lower relative $L_2$ error across these problems. On matched structured discretizations, wall-clock time per training iteration is reduced by factors of 1.8 to 2.5 relative to Transolver. We also scale the approach and apply it to a three-dimensional car aerodynamics dataset with 32,186 unstructured mesh points per sample. Together, these results indicate that LLT provides an accurate and computationally efficient operator for PDE problems across discretizations, mesh types, and problem settings.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Journal of Machine Learning Research , volume=
Neural operator: Learning maps between function spaces with applications to pdes , author=. Journal of Machine Learning Research , volume=
-
[2]
Nature machine intelligence , volume=
Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators , author=. Nature machine intelligence , volume=. 2021 , publisher=
2021
-
[3]
arXiv preprint arXiv:2003.03485 , year=
Neural operator: Graph kernel network for partial differential equations , author=. arXiv preprint arXiv:2003.03485 , year=
arXiv 2003
-
[4]
Journal of Machine Learning Research , volume=
Fourier neural operator with learned deformations for pdes on general geometries , author=. Journal of Machine Learning Research , volume=
-
[5]
ACM/IMS Journal of Data Science , volume=
Physics-informed neural operator for learning partial differential equations , author=. ACM/IMS Journal of Data Science , volume=. 2024 , publisher=
2024
-
[6]
Proceedings of the 40th International Conference on Machine Learning , pages=
GNOT: a general neural operator transformer for operator learning , author=. Proceedings of the 40th International Conference on Machine Learning , pages=
-
[7]
Computer Methods in Applied Mechanics and Engineering , volume=
Vito: Vision transformer-operator , author=. Computer Methods in Applied Mechanics and Engineering , volume=. 2024 , publisher=
2024
-
[8]
Computer Methods in Applied Mechanics and Engineering , volume=
Real-time inference and extrapolation with Time-Conditioned UNet: Applications in hypersonic flows, incompressible flows, and global temperature forecasting , author=. Computer Methods in Applied Mechanics and Engineering , volume=. 2025 , publisher=
2025
Show all 60 references
-
[9]
1st Workshop on the Synergy of Scientific and Machine Learning Modeling@ ICML2023 , year=
Understanding the Efficacy of U-Net & Vision Transformer for Groundwater Numerical Modelling , author=. 1st Workshop on the Synergy of Scientific and Machine Learning Modeling@ ICML2023 , year=
-
[10]
Nature Reviews Physics , volume=
Neural operators for accelerating scientific simulations and design , author=. Nature Reviews Physics , volume=. 2024 , publisher=
2024
-
[11]
Advances in Neural Information Processing Systems , volume=
Latent neural operator for solving forward and inverse pde problems , author=. Advances in Neural Information Processing Systems , volume=
-
[12]
ICLR , year=
Fourier Neural Operator for Parametric Partial Differential Equations , author=. ICLR , year=
-
[13]
Advances in Water Resources , year=
U-FNO--An enhanced Fourier neural operator-based deep-learning model for multiphase flow , author=. Advances in Water Resources , year=
-
[14]
TMLR , year=
U-no: U-shaped neural operators , author=. TMLR , year=
-
[15]
International Conference on Learning Representations , year=
Factorized Fourier Neural Operators , author=. International Conference on Learning Representations , year=
-
[16]
ICML , year=
Solving High-Dimensional PDEs with Latent Spectral Models , author=. ICML , year=
-
[17]
NeurIPS , year=
Choose a Transformer: Fourier or Galerkin , author=. NeurIPS , year=
-
[18]
Liu, Xinliang and Xu, Bo and Zhang, Lei , journal=
-
[19]
Transformer for Partial Differential Equations
Zijie Li and Kazem Meidani and Amir Barati Farimani , journal=. Transformer for Partial Differential Equations
-
[20]
Scalable Transformer for PDE Surrogate Modeling , journal=
Zijie Li and Dule Shu and Amir Barati Farimani , year=. Scalable Transformer for PDE Surrogate Modeling , journal=
-
[21]
Nature Machine Intelligence , volume=
Blending neural operators and relaxation methods in PDE numerical solvers , author=. Nature Machine Intelligence , volume=. 2024 , publisher=
2024
-
[22]
Proceedings of the 41st International Conference on Machine Learning , year=
Improved Operator Learning by Orthogonal Attention , author=. Proceedings of the 41st International Conference on Machine Learning , year=
-
[23]
ICML , year=
Transolver: A Fast Transformer Solver for PDEs on General Geometries , author=. ICML , year=
-
[24]
Forty-second International Conference on Machine Learning , year=
Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries , author=. Forty-second International Conference on Machine Learning , year=
-
[25]
Advances in Neural Information Processing Systems , year=
Attention is All You Need , author=. Advances in Neural Information Processing Systems , year=
-
[26]
International Conference on Machine Learning , year=
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention , author=. International Conference on Machine Learning , year=
-
[27]
International Conference on Learning Representations , year=
Rethinking Attention with Performers , author=. International Conference on Learning Representations , year=
-
[28]
European Conference on Computer Vision , year=
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis , author=. European Conference on Computer Vision , year=
-
[29]
Advances in Neural Information Processing Systems , year=
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains , author=. Advances in Neural Information Processing Systems , year=
-
[30]
Shazeer, Noam , journal=
-
[31]
Advances in Neural Information Processing Systems , year=
Root Mean Square Layer Normalization , author=. Advances in Neural Information Processing Systems , year=
-
[32]
Applied Mathematics and Computation , volume=
A convolutional dispersion relation preserving scheme for the acoustic wave equation , author=. Applied Mathematics and Computation , volume=. 2024 , publisher=
2024
-
[33]
International Conference on Learning Representations , year=
Fast and Accurate Deep Network Learning by Exponential Linear Units , author=. International Conference on Learning Representations , year=
-
[34]
International Conference on Learning Representations , year=
Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=
-
[35]
arXiv preprint arXiv:2307.11833 , year=
Pinnsformer: A transformer-based framework for physics-informed neural networks , author=. arXiv preprint arXiv:2307.11833 , year=
-
[36]
Machine Learning: Science and Technology , volume=
Physics informed token transformer for solving partial differential equations , author=. Machine Learning: Science and Technology , volume=. 2024 , publisher=
2024
-
[37]
ACM Transactions on Graphics , volume=
Learning Three-Dimensional Flow for Interactive Aerodynamic Design , author=. ACM Transactions on Graphics , volume=. 2018 , doi=
2018
-
[38]
Chang, Angel X and Funkhouser, Thomas and Guibas, Leonidas and Hanrahan, Pat and Huang, Qixing and Li, Zimo and Savarese, Silvio and Savva, Manolis and Song, Shuran and Su, Hao and Xiao, Jian and Yi, Li and Yu, Fisher , journal=
-
[39]
2012 , publisher=
The Finite Element Method: Linear Static and Dynamic Finite Element Analysis , author=. 2012 , publisher=
2012
-
[40]
Matemati
Finite difference method for numerical computation of discontinuous solutions of the equations of fluid dynamics , author=. Matemati
-
[41]
Handbook of Numerical Analysis , volume=
Finite volume methods , author=. Handbook of Numerical Analysis , volume=. 2000 , publisher=
2000
-
[42]
2002 , publisher=
Finite volume methods for hyperbolic problems , author=. 2002 , publisher=
2002
-
[43]
Journal of Computational physics , volume=
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , author=. Journal of Computational physics , volume=. 2019 , publisher=
2019
-
[44]
International conference on machine learning , pages=
Pde-net: Learning pdes from data , author=. International conference on machine learning , pages=. 2018 , organization=
2018
-
[45]
arXiv preprint arXiv:1908.04463 , year=
DL-PDE: Deep-learning based data-driven discovery of partial differential equations from discrete and noisy data , author=. arXiv preprint arXiv:1908.04463 , year=
1908 arXiv
-
[46]
Advances in Neural Information Processing Systems , volume=
PDEBench: An extensive benchmark for scientific machine learning , author=. Advances in Neural Information Processing Systems , volume=
-
[47]
arXiv preprint arXiv:2209.15616 , year=
Towards multi-spatiotemporal-scale generalized pde modeling , author=. arXiv preprint arXiv:2209.15616 , year=
-
[48]
Journal of Computational Physics , volume=
Beyond the Courant-Friedrichs-Lewy condition: Numerical methods for the wave problem using deep learning , author=. Journal of Computational Physics , volume=. 2021 , publisher=
2021
-
[49]
Acta Mechanica Sinica , volume=
Physics-informed neural networks (PINNs) for fluid mechanics: A review , author=. Acta Mechanica Sinica , volume=. 2021 , publisher=
2021
-
[50]
arXiv preprint arXiv:2208.13273 , year=
A hybrid iterative numerical transferable solver (HINTS) for PDEs based on deep operator network and relaxation methods , author=. arXiv preprint arXiv:2208.13273 , year=
-
[51]
2020 , institution=
Benchmark problems for the mesoscale multiphysics phase field simulator (MEMPHIS) , author=. 2020 , institution=
2020
-
[52]
npj Computational Materials , volume=
Learning two-phase microstructure evolution using neural operators and autoencoder architectures , author=. npj Computational Materials , volume=. 2022 , publisher=
2022
-
[53]
Physical Review Fluids , volume=
Physics-informed machine learning approach for augmenting turbulence models: A comprehensive framework , author=. Physical Review Fluids , volume=. 2018 , publisher=
2018
-
[54]
Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages=
Towards physics-informed deep learning for turbulent flow prediction , author=. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages=
-
[55]
Energies , volume=
A review of physics-informed machine learning in fluid mechanics , author=. Energies , volume=. 2023 , publisher=
2023
-
[56]
Physics of Fluids , volume=
A comprehensive review of advances in physics-informed neural networks and their applications in complex fluid dynamics , author=. Physics of Fluids , volume=. 2024 , publisher=
2024
-
[57]
arXiv preprint arXiv:2111.13802 , year=
Factorized fourier neural operators , author=. arXiv preprint arXiv:2111.13802 , year=
-
[58]
arXiv preprint arXiv:2202.06258 , year=
Flowformer: Linearizing transformers with conservation flows , author=. arXiv preprint arXiv:2202.06258 , year=
-
[59]
Advances in neural information processing systems , volume=
Implicit neural representations with periodic activation functions , author=. Advances in neural information processing systems , volume=
-
[60]
Advances in Neural Information Processing Systems , volume=
Geometry-informed neural operator for large-scale 3d pdes , author=. Advances in Neural Information Processing Systems , volume=
Reviewed July 11, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.