REVIEW 8 cited by
Multiple Physics Pretraining for Physical Surrogate Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling of spatiotemporal systems with transformers. In MPP, rather than training one model on a specific physical system, we train a backbone model to predict the dynamics of multiple heterogeneous physical systems simultaneously in order to learn features that are broadly useful across systems and facilitate transfer. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks over a broad fluid mechanics-oriented benchmark. We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on all pretraining sub-tasks without the need for finetuning. For downstream tasks, we demonstrate that finetuning MPP-trained models results in more accurate predictions across multiple time-steps on systems with previously unseen physical components or higher dimensional systems compared to training from scratch or finetuning pretrained video foundation models. We open-source our code and model weights trained at multiple scales for reproducibility.
Forward citations
Cited by 8 Pith papers
-
TIDE: A Physically Diverse 3D Turbulence Benchmark Dataset for Advancing Scientific Machine Learning
TIDE is a DNS-verified, physically diverse 3D turbulence benchmark with independent ensembles that shows current neural operators barely beat persistence and that low pointwise error does not guarantee physical fidelity.
-
Neural operator discovery from heterogeneous trajectories
Trajectory grouping plus a low-dimensional latent bottleneck lets a neural operator discover each system's hidden governing factors and extrapolate to unseen systems.
-
Towards a Physics Foundation Model
A single transformer-based model, GPhyT, trained on diverse 2D simulation data, predicts next states across several fluid and heat-transfer systems and extrapolates to similar unseen regimes with plausible results.
-
Probabilistic operator learning: generative modeling and uncertainty quantification for foundation models of differential equations
ICON is shown to compute the posterior predictive mean of differential equation solutions, and a generative extension, GenICON, provides samples from this distribution for uncertainty quantification.
-
Pixel-Resolved Long-Context Learning for Turbulence at Exascale: Resolving Small-scale Eddies Toward the Viscous Limit
A multiscale transformer with a new collective-based parallel attention method is claimed to be the first deep-learning model to reproduce small-scale turbulence statistics down to the viscous limit in 3D flow.
-
Materials Behavior as Mechanism Ensembles: A Probabilistic Framework for Emergent Behaviors
Materials phenomena such as fatigue crack growth are framed as conditional probability landscapes over competing unit mechanisms, to be inferred from multiscale simulation and multimodal data and then optimized toward...
-
Foundation Models for Astrophysics
Astronomical 'foundation models' largely reuse transformers and self-supervised pretraining, but evidence of transfer to new instruments, populations, or tasks remains rare; the paper argues such evidence, not archite...
-
Machine learning for modelling unstructured grid data in computational physics: a review
A broad review of machine learning techniques for modeling unstructured mesh data in computational physics, with a taxonomy, a qualitative comparison, and a list of public benchmarks.
Discussion (0). Continue with ORCID to comment.