Pith. sign in

REVIEW 2 major objections 6 minor 2 cited by

BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics

T0 review · 2 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A block causal transformer that predicts whole frames instead of single tokens reaches 1.18% average relative L2 error across six fluid-dynamics datasets and outperforms a 523M-parameter prior model while using 156M parameters.

desk verdict Zero-shot results and ablations are solid; the fine-tuning transfer claim is undercut by unequal supervision, and the abstract overstates it. read the letter →

arxiv 2501.18972 v2 pith:V63ZFRIJ submitted 2025-01-31 cs.LG cs.NAmath.NA

Yuxuan Liu , Jingmin Sun , Hayden Schaeffer This is my paper · ORCID
classification cs.LGcs.NAmath.NA
keywords PDEfoundationmodelblockcausaltransformernextframepredictionfluiddynamicsNavier-Stokesequationszero-shotgeneralizationautoregressiveforecastingMuonoptimizer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes BCAT, a pretrained transformer for forecasting two-dimensional fluid flows across multiple equation families. Its central claim is that predicting the next frame as a whole, instead of the next token, is the right objective for spatiotemporal dynamics, and that a block causal attention mask makes this objective practical. Trained once on about 69K trajectories from shallow-water, incompressible Navier-Stokes, and compressible Navier-Stokes datasets, the model reports an average relative $\mathrm{L}^2$ error of 1.18% across six evaluation families, outperforming prior PDE foundation models including one with 3.4 times more parameters (45% lower error) and similar-sized models (66% lower error). If these results are correct, a standard decoder-only transformer with next-frame prediction can serve as the backbone of a PDE foundation model, with flexible context windows, parallel training, and key-value cache inference.

What carries the argument

The central object is the block lower triangular attention mask $M$, partitioned into $N\times N$ blocks where $N$ is the number of tokens in one frame; entries below and on the diagonal are all ones and entries above are zeros. This mask is what makes next frame prediction coherent: tokens within a frame attend bidirectionally, while frames remain temporally causal. It is the mechanism that lets the model use flexible history lengths, train with one pass over many output frames, and cache keys and values at inference.

What would settle it

Evaluate BCAT on the PDEBench compressible Navier-Stokes trajectories at their native 512x512 resolution without downsampling to 128x128; if the average relative $\mathrm{L}^2$ error deteriorates sharply or rollout diverges at high wavenumbers, the fixed-grid resampling that the accuracy claim relies on is the failure point.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the unit of autoregressive prediction for PDEs should be the frame, not the token. A next-frame model generates all visual tokens of the next timestep in parallel, conditioned on all tokens of every previous frame; an ablation on the PDEArena Navier-Stokes dataset shows this reduces relative $\mathrm{L}^2$ error by 3.5x compared with an otherwise identical next-token model and makes rollout 185x faster because only $T$ model calls are needed for $T$ future frames. The same choice, combined with the Muon optimizer, yields zero-shot averages of 1.18% across six fluid families and a fine-tuned turbulence error of 3.87% after only 500 gradient updates, versus 6.76% for a 523M-parameter prior model. The author's claim is that next frame prediction is better conditioned for multi-step forecasting than next token prediction, and that this is the mechanism behind the accuracy and transfer gains.

Load-bearing premise

All datasets are resampled to a fixed 128 by 128 grid and all physical fields are zero-padded to four channels; if that resampling removes small-scale turbulent structures, the reported accuracy will not carry over to high-resolution flows.

Editorial extensions

If this is right

  • A single BCAT model can zero-shot forecast six fluid-dynamics equation families at an average relative $\mathrm{L}^2$ error of 1.18%, so a pretrained PDE foundation model does not need per-equation training.
  • Next frame prediction makes inference about 185x faster than next token prediction and enables one-pass parallel loss computation over many output frames, lowering the compute barrier for larger pretraining runs.
  • The block causal mask gives a flexible context window: a model trained on 10 input frames can directly consume longer histories, while fixed-window update rules must discard earlier frames.
  • Brief fine-tuning (500 gradient updates on a 1.6K-trajectory turbulence set) drops the error to 3.87%, indicating the pretrained representation adapts to new regimes with very little data.
  • Because the backbone is a conventional decoder-only transformer, engineering improvements such as key-value caching, bfloat16 mixed precision, and memory-efficient attention transfer directly to PDE forecasting; the paper reports 59 hours of training time versus 120 hours for MPP-B and 80 hours for DPOT-M.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the six tested families, the next-frame objective is a plausible fit for any spatiotemporal system with Markovian dynamics, such as weather, ocean, or plasma modeling; the paper does not test these, so this is an extrapolation.
  • The fixed 128x128 grid and zero-padding of physical fields to four channels set a resolution ceiling; a natural extension is to test whether accuracy degrades on native high-resolution turbulent flows or on systems with more than four state variables.
  • The Muon optimizer result appears orthogonal to the architecture: the paper reports that drop-in replacement of PROSE-FD's optimizer improves it from 3.32% to 3.04%, so the optimizer's benefit may transfer to other PDE foundation models independent of BCAT.
  • Since BCAT is a standard transformer, its scaling behavior can likely be described by known transformer scaling laws; the paper does not establish this, but the architecture makes such a study a direct next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The manuscript proposes BCAT, a decoder-only transformer with a block causal attention mask, trained to predict the next frame of 2D fluid dynamics simulations from ten input frames. The model is pretrained on a mixture of six dataset families from PDEBench, PDEArena, and CFDBench at a fixed 128x128 resolution, with zero-padding of physical channels to four fields. In zero-shot evaluation the authors report a mean relative L2 error of 1.18%, outperforming several baselines, including DPOT-L (523M parameters) at 2.16%. The paper also reports a fine-tuning experiment on PDEBench Turbulence with a claimed >40% improvement over DPOT-L and MPP-L, and ablations over the optimizer, mask type, patch size, and next-frame vs. next-token objectives.

Significance. The zero-shot comparison is more careful than many prior PDE foundation model papers: the authors retrain MPP and DPOT on the same data with the same context window and metric, rather than quoting published numbers. If confirmed, the result shows that a standard decoder-only transformer with a block causal mask and next-frame prediction is a strong, parameter-efficient baseline for PDE forecasting. The release of code and the detailed ablations are useful. The main weaknesses are the uncontrolled supervision budget in the transfer experiment and the lack of seed variance, both of which currently prevent the paper from fully supporting its headline claims.

major comments (2)
  1. [§4.3, Table 2] The fine-tuning comparison does not control for the number of output-frame supervision signals. As stated in §3.5 and §4.3, MPP and DPOT use fixed-window update rules that yield one output step per forward pass, while BCAT's next-frame prediction yields T=10 output steps per forward pass. With 500 gradient updates at batch size 64, BCAT receives 500×64×10 frame-level targets versus 500×64×1 for the baselines. The reported 42.8% relative improvement (6.76% vs. 3.87%) may therefore be explained by this 10-fold difference in supervision, rather than by superior transfer ability or parameter efficiency. To support the abstract's claim of '>40% better accuracy' in adaptation, the authors should add an equal-supervision control, e.g., train the baselines for 10× more updates, or down-weight BCAT's per-frame loss to match one output step per update, or compare after an equal number of epochs over the fine-tuning data.
  2. [§4.2, Tables 1-7] No multiple seeds or confidence intervals are reported for any experimental result. The central zero-shot advantage (e.g., 1.18% vs. 2.16% for DPOT-L in Table 1) and all ablation conclusions rest on single training runs. Because the baselines are retrained for this paper, there is no evidence about the run-to-run variability of the comparison. The authors should run at least three seeds for BCAT and the strongest baselines (DPOT-L, MPP-L) and for the key ablations (Tables 3-7), and report means with standard deviations or confidence intervals.
minor comments (6)
  1. [Abstract and §4.1] The abstract states that the model was 'tested on about 8K trajectories', but Section 4.1 says the total dataset is about 69K trajectory sequences with a 10% test split, which yields roughly 6.9K test trajectories. This discrepancy should be resolved; if the 'about 8K' figure includes additional trajectories from other sources or a different split, that should be stated explicitly.
  2. [Abstract and §4.4] The '3.5x accuracy improvement' of next-frame over next-token prediction is reported in the abstract and contributions without the caveat that the next-token model only converged on the PDEArena NS dataset; the comparison was not run on the full mixture as noted in §4.4. Please qualify the claim accordingly.
  3. [Table 1] The average error in the last column is an unweighted mean over the six families; since the families have very different sample sizes, reporting a trajectory-weighted average would clarify whether the headline 1.18% is representative of the whole test set.
  4. [Appendix B.3] For PROSE-FD, the paper states that results are taken directly from [30]. It should be stated explicitly whether the same data split, evaluation code, and metric were used, so readers can judge the fairness of the comparison.
  5. [Section 4.3] The statement that 'zero-shot performance is an unbiased metric' is too strong; zero-shot performance is a useful and standard metric, but it can still be influenced by pretraining data distribution overlap. Consider softening this wording.
  6. [Appendix A] Equation references in the captions of Figure 6 appear to be off by one (the compressible Navier-Stokes system is Eqs. (9)-(11) and the PDEArena Navier-Stokes system is Eqs. (12)-(14)); please check the cross-references.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark numbers are empirical and do not reduce to fitted inputs or self-citations.

full rationale

The paper's central claims are empirical benchmark results, not first-principles derivations. Table 1 is self-contained in the relevant sense: MPP and DPOT baselines are retrained on the same data and evaluated with the same metric, and the other classical baselines are independent. The only baseline taken directly from prior work is PROSE-FD (Appendix B.3), which is a self-citation, but it is not load-bearing because BCAT also outperforms the independently retrained MPP-L, DPOT-L, and ViT baselines. The block-causal mask is credited to both VICON (self) and VAR (external), with no uniqueness theorem imported. The 3.5x next-frame advantage is an ablation, not an identity. The transfer experiment in Section 4.3 does give BCAT roughly 10x more output supervision per update (10 frames vs 1 frame per forward pass), which is a fairness confound for the '>40% better transfer' claim, but it is a benchmarking/correctness issue, not a circular reduction: the reported error is not definitionally equal to the training loss or to a fitted parameter. No equation in the paper is shown to be equivalent to its own input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities, forces, or conservation laws. It depends on hand-chosen hyperparameters and domain assumptions about Markovian dynamics, resolution unification, channel padding, and cross-dataset transfer.

free parameters (4)
  • Patch size P = 8
    Chosen via ablation (Section 4.4); smaller patches improve accuracy but increase sequence length and compute. This choice directly affects performance.
  • Context window T0 = 10
    Number of input frames for training and evaluation; selected to match dataset temporal resolutions and not tuned systematically.
  • Output steps T = 10 (4 for PDEArena NS)
    Evaluation horizon; for PDEArena NS with only 14 frames, T=4 is used. This varies across datasets and affects error accumulation.
  • Learning rate = 1e-3
    Core training hyperparameter; standard for transformer fine-tuning, chosen with a warmup-stable-decay schedule.
assumptions (4)
  • domain assumption The solution dynamics are approximately Markovian, so a fixed context window of T0 frames suffices to predict the next frame.
    Section 3.5 states that next frame prediction is natural for Markovian sequences, and the entire model is built on this assumption.
  • domain assumption All datasets can be resampled to 128x128 resolution without losing essential physics.
    Section 3.3 and Appendix A describe upsampling or downsampling all data to 128x128. If this introduces significant aliasing, results may not generalize.
  • domain assumption Zero-padding physical channels to c=4 does not harm learned representations.
    Section 3.3 says zero-padding unifies channels across PDEs. This assumes missing fields can be treated as zero without misleading the model.
  • domain assumption A single model trained on a mixture of heterogeneous fluid PDE datasets will transfer to unseen parameter regimes and datasets.
    This is the premise of a PDE foundation model; the paper evaluates it empirically but does not prove it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics." pith.science (2026). https://pith.science/paper/V63ZFRIJ

@misc{pith2026250118972,
  author       = {Pith},
  title        = {Pith review of: BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V63ZFRIJ}},
  note         = {Machine review of arXiv:2501.18972}
}
read the original abstract

We introduce BCAT, a PDE foundation model designed for autoregressive prediction of solutions to two dimensional fluid dynamics problems. Our approach uses a block causal transformer architecture to model next frame predictions, leveraging previous frames as contextual priors rather than relying solely on sub-frames or pixel-based inputs commonly used in image generation methods. This block causal framework more effectively captures the spatial dependencies inherent in nonlinear spatiotemporal dynamics and physical phenomena. In an ablation study, next frame prediction demonstrated a 3.5x accuracy improvement over next token prediction. BCAT is trained on a diverse range of fluid dynamics datasets, including incompressible and compressible Navier-Stokes equations across various geometries and parameter regimes, as well as the shallow-water equations. The model's performance was evaluated on 6 distinct downstream prediction tasks and tested on about 8K trajectories to measure robustness on a variety of fluid dynamics simulations. BCAT achieved an average relative error of 1.18% across all evaluation tasks, outperforming prior approaches on standard benchmarks. With fine-tuning on a turbulence dataset, we show that the method adapts to new settings with more than 40% better accuracy over prior methods.

Figures

Figures reproduced from arXiv: 2501.18972 by the authors.

Figure 1
Figure 1. BCAT model overview. The inputs to the model are the initial frames sampled from the datasets, which are patchified and converted into a sequence of features. Transformer layers then take the input sequence to perform next frame prediction, where a block causal mask allows spatial interactions within a frame and temporal causality across varying-length context windows. The processed features are then transformed bac… view at source ↗
Figure 2
Figure 2. Evaluating BCAT, DPOT-L, and MPP-L for more output time steps on PDEArena NS-cond dataset. Rollout is used to obtain outputs beyond the training timesteps [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Comparing BCAT trained with Muon vs. AdamW optimizer. The target (first row) is the first 6 output steps from the PDEArena Navier-Stokes (conditioned) dataset (particle density channel). For each optimizer (each row), we display the difference between the target and model output. Relative L 2 errors for the full trajectories are listed after the optimizer names. instead of block causal). When trained on a mixture of… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Transformer Layers used in BCAT model. B.3 Baselines and Comparisons In this section, we include more details about the compared models. DeepONet [33]. We employ the unstacked DeepONet architecture, consisting of a single trunk network and a single branch network. Init…
Figure 5
Figure 5. Figure 5: Example outputs from the BCAT model. 4 output steps for PDEArena Navier￾Stokes dataset. The channel plotted is the particle density in equation (12). Each column represents a different timestamp. For this trajectory, the relative L 2 error is 1.93%. 25 [PITH_FULL_IMAG…
Figure 6
Figure 6. Figure 6: Two example outputs from the BCAT model. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: Comparing outputs from different models. The target (first row) is the first 6 output steps from the PDEArena Navier-Stokes (conditioned) dataset (particle density channel). For each model (each row), we display the difference between the target and model output. Relat…
Figure 8
Figure 8. Figure 8: Comparing outputs from different models. The target is the first 6 output steps from the PDEBench Compressible Navier-Stokes dataset (y-velocity channel). For each model (each row), we display the difference between target and model output. Relative L 2 errors for the …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Probabilistic operator learning: generative modeling and uncertainty quantification for foundation models of differential equations

    stat.ML 2025-09 conditional novelty 6.0 of 10

    ICON is shown to compute the posterior predictive mean of differential equation solutions, and a generative extension, GenICON, provides samples from this distribution for uncertainty quantification.

  2. PDEformer-2: A Versatile Foundation Model for Two-Dimensional Partial Differential Equations

    math.NA 2025-07 conditional novelty 6.0 of 10

    PDEformer-2 is a pretrained graph-transformer and implicit-neural-representation model that solves a broad class of 2D PDEs from their symbolic form, with zero-shot, few-shot, and inverse-problem capabilities.

Reference graph

Works this paper leans on

63 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Phi-4 technical report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024

    arXiv 2024

  3. [3]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [4]

    Is space-time attention all you need for video understanding? InICML, volume 2, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? InICML, volume 2, page 4, 2021

    work page 2021

  5. [5]

    Aurora: A foundation model of the atmosphere.arXiv preprint arXiv:2405.13063, 2024

    Cristian Bodnar, Wessel P Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, et al. Aurora: A foundation model of the atmosphere.arXiv preprint arXiv:2405.13063, 2024

    arXiv 2024

  6. [6]

    On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  7. [7]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators, 2024

    2024

  8. [8]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877– 1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877– 1901, 2020. 15

    work page 1901

Show all 63 references
  1. [9]

    Vicon: Vision in-context operator networks for multi-physics fluid dynamics prediction.arXiv preprint arXiv:2411.16063, 2024

    Yadi Cao, Yuxuan Liu, Liu Yang, Rose Yu, Hayden Schaeffer, and Stanley Osher. Vicon: Vision in-context operator networks for multi-physics fluid dynamics prediction.arXiv preprint arXiv:2411.16063, 2024

    2024

  2. [10]

    Llm4ts: Aligning pre-trained llms as data-efficient time-series forecasters.arXiv preprint arXiv:2308.08469, 2024

    Ching Chang, Wei-Yao Wang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Aligning pre-trained llms as data-efficient time-series forecasters.arXiv preprint arXiv:2308.08469, 2024

    2024 arXiv

  3. [11]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020

    2020

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  5. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale.ICLR, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  6. [14]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Large language models are zero-shot time series forecasters.Advances in Neural Information Processing Systems, 36, 2024

    Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. Large language models are zero-shot time series forecasters.Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [16]

    Towards multi-spatiotemporal-scale generalized pde modeling

    Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616, 2022

    2022 arXiv

  9. [17]

    Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training.arXiv preprint arXiv:2403.03542, 2024

    Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training.arXiv preprint arXiv:2403.03542, 2024

    2024 arXiv

  10. [18]

    Query-key normalization for transformers.arXiv preprint arXiv:2010.04245, 2020

    Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers.arXiv preprint arXiv:2010.04245, 2020

    2010 arXiv

  11. [19]

    Poseidon: Efficient foundation models for pdes

    Maximilian Herde, Bogdan Raonić, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient foundation models for pdes. arXiv preprint arXiv:2405.19101, 2024

    2024 arXiv

  12. [20]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  13. [21]

    Minicpm: Unveiling the potential of small language models with scalable training strategies.arXiv preprint arXiv:2404.06395, 2024

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies.arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  14. [22]

    Time-llm: Time series forecasting by reprogramming large language models.arXiv preprint arXiv:2310.01728, 2023

    Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models.arXiv preprint arXiv:2310.01728, 2023. 16

    arXiv

  15. [23]

    Time-series forecasting, knowledge distillation, and refinement within a multimodal pde foundation model.arXiv preprint arXiv:2409.11609, 2024

    Derek Jollie, Jingmin Sun, Zecheng Zhang, and Hayden Schaeffer. Time-series forecasting, knowledge distillation, and refinement within a multimodal pde foundation model.arXiv preprint arXiv:2409.11609, 2024

    2024 arXiv

  16. [24]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024

    2024

  17. [25]

    Transformer for partial differential equations’ operator learning.arXiv preprint arXiv:2205.13671, 2022

    Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations’ operator learning.arXiv preprint arXiv:2205.13671, 2022

    2022 arXiv

  18. [26]

    Fourier neural operator for parametric partial differential equations

    Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020

    2010 arXiv

  19. [27]

    Muon is scalable for llm training.arXiv preprint arXiv:2502.16982, 2025

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training.arXiv preprint arXiv:2502.16982, 2025

    2025 arXiv

  20. [28]

    How can large language models understand spatial-temporal data?arXiv preprint arXiv:2401.14192, 2024

    Lei Liu, Shuo Yu, Runze Wang, Zhenxun Ma, and Yanming Shen. How can large language models understand spatial-temporal data?arXiv preprint arXiv:2401.14192, 2024

    2024 arXiv

  21. [29]

    Gated transformer networks for multivariate time series classification.arXiv preprint arXiv:2103.14438, 2021

    Minghao Liu, Shengqi Ren, Siyuan Ma, Jiahui Jiao, Yizhou Chen, Zhiguang Wang, and Wei Song. Gated transformer networks for multivariate time series classification.arXiv preprint arXiv:2103.14438, 2021

    2021 arXiv

  22. [30]

    Prose-fd: A multimodal pde foundation model for learning multiple operators for forecasting fluid dynamics

    Yuxuan Liu, Jingmin Sun, Xinjie He, Griffin Pinney, Zecheng Zhang, and Hayden Schaeffer. Prose-fd: A multimodal pde foundation model for learning multiple operators for forecasting fluid dynamics. arXiv preprint arXiv:2409.09811, 2024

    2024 arXiv

  23. [31]

    Prose: Predicting multiple operators and symbolic expressions using multimodal transformers.Neural Networks, 180:106707, 2024

    Yuxuan Liu, Zecheng Zhang, and Hayden Schaeffer. Prose: Predicting multiple operators and symbolic expressions using multimodal transformers.Neural Networks, 180:106707, 2024

    2024

  24. [32]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations

  25. [33]

    Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators

    Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019

    1910 arXiv

  26. [34]

    Cfdbench: A comprehensive benchmark for machine learning methods in fluid dynamics.arXiv preprint arXiv:2310.05963, 2023

    Yining Luo, Yingfa Chen, and Zhen Zhang. Cfdbench: A comprehensive benchmark for machine learning methods in fluid dynamics.arXiv preprint arXiv:2310.05963, 2023

    2023 arXiv

  27. [35]

    Multiple physics pretraining for physical surrogate models.arXiv preprint arXiv:2310.02994, 2023

    Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, et al. Multiple physics pretraining for physical surrogate models.arXiv preprint arXiv:2310.02994, 2023

    arXiv

  28. [36]

    Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators

    Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopad- hyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural op...

    2022 arXiv

  29. [37]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. 17

    2018

  30. [38]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  31. [39]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  32. [40]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InInternational conference on machine learning, pages 8821–8831. Pmlr, 2021

    2021

  33. [41]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  34. [42]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part ...

    2015

  35. [43]

    Iterative berechung der reziproken matrix.ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik, 13(1):57– 59, 1933

    Günther Schulz. Iterative berechung der reziproken matrix.ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik, 13(1):57– 59, 1933

    1933

  36. [44]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  37. [45]

    Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior.Advances in Neural Information Processing Systems, 36, 2024

    Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior.Advances in Neural Information Processing Systems, 36, 2024

    2024

  38. [46]

    Towards a foundation model for partial differential equation: Multi-operator learning and extrapolation.arXiv preprint arXiv:2404.12355, 2024

    Jingmin Sun, Yuxuan Liu, Zecheng Zhang, and Hayden Schaeffer. Towards a foundation model for partial differential equation: Multi-operator learning and extrapolation.arXiv preprint arXiv:2404.12355, 2024

    2024 arXiv

  39. [47]

    Lemon: Learning to learn multi-operator networks

    Jingmin Sun, Zecheng Zhang, and Hayden Schaeffer. Lemon: Learning to learn multi-operator networks. arXiv preprint arXiv:2408.16168, 2024

    2024 arXiv

  40. [48]

    Pdebench: An extensive benchmark for scientific machine learning.Advances in Neural Information Processing Systems, 35:1596–1611, 2022

    Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning.Advances in Neural Information Processing Systems, 35:1596–1611, 2022

    2022

  41. [49]

    Are language models actually useful for time series forecasting? InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    Mingtian Tan, Mike A Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. Are language models actually useful for time series forecasting? InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  42. [50]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  43. [51]

    Visual autore- gressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autore- gressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905, 2024. 18

    2024 arXiv

  44. [52]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  45. [53]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  46. [54]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InProceedings of the 31st International Conference on Neural Information Processing Systems, 2017

    2017

  47. [55]

    Scaling laws in patchification: An image is worth 50,176 tokens and more.arXiv preprint arXiv:2502.03738, 2025

    Feng Wang, Yaodong Yu, Guoyizhe Wei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Scaling laws in patchification: An image is worth 50,176 tokens and more.arXiv preprint arXiv:2502.03738, 2025

    2025

  48. [56]

    Transolver: A fast transformer solver for pdes on general geometries.arXiv preprint arXiv:2402.02366, 2024

    Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long. Transolver: A fast transformer solver for pdes on general geometries.arXiv preprint arXiv:2402.02366, 2024

    2024 arXiv

  49. [57]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  50. [58]

    In-context operator learning with data prompts for differential equation problems.Proceedings of the National Academy of Sciences, 120(39):e2310142120, 2023

    Liu Yang, Siting Liu, Tingwei Meng, and Stanley J Osher. In-context operator learning with data prompts for differential equation problems.Proceedings of the National Academy of Sciences, 120(39):e2310142120, 2023

    2023

  51. [59]

    Prompting in-context operator learning with sensor data, equations, and natural language.arXiv preprint arXiv:2308.05061, 2023

    Liu Yang, Tingwei Meng, Siting Liu, and Stanley J Osher. Prompting in-context operator learning with sensor data, equations, and natural language.arXiv preprint arXiv:2308.05061, 2023

    2023 arXiv

  52. [60]

    Pde generalization of in-context operator networks: A study on 1d scalar nonlinear conservation laws.arXiv preprint arXiv:2401.07364, 2024

    Liu Yang and Stanley J Osher. Pde generalization of in-context operator networks: A study on 1d scalar nonlinear conservation laws.arXiv preprint arXiv:2401.07364, 2024

    2024 arXiv

  53. [61]

    Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    2019

  54. [62]

    Deeponet as a multi-operator extrapolation model: Distributed pretraining with physics-informed fine-tuning

    Zecheng Zhang, Christian Moya, Lu Lu, Guang Lin, and Hayden Schaeffer. Deeponet as a multi-operator extrapolation model: Distributed pretraining with physics-informed fine-tuning. arXiv preprint arXiv:2411.07239, 2024

    2024 arXiv

  55. [63]

    Can llms understand time series anomalies?arXiv preprint arXiv:2410.05440, 2024

    Zihao Zhou and Rose Yu. Can llms understand time series anomalies?arXiv preprint arXiv:2410.05440, 2024. 19 A Dataset Details The data was obtained from the PDEBench [48], PDEArena [16], and CFDBench [34] datasets. Unless otherwise specified, the space resolution is128 × 128. ...

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.