Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PDE-Transformer is a transformer architecture for physics simulations that, pre-trained on 16 generic PDEs, predicts out-of-distribution flows (active matter, Rayleigh-Bénard convection, shear flow) more accurately than competing…

desk verdict Solid empirical architecture paper; the separate-channel tokenization and transfer results are worth engaging, but the headline margins are single-run and a few numbers do not reconcile. read the letter →

arxiv 2505.24717 v1 pith:X37RSAO6 submitted 2025-05-30 cs.LG

Benjamin Holzschuh , Qiang Liu , Georg Kohl , Nils Thuerey This is my paper
classification cs.LG
keywords PDEsurrogatemodelingtransformerarchitecturefoundationmodelspre-trainingandfine-tuningphysicssimulationsdiffusionshiftedwindowattentionregulargrids
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces PDE-Transformer, a transformer backbone for surrogate modeling of physics simulations on 2D regular grids, and claims it beats state-of-the-art vision-transformer architectures on a pre-training set of 16 PDEs while training faster. The paper's core empirical claim is that pre-training on generic, periodic, square-domain data transfers to harder out-of-distribution tasks—active matter, Rayleigh-Bénard convection, and shear flow—where the pre-trained model outperforms training from scratch and beats other physics foundation-model architectures. A separate-channel token embedding, in which each physical channel is embedded as its own spatio-temporal token and channels interact only through channel-wise axial attention, is presented as the mechanism that preserves token information density across PDEs with different channel counts. If correct, this establishes a reusable accuracy-per-compute-strong backbone for physics foundation models.

What carries the argument

The load-bearing mechanism is the separate-channel (SC) tokenization scheme: each physical channel is embedded independently as spatio-temporal tokens, and tokens from different channels interact only through a channel-wise axial multi-head self-attention, while within-channel interactions use shifted-window attention with log-spaced relative positions. This is combined with a U-shaped multi-scale transformer that down- and upsamples tokens via PixelShuffle and PixelUnshuffle layers, adaLN-Zero conditioning, and RMSNorm on queries and keys. The SC design keeps the per-token expansion rate constant as the number of channels changes, which the paper argues preserves information density and improves transfer to unseen PDEs.

What would settle it

Retrain PDE-S, UDiT-S, and the mixed-channel and separate-channel variants on the same 16-PDE data with at least three seeds each and report nRMSE with confidence intervals; if the step-1 nRMSE intervals of PDE-S and UDiT-S overlap (0.044 versus 0.042) or the mixed-channel improvement intervals include zero, the headline outperformance claim is not supported.

Watch

Extended reading notes

Core claim

The paper claims PDE-Transformer outperforms state-of-the-art transformer architectures for computer vision—DiT, UDiT, scOT, and others—on a dataset of 16 types of PDEs, reaching nRMSE 0.044 at the first step and 0.36 after 10 rollout steps in the S configuration, versus 0.042 and 0.39 for UDiT-S, while training in 7 hours 42 minutes compared to UDiT-S's 18 hours 30 minutes. Pre-trained PDE-S additionally reduces average rollout nRMSE on active matter, Rayleigh-Bénard convection, and shear flow relative to training from scratch, with the separate-channel variant delivering 2.7x to 4.4x larger pre-training gains than the mixed-channel variant. The paper presents the architecture as a scalable backbone for building large-scale foundation models for physical sciences.

Load-bearing premise

The comparisons assume the reported nRMSE gaps and pre-training improvements are larger than run-to-run variability, because each configuration is evaluated from a single training run with no seed or variance information.

Editorial extensions

If this is right

  • Pre-training once on generic periodic-square PDE data improves downstream accuracy on tasks with non-periodic boundary conditions, non-square domains, and different physical channels compared to training from scratch.
  • At equal S configuration, PDE-Transformer achieves better or comparable nRMSE than UDiT-S with about 2.4x less training time, and scales to larger domains with lower GFlops and GPU memory than DiT-S and UDiT-S.
  • The separate-channel variant matches mixed-channel accuracy at the same configuration while giving larger pre-training gains on all three downstream tasks: 7.89% versus 2.84% on active matter, 32.90% versus 11.56% on Rayleigh-Bénard convection, and 37.19% versus 8.43% on shear flow.
  • The same architecture can be trained supervised with an MSE loss for fast one-step inference, or with flow matching for sampling from the posterior, with supervised nRMSE slightly better but the diffusion version close enough to be useful for downstream tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because every headline comparison is reported from a single run with no seed or variance information, the closest results—PDE-S versus UDiT-S at step 1, and the smaller mixed-channel pre-training gains—could plausibly move if re-run across seeds; a multi-seed study would make the robustness claim testable.
  • If the transfer result is real, then generic periodic spectral-solver data may be sufficient source material for physics foundation models, reducing the need to collect realistic or multi-geometry pre-training corpora.
  • The separate-channel decoupling implies the backbone can accept an arbitrary number of physical channels without re-tokenizing, which points toward multi-physics and coupled-system applications the paper does not test.
  • The accuracy-compute sweet spot found at small windows and patch size 4 is measured on the 16-PDE set; on flows with stronger non-locality or higher effective resolution, larger windows may become necessary, which is a testable hypothesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces PDE-Transformer, a transformer backbone for autoregressive surrogate modeling of 2D PDEs on regular grids. It combines a DiT-style transformer with a U-shaped multi-scale token hierarchy, shifted-window self-attention, relative log-spaced positional encodings, and adaLN-Zero conditioning. The main novel component is a separate-channel (SC) tokenization in which each physical channel is embedded as its own token set and interactions across channels are carried out by a dedicated channel-axis self-attention, as opposed to a mixed-channel (MC) tokenization. The authors train on a 16-PDE pre-training corpus generated with Exponax/APEBench, compare against DiT, UDiT, scOT, FactFormer, and a UNet, and fine-tune on three downstream tasks from the Well dataset (active matter, Rayleigh–Bénard convection, shear flow). They report that PDE-Transformer achieves better accuracy and lower training time than the compared transformers, that pre-training improves downstream accuracy over training from scratch, and that SC yields 2.7–4.4x larger pre-training gains than MC. The paper releases source code.

Significance. If the reported results are statistically reliable, the paper is a useful engineering contribution: it demonstrates that a carefully combined set of known transformer components (DiT, shifted-window attention, multi-scale down/up-sampling, adaLN-Zero, flow matching) yields a favorable accuracy-compute trade-off for 2D PDE emulation, and it proposes a plausible mechanism (separate-channel tokenization) for making multi-physics pre-training transfer better to out-of-distribution tasks. Strengths of the manuscript include the release of source code, the use of the externally defined Well benchmark for downstream evaluation, a broad experiment set across 16 pre-training PDEs, and honest reporting of the DiT-S stability problem and the UDiT-S step-1 advantage. The main weakness is that every headline comparison rests on a single training run per configuration, with no seed counts, error bars, or variance estimates, which leaves the central quantitative claims (outperforming SOTA transformers; SC transfer gains of 2.7–4.4x) not yet statistically secured.

major comments (2)
  1. [Section 4.1 and 4.2, Tables 1 and 3] All reported accuracy numbers come from a single training run per configuration; no seeds, standard deviations, or confidence intervals are given anywhere in the paper or appendices. This is load-bearing for the central claims. In Table 1, PDE-S (0.044) is actually worse than UDiT-S (0.042) at step 1 and only 0.03 better at step 10 (0.36 vs 0.39), so the aggregate outperformance claim depends on how the two timesteps and the training-time advantage are weighted. In Table 3, the active-matter SC improvement over scratch is 7.89% versus 2.84% for MC, a gap that could easily reverse under run-to-run variation; the RBC and shear-flow ratios are larger but are still single runs. Please report mean and standard deviation over at least 3–5 independent seeds for the key comparisons in Tables 1, 2, and 3, and for Figures 4, 5, 6, and 9, or otherwise provide a statistical justification for why single runs are sufficient.
  2. [Table 1 vs. Table 5] The reported parameter count for PDE-S is 33.2M in Table 1 but 46.57M in Table 5. This discrepancy affects the interpretability of the accuracy-efficiency comparison in Table 1 (where PDE-S is presented as smaller than UDiT-S's 58.9M) and the fairness claim in Section 4.2 that 'all models have a similar number of trainable parameters.' Please reconcile these numbers and state which parameter count corresponds to the models used in each experiment.
minor comments (6)
  1. [Algorithm 1] The pseudocode for EMA gradient clipping contains typos and an apparent variable mix-up: the title says 'Gradinet Clip'; the update for g2 uses β1 instead of β2; the clipping condition uses g2/(1−β2^i) while the clipped gradient uses g1/(1−β1^i). Please correct the pseudocode and clarify which EMA is the threshold and which is the clipping value.
  2. [Section 3.1, notation] The symbol T is used both for the total simulation horizon in the notation (system S) and for the temporal dimension of the input tensor in the patching description and in E(p) = d/(p^2 T). This overloading is confusing; consider using T_p or a separate symbol for the number of input snapshots.
  3. [Table 2 vs. Table 1] Table 2 reports nRMSE1 = 0.045 for PDE-S, while Table 1 reports nRMSE1 = 0.044 for the same configuration. Please clarify whether these are different runs, different channel variants, or a rounding difference.
  4. [Section 4.1, DiT evaluation] It is stated that for DiT-S the checkpoint with the lowest validation loss is used, whereas the other models appear to use the final EMA weights. Please state explicitly which checkpoint is used for every model, so that the comparison is defined uniformly.
  5. [Section 4.2 and Figure 6] The text says 'we will focus on [SC] in the following' but it is not explicitly stated whether the 'Ours-S' and 'Ours-S-pretrained' results in Figure 6 and Table 7 use the SC or MC variant. Please state this clearly in the caption and in the text.
  6. [Section 4.2, Table 3] The phrase '2.7× to 4.4× higher' is a rounded statement of ratios 2.78, 2.85, and 4.41. Please report the exact ratios or state the rounding explicitly to avoid reader confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline results are empirical benchmark comparisons against external data and baselines, with no fitted parameter or self-citation chain standing in for a prediction.

full rationale

The paper's central claims are empirical benchmark results, not derived quantities, so there is no derivation chain that could collapse into its inputs. The supervised loss (Eq. 1) and flow-matching loss (Eq. 3) are standard external objectives, and the nRMSE metric (Eq. 4) is defined directly from test trajectories against a reference solution. The architecture ablations (patch size, window size, MC vs. SC, S/B/L scaling) are evaluated on held-out test data from the pre-training set, whose trajectories come from the Exponax spectral solver rather than from the transformer itself. Downstream evaluation uses the independent Well repository and external baselines (FNO, OFormer, Galerkin transformer, scOT, and the official Poseidon weights), so the claimed pre-training improvements are not equivalent by construction to any parameter fitted on those tasks. Self-citations to APEBench/Exponax supply the pre-training corpus but not the conclusions; they are dataset citations, not load-bearing theoretical premises, and no uniqueness or ansatz result is imported from the authors' prior work. The single-seed / no-variance reporting is a statistical robustness concern, but under the stated rules that is a correctness risk, not circularity. No quoted passage exhibits a claim that reduces to its own input.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

This is an empirical architecture paper rather than a derivation, so the ledger is dominated by hand-set hyperparameters and dataset assumptions. No new physical entities are postulated. The central claims rest on benchmark performance, not on a derivation that could be circular.

free parameters (5)
  • patch_size p = 4
    Chosen via ablation of the accuracy-compute tradeoff at constant receptive field p times w equals 32 (Figure 5); the default configuration of the reported results depends on this choice.
  • window_size w = 8
    Chosen via window-size ablation (Figure 5); this sets the locality assumption of the shifted-window attention in every block.
  • learning_rate = 4e-5
    Reduced from the DiT default of 1.0e-4 because the DiT training configuration produced loss spikes on PDE data (Section 3.1).
  • EMA gradient clipping coefficients (beta1, beta2, alpha, kappa) = 0.99, 0.999, 2, 1.1
    Hand-set stabilizer introduced for this paper (Algorithm 1); as printed, the second EMA g2 is updated with beta1 instead of beta2.
  • flow matching sigma_min = 1e-4
    Standard flow-matching hyperparameter from Lipman et al. (2023); not individually validated in this paper.
assumptions (5)
  • standard math Flow matching ODE theory (Lipman et al. 2023; Liu et al. 2023) defines the diffusion training target in Eq. (3).
    Invoked in Section 3.2 without proof; standard generative modeling theory.
  • domain assumption The Exponax/APEBench spectral-solver corpus (16 PDE families, periodic square domains, downsampled from 2048 to 256) is a representative pre-training substrate for generic PDE surrogate modeling.
    Section 4.1 and Appendix C; the central pre-training comparison lives entirely on this self-generated data.
  • domain assumption Single-step autoregressive training with Tp equals 1 and 30-step rollouts transfers to the 200-step test rollouts of ks, decay-turb, and kolm-flow.
    Section 4.1; the test sets for these chaotic systems are longer than the training rollout length.
  • domain assumption Local shifted-window attention (w equals 8, about 32 grid cells) is sufficient for the non-local interactions of the downstream dynamics.
    Figure 5 shows larger windows do not improve in-distribution test nRMSE, but the downstream tasks (RBC, shear flow) are more complex and were not part of the window-size ablation.
  • ad hoc to paper Separate-channel tokenization maintains a consistent information density across channel counts, which is why SC improves transfer.
    Motivated qualitatively in Section 3.1 and supported only by the downstream gains of Table 3; the mechanism itself is not directly measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations." pith.science (2026). https://pith.science/paper/X37RSAO6

@misc{pith2026250524717,
  author       = {Pith},
  title        = {Pith review of: PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X37RSAO6}},
  note         = {Machine review of arXiv:2505.24717}
}
read the original abstract

We introduce PDE-Transformer, an improved transformer-based architecture for surrogate modeling of physics simulations on regular grids. We combine recent architectural improvements of diffusion transformers with adjustments specific for large-scale simulations to yield a more scalable and versatile general-purpose transformer architecture, which can be used as the backbone for building large-scale foundation models in physical sciences. We demonstrate that our proposed architecture outperforms state-of-the-art transformer architectures for computer vision on a large dataset of 16 different types of PDEs. We propose to embed different physical channels individually as spatio-temporal tokens, which interact via channel-wise self-attention. This helps to maintain a consistent information density of tokens when learning multiple types of PDEs simultaneously. We demonstrate that our pre-trained models achieve improved performance on several challenging downstream tasks compared to training from scratch and also beat other foundation model architectures for physics simulations.

Figures

Figures reproduced from arXiv: 2505.24717 by the authors.

Figure 1
Figure 1. PDE-Transformer is a transformer model tailored to scientific data, the images above show its autoregressive predictions after 20 time steps on a large dataset comprising 16 different PDE dynamics, given initial conditions only. Additional simulation parameters (viscosity, domain extent, etc.) are unknown to the model and need to be inferred from the observed data. PDE-Transformer is especially well suited to be pre… view at source ↗
Figure 2
Figure 2. Architecture overview of PDE-Transformer. The multi-scale architecture combines up- and downsampling of tokens with skip connections between transformer stages of the same resolution. The attention operation is restricted to a local window of tokens. The window is shifted between two adjacent transformer blocks. Conditionings are embedded and used to scale and shift the intermediate token representations. The mixed … view at source ↗
Figure 3
Figure 3. Scaling the domain size for patch size p = 4. An input resolution of 256 × 256 corresponds to 64 × 64 tokens. The area of a each point corresponds to the required GPU memory for inference (batch size 1). Evaluation metrics For the evaluation, we use the nor￾malized RMSE, defined as nRMSE = 1 M X M i=1 s MSE(uˆout, uout) MSE(0, uout) , (4) where uˆout is the network prediction and M corresponds to the number of traje… view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: nRMSE evaluation of supervised training vs. training with flow matching and sampling from the posterior (top). Com￾parison of mixed channels (MC) vs. separate channel (SC) with axial attention over channel dimension (bottom). the following, showing its improved versati…
Figure 6
Figure 6. Figure 6: shows the results of this comparison. PDE-S with pre-training consistently yields more accurate predictions than the other baselines across the full range of difficult Well tasks. It provides an average improvement of 42% in terms of nRMSE, compared the second best mod…
Figure 7
Figure 7. Figure 7: Training loss of supervised (left) and diffusion training (right) for configurations S, B and L [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: We evaluate the nRMSE against the number of inference steps for PDE-Transformer with configuration L. We use the explicit Euler method to solve the ODE for sampling. B.2. Downstream Tasks Details of the comparison with the pre-trained Poseidon model scOT-B over time ar…
Figure 9
Figure 9. Figure 9: Average trajectory nRMSE of PDE-Transfomer and scOT with pre-trained weights on active matter tasks. A representative frame at f = 8 is shown left. B.3. Numerical Values [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Average rollout nRMSE of different models on downstream tasks. (Buehler & Buehler, 2024; Zeng et al., 2024). Despite these advancements, the application of LoRA in foundation models for partial differential equations (PDEs) remains relatively unexplored. Given the inp…
Figure 11
Figure 11. Figure 11: The number of trainable parameters of different sizes of PDE-Transformer w.r.t different LoRA ranks. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Performance of the LoRA finetuning on downstream tasks. C. Pre-training Datasets The sources for our datasets are chosen to ensure a wide range of different PDEs, at high spatial resolutions, and with varying physical quantities across the simulations in each dataset.…
Figure 13
Figure 13. Figure 13: Random example simulations from diff. Fisher-KPP (fisher) contains simulations of a reaction-diffusion system according to the Fisher-KPP equation. It describes how the concentration of a substance changes over time and space due to a reaction process controlled by a …
Figure 14
Figure 14. Figure 14: Random example simulations from fisher and sh (from top to bottom). Swift-Hohenberg (sh) features simulations of the Swift-Hohenberg equation, which describes certain pattern formation processes. It can be applied to describe the morphology of wrinkles in curved elast…
Figure 15
Figure 15. Figure 15: Random example simulations from steady configurations of the Gray-Scott model of a reaction-diffusion system: gs-delta, gs-theta, gs-iota, and gs-kappa. • Fields: velocity (x, y) • Varied Parameters: viscosity ∈ [0.00005, 0.0003[ • Validation Set: random 15% split of …
Figure 16
Figure 16. Figure 16: Random example simulations from unsteady configurations of the Gray-Scott model of a reaction-diffusion system:gs-alpha, gs-beta, gs-gamma, and gs-epsilon. • Time Step of Stored Data: 0.05 (with 10 substeps for the simulation) • Spatial Domain Size of Simulation: vari…
Figure 17
Figure 17. Figure 17: Random example simulations from test sets with longer rollout from gs-alpha, gs-beta, gs-gamma, and gs-epsilon. • Initial Conditions: random truncated Fourier / Gaussian random field / diffused noise • Boundary Conditions: periodic • Time Step of Stored Data: 0.5 (wit…
Figure 18
Figure 18. Figure 18: Random example simulations from burgers, kdv, ks, and the test set of ks with longer rollout. • Dimensionality: s = 600, t = 30, f = 1, x = 2048, y = 2048 • Initial Conditions: random truncated Fourier / Gaussian random field / diffused noise • Boundary Conditions: pe…
Figure 19
Figure 19. Figure 19: Random example simulations from decay-turb, and kolm-flow, with examples from each corresponding test set with longer rollout. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]
Figure 20
Figure 20. Figure 20: Visualizations of model’s prediction on diff, fisher and sh. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_20.png]
Figure 21
Figure 21. Figure 21 [PITH_FULL_IMAGE:figures/full_fig_p034_21.png]
Figure 22
Figure 22. Figure 22 [PITH_FULL_IMAGE:figures/full_fig_p035_22.png]
Figure 23
Figure 23. Figure 23: Visualizations of model’s prediction on burgers, kdv, ks, decay-turb and kolm-flow. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_23.png]
Figure 24
Figure 24. Figure 24: Random example simulations from active matter, Rayleigh-Benard convection, and shear flow. ´ 38 [PITH_FULL_IMAGE:figures/full_fig_p038_24.png]
Figure 25
Figure 25. Figure 25: Active Matter. Autoregressive prediction with pretrained PDE-S (SC). 39 [PITH_FULL_IMAGE:figures/full_fig_p039_25.png]
Figure 26
Figure 26. Figure 26: Rayleigh-Benard Convection. Autoregressive prediction with pretrained PDE-S (SC). ´ 40 [PITH_FULL_IMAGE:figures/full_fig_p040_26.png]
Figure 27
Figure 27. Figure 27: Shear Flow. Autoregressive prediction with pretrained PDE-S (SC). 41 [PITH_FULL_IMAGE:figures/full_fig_p041_27.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Extending Fourier Neural Operators for Modeling Parameterized and Coupled PDEs

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Lightweight hypernetwork bias modulation plus Fourier-space coupling yields FNOs that cut errors 55–72% on parameterized coupled PDEs, including a new 1D plasma sheath benchmark.

Reference graph

Works this paper leans on

95 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Convlora and adabn based domain adaptation via self-training

    Aleem, S., Dietlmeier, J., Arazo, E., and Little, S. Convlora and adabn based domain adaptation via self-training. In IEEE International Symposium on Biomedical Imaging, ISBI 2024, Athens, Greece, May 27-30, 2024 , pp.\ 1--5. IEEE , 2024. doi:10.1109/ISBI56570.2024.10635661. URL https://doi.org/10.1109/ISBI56570.2024.10635661

    arXiv 2024

  3. [3]

    C., Pletcher, R

    Anderson, D., Tannehill, J. C., Pletcher, R. H., Munipalli, R., and Shankar, V. Computational fluid mechanics and heat transfer. CRC press, 2020

    2020

  4. [4]

    M., Cholakkal, H., Shah, M., Yang, M.-H., and Khan, F

    Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., and Khan, F. S. Foundation models defining a new era in vision: a survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp.\ 1--20, 2025. doi:10.1109/TPAMI.2024.3506283

    work page arXiv 2025

  5. [5]

    All are worth words: A vit backbone for diffusion models

    Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, J. All are worth words: A vit backbone for diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp.\ 22669--22679. IEEE , 2023. URL https://doi.org/10.1109/CVPR52729.2023.02171

    arXiv 2023

  6. [6]

    and Sochen, N

    Bar, L. and Sochen, N. Unsupervised deep learning algorithm for pde-based forward and inverse problems. arXiv preprint arXiv:1904.05417, 2019

    arXiv 1904

  7. [7]

    Bar-Sinai , Y., Hoyer, S., Hickey, J., and Brenner, M. P. Learning data driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116 0 (31): 0 15344--15349, 2019. ISSN 0027-8424, 1091-6490. doi:10.1073/pnas.1814058116

    doi:10.1073/pnas.1814058116 2019

  8. [8]

    P., Lucic, A., Stanley, M., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J., Dong, H., Vaughan, A., et al

    Bodnar, C., Bruinsma, W. P., Lucic, A., Stanley, M., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J., Dong, H., Vaughan, A., et al. Aurora: A foundation model of the atmosphere. arXiv preprint arXiv:2405.13063, 2024

    arXiv 2024

Show all 95 references
  1. [9]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  2. [10]

    Neural galerkin schemes with active learning for high-dimensional evolution equations

    Bruna, J., Peherstorfer, B., and Vanden - Eijnden, E. Neural galerkin schemes with active learning for high-dimensional evolution equations. J. Comput. Phys., 496: 0 112588, 2024. doi:10.1016/J.JCP.2023.112588. URL https://doi.org/10.1016/j.jcp.2023.112588

    2024

  3. [11]

    Buehler, E. L. and Buehler, M. J. X-lora: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design, 2024. URL https://arxiv.org/abs/2402.07148

    2024 arXiv

  4. [12]

    J., Vasil, G

    Burns, K. J., Vasil, G. M., Oishi, J. S., Lecoanet, D., and Brown, B. P. Dedalus: A flexible framework for numerical simulations with spectral methods. Physical Review Research, 2 0 (2): 0 023068, 2020. doi:10.1103/PhysRevResearch.2.023068. URL https://doi.org/10.1103/PhysRevR...

    2020 doi

  5. [13]

    Choose a transformer: F ourier or G alerkin

    Cao, S. Choose a transformer: F ourier or G alerkin. In Advances in Neural Information Processing Systems (NeurIPS 2021), volume 34, 2021. URL https://openreview.net/forum?id=ssohLcmn4-r

    2021

  6. [14]

    Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Mach...

    2020

  7. [15]

    P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Ruiz, C

    Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Ruiz, C. R., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., van Steenkiste,...

    2023

  8. [16]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu...

    2019 doi

  9. [17]

    and Nichol, A

    Dhariwal, P. and Nichol, A. Q. Diffusion models beat gans on image synthesis. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems...

    2021

  10. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on...

    2021

  11. [19]

    C., Zepeda-Nunez, L., Smith, J., Brenner, M., and Hoyer, S

    Dresdner, G., Kochkov, D., Norgaard, P. C., Zepeda-Nunez, L., Smith, J., Brenner, M., and Hoyer, S. Learning to correct spectral methods for simulating turbulent flows. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=wNBARGxoJn

    2023

  12. [20]

    Turbulence modeling in the age of data

    Duraisamy, K., Iaccarino, G., and Xiao, H. Turbulence modeling in the age of data. Annual Review of Fluid Mechanics, 51 0 (1): 0 357--377, jan 2019. doi:10.1146/annurev-fluid-010518-040547. URL https://doi.org/10.1146

    2019 doi

  13. [21]

    Scaling rectified flow transformers for high-resolution image synthesis

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M \" u ller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., and Rombach, R. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first Internatio...

    2024

  14. [22]

    and The PyTorch Lightning team

    Falcon, W. and The PyTorch Lightning team . PyTorch Lightning , March 2019. URL https://github.com/Lightning-AI/lightning

    2019

  15. [23]

    and Zabaras, N

    Geneva, N. and Zabaras, N. Quantifying model form uncertainty in reynolds-averaged turbulence models with bayesian deep neural networks. Journal of Computational Physics, 383: 0 125--147, 2019. ISSN 0021-9991. doi:https://doi.org/10.1016/j.jcp.2019.01.021. URL https://www.scie...

    2019 doi

  16. [24]

    D., and Karniadakis, G

    Goswami, S., Kontolati, K., Shields, M. D., and Karniadakis, G. E. Deep transfer operator learning for partial differential equations under conditional shift. Nature Machine Intelligence, 4 0 (12): 0 1155--1164, 2022

    2022

  17. [25]

    B., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K

    Goyal, P., Doll \' a r, P., Girshick, R. B., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch SGD: training imagenet in 1 hour. CoRR, abs/1706.02677, 2017. URL http://arxiv.org/abs/1706.02677

    2017 arXiv

  18. [26]

    Beyond lora: Exploring efficient fine-tuning techniques for time series foundational models, 2024 a

    Gupta, D., Bhatti, A., and Parmar, S. Beyond lora: Exploring efficient fine-tuning techniques for time series foundational models, 2024 a . URL https://arxiv.org/abs/2409.11302

    2024 arXiv

  19. [27]

    Low-rank adaptation of time series foundational models for out-of-domain modality forecasting

    Gupta, D., Bhatti, A., Parmar, S., Dan, C., Liu, Y., Shen, B., and Lee, S. Low-rank adaptation of time series foundational models for out-of-domain modality forecasting. In Hung, H., Oertel, C., Soleymani, M., Chaspari, T., Dibeklioglu, H., Shukla, J., and Truong, K. P. (eds.)...

    2024

  20. [28]

    Gupta, J. K. and Brandstetter, J. Towards multi-spatiotemporal-scale generalized PDE modeling. Trans. Mach. Learn. Res., 2023, 2023. URL https://openreview.net/forum?id=dPSTDbGtBY

    2023

  21. [30]

    Poseidon: Efficient foundation models for pdes

    Herde, M., Raonic, B., Rohner, T., K \" a ppeli, R., Molinaro, R., de B \' e zenac, E., and Mishra, S. Poseidon: Efficient foundation models for pdes. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Inf...

    2024

  22. [31]

    Axial attention in multidimensional transformers

    Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T. Axial attention in multidimensional transformers. CoRR, abs/1912.12180, 2019. URL http://arxiv.org/abs/1912.12180

    1912 arXiv

  23. [32]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 6840--6851. Curran Associates, Inc., 2020

    2020

  24. [33]

    J., Vegetti, S., and Thuerey, N

    Holzschuh, B. J., Vegetti, S., and Thuerey, N. Solving inverse physics problems with score matching. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informati...

    2023

  25. [34]

    simple diffusion: End-to-end diffusion for high resolution images

    Hoogeboom, E., Heek, J., and Salimans, T. simple diffusion: End-to-end diffusion for high resolution images. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Ho...

    2023

  26. [35]

    J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022....

    2022

  27. [36]

    Cocogen: Physically consistent and conditioned score-based generative models for forward and inverse problems

    Jacobsen, C., Zhuang, Y., and Duraisamy, K. Cocogen: Physically consistent and conditioned score-based generative models for forward and inverse problems. SIAM J. Sci. Comput. , 47 0 (2): 0 399, 2025. doi:10.1137/24M1636071. URL https://doi.org/10.1137/24m1636071

    2025 doi

  28. [37]

    C., Thuerey, N., Kim, T., Gross, M., and Solenthaler, B

    Kim, B., Azevedo, V. C., Thuerey, N., Kim, T., Gross, M., and Solenthaler, B. Deep Fluids: A Generative Network for Parameterized Fluid Simulations . Comput. Graph. Forum, 38 0 (2): 0 12, 2019. URL http://www.byungsoo.me/project/deep-fluids/

    2019

  29. [38]

    A., Alieva, A., Wang, Q., Brenner, M

    Kochkov, D., Smith, J. A., Alieva, A., Wang, Q., Brenner, M. P., and Hoyer, S. Machine learning–accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118 0 (21): 0 e2101784118, 2021. doi:10.1073/pnas.2101784118. URL https://www.pnas.org/doi...

    2021 doi

  30. [39]

    Apebench: A benchmark for autoregressive neural emulators of pdes

    Koehler, F., Niedermayr, S., Westermann, R., and Thuerey, N. Apebench: A benchmark for autoregressive neural emulators of pdes. In Advances in Neural Information Processing Systems 37 (Datasets and Benchmarks Track), 2024. URL https://openreview.net/forum?id=iWc0qE116u

    2024

  31. [40]

    Benchmarking autoregressive conditional diffusion models for turbulent flow simulation, 2024

    Kohl, G., Chen, L.-W., and Thuerey, N. Benchmarking autoregressive conditional diffusion models for turbulent flow simulation, 2024. URL https://arxiv.org/abs/2309.01745

    2024 arXiv

  32. [41]

    J., Bonev, B., Azizzadenesheli, K., Berner, J., and Anandkumar, A

    Kossaifi, J., Kovachki, N., Li, Z., Pitt, D., Liu-Schiaffini, M., George, R. J., Bonev, B., Azizzadenesheli, K., Berner, J., and Anandkumar, A. A library for learning neural operators, 2024. URL https://arxiv.org/abs/2412.10354

    2024

  33. [42]

    B., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A

    Kovachki, N. B., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A. M., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. J. Mach. Learn. Res., 24: 0 89:1--89:97, 2023. URL https://jmlr.org/papers/v24/21-1524.html

    2023

  34. [43]

    B., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A

    Li, Z., Kovachki, N. B., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A. M., and Anandkumar, A. Neural operator: Graph kernel network for partial differential equations. CoRR, abs/2003.03485, 2020. URL https://arxiv.org/abs/2003.03485

    2003 arXiv

  35. [44]

    Physics-informed neural operator for learning partial differential equations

    Li, Z., Zheng, H., Kovachki, N., Jin, D., Chen, H., Liu, B., Azizzadenesheli, K., and Anandkumar, A. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794, 2021

    2021 arXiv

  36. [45]

    Li, Z., Meidani, K., and Farimani, A. B. Transformer for partial differential equations operator learning. Transactions on Machine Learning Research, 2023 a . ISSN 2835-8856. URL https://openreview.net/forum?id=EPPqt3uERT

    2023

  37. [46]

    Li, Z., Shu, D., and Farimani, A. B. Scalable transformer for PDE surrogate modeling. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing S...

    2023

  38. [47]

    Foundation models for time series analysis: A tutorial and survey

    Liang, Y., Wen, H., Nie, Y., Jiang, Y., Jin, M., Song, D., Pan, S., and Wen, Q. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp.\ 6555–6565, New York, NY, US...

    2024

  39. [48]

    Tracking meets lora: Faster training, larger model, stronger performance

    Lin, L., Fan, H., Zhang, Z., Wang, Y., Xu, Y., and Ling, H. Tracking meets lora: Faster training, larger model, stronger performance. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vision - ECCV 2024 - 18th European Conferen...

    2024 doi

  40. [49]

    Lipman, Y., Chen, R. T. Q., Ben - Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=...

    2023

  41. [50]

    and Thuerey, N

    Liu, Q. and Thuerey, N. Uncertainty-aware surrogate models for airfoil flow simulations with denoising diffusion probabilistic models. AIAA Journal, 62 0 (8): 0 2912--2933, 2024. doi:10.2514/1.J063440. URL https://doi.org/10.2514/1.J063440

    2024 doi

  42. [51]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/...

    2023

  43. [52]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  44. [54]

    Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3 0 (3): 0 218--229, 2021

    2021

  45. [55]

    Lcm-lora: A universal stable-diffusion acceleration module, 2023

    Luo, S., Tan, Y., Patil, S., Gu, D., von Platen, P., Passos, A., Huang, L., Li, J., and Zhao, H. Lcm-lora: A universal stable-diffusion acceleration module, 2023. URL https://arxiv.org/abs/2311.05556

    2023 arXiv

  46. [56]

    Maddu, S., Weady, S., and Shelley, M. J. Learning fast, accurate, and stable closures of a kinetic theory of an active fluid. Journal of Computational Physics, 504: 0 112869, 2024. URL https://doi.org/10.1016/j.jcp.2024.112869

    2024

  47. [57]

    A survey on lora of large language models

    Mao, Y., Ge, Y., Fan, Y., Xu, W., Mi, Y., Hu, Z., and Gao, Y. A survey on lora of large language models. Frontiers of Computer Science, 19 0 (7): 0 197605, 2025

    2025

  48. [58]

    Comparing vision transformers and convolutional neural networks for image classification: A literature review

    Maurício, J., Domingues, I., and Bernardino, J. Comparing vision transformers and convolutional neural networks for image classification: A literature review. Applied Sciences, 13 0 (9), 2023. ISSN 2076-3417. doi:10.3390/app13095521. URL https://www.mdpi.com/2076-3417/13/9/5521

    2023 doi

  49. [59]

    R.-S., Parker, L., Ohana, R., Cranmer, M., Bietti, A., Eickenberg, M., Golkar, S., Krawezik, G., Lanusse, F., Pettee, M., Tesileanu, T., Cho, K., and Ho, S

    McCabe, M., Blancard, B. R.-S., Parker, L., Ohana, R., Cranmer, M., Bietti, A., Eickenberg, M., Golkar, S., Krawezik, G., Lanusse, F., Pettee, M., Tesileanu, T., Cho, K., and Ho, S. Multiple physics pretraining for physical surrogate models. In NeurIPS 2023 AI for Science Work...

    2023

  50. [60]

    T., Morel, R., Agocs, F

    Ohana, R., McCabe, M., Meyer, L. T., Morel, R., Agocs, F. J., Beneitez, M., Berger, M., Burkhart, B., Dalziel, S. B., Fielding, D. B., Fortunato, D., Goldberg, J. A., Hirashima, K., Jiang, Y.-F., Kerswell, R., Maddu, S., Miller, J. M., Mukhopadhyay, P., Nixon, S. S., Shen, J.,...

    2024

  51. [61]

    Pearson, J. E. Complex patterns in a simple system. Science, 261 0 (5118): 0 189--192, 1993. doi:10.1126/science.261.5118.189. URL https://www.science.org/doi/abs/10.1126/science.261.5118.189

    1993 doi

  52. [62]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 4172--4182, 2023. doi:10.1109/ICCV51070.2023.00387

    2023

  53. [63]

    Perez, E., Strub, F., de Vries, H., Dumoulin, V., and Courville, A. C. Film: Visual reasoning with a general conditioning layer. In McIlraith, S. A. and Weinberger, K. Q. (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th i...

    2018

  54. [64]

    Improving language understanding by generative pre-training

    Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. OpenAI blog, 2018

    2018

  55. [65]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  56. [66]

    Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378: 0 686--707, 2019

    2019

  57. [67]

    Ren, S., Padilla, W., and Malof, J. M. Benchmarking deep inverse models over time, and the neural-adjoint method. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Info...

    2020

  58. [68]

    Comprehensive comparison between vision transformers and convolutional neural networks for face recognition tasks

    Rodrigo, M., Cuevas, C., and Garc \' a, N. Comprehensive comparison between vision transformers and convolutional neural networks for face recognition tasks. Scientific reports, 14 0 (1): 0 21392, 2024

    2024

  59. [69]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pp.\ 10674--10685. IEEE , 2022....

    2022

  60. [70]

    U-net: Convolutional networks for biomedical image segmentation

    Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention--MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, ...

    2015

  61. [71]

    and MacArt, J

    Sirignano, J. and MacArt, J. F. Deep learning closure models for large-eddy simulation of flows around bluff bodies. Journal of Fluid Mechanics, 966, jul 2023. doi:10.1017/jfm.2023.446. URL https://doi.org/10.1017

    2023 doi

  62. [72]

    Smith, G. D. Numerical solution of partial differential equations: finite difference methods. Oxford university press, 1985

    1985

  63. [73]

    S., Hsu, Y., Zhang, L., Hua, T., Kira, Z., Shen, Y., and Jin, H

    Smith, J. S., Hsu, Y., Zhang, L., Hua, T., Kira, Z., Shen, Y., and Jin, H. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. Trans. Mach. Learn. Res., 2024, 2024. URL https://openreview.net/forum?id=TZdEgwZ6f3

    2024

  64. [74]

    W., and Gholami, A

    Subramanian, S., Harrington, P., Keutzer, K., Bhimji, W., Morozov, D., Mahoney, M. W., and Gholami, A. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Lev...

    2023

  65. [75]

    Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data

    Sun, L., Gao, H., Pan, S., and Wang, J.-X. Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361: 0 112732, 2020. ISSN 0045-7825. doi:https://doi.org/10.1016/j.cma.2019....

    2020

  66. [76]

    Comparison of vision transformers and convolutional neural networks in medical image analysis: a systematic review

    Takahashi, S., Sakaguchi, Y., Kouno, N., Takasawa, K., Ishizu, K., Akagi, Y., Aoyama, R., Teraya, N., Bolatkan, A., Shinkai, N., et al. Comparison of vision transformers and convolutional neural networks in medical image analysis: a systematic review. Journal of Medical System...

    2024

  67. [77]

    Learning neural PDE solvers with parameter-guided channel attention

    Takamoto, M., Alesiani, F., and Niepert, M. Learning neural PDE solvers with parameter-guided channel attention. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023...

    2023

  68. [78]

    Physics-based Deep Learning

    Thuerey, N., Holzschuh, B., Holl, P., Kohl, G., Lino, M., Liu, Q., Schnell, P., and Trost, F. Physics-based Deep Learning. WWW, 2021. URL https://physicsbaseddeeplearning.org

    2021

  69. [79]

    U-dits: Downsample tokens in u-shaped diffusion transformers

    Tian, Y., Tu, Z., Chen, H., Hu, J., Xu, C., and Wang, Y. U-dits: Downsample tokens in u-shaped diffusion transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=SRWs2wxNs7

    2024

  70. [80]

    R., Holl, P., and Thuerey, N

    Um, K., Brand, R., Fei, Y. R., Holl, P., and Thuerey, N. Solver-in-the-loop: Learning from differentiable physics to interact with iterative pde-solvers. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Syst...

    2020

  71. [81]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp.\ 6000–6010, Red Hook, NY, USA, 201...

    2017

  72. [82]

    Mosaic flows: A transferable deep learning framework for solving pdes on unseen domains

    Wang, H., Planas, R., Chandramowlishwaran, A., and Bostanabad, R. Mosaic flows: A transferable deep learning framework for solving pdes on unseen domains. Computer Methods in Applied Mechanics and Engineering, 389: 0 114424, 2022

    2022

  73. [83]

    Z., Khabsa, M., Fang, H., and Ma, H

    Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. CoRR, abs/2006.04768, 2020. URL https://arxiv.org/abs/2006.04768

    2006 arXiv

  74. [84]

    Online-lora: Task-free online continual learning via low rank adaptation

    Wei, X., Li, G., and Marculescu, R. Online-lora: Task-free online continual learning via low rank adaptation. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2025, Tucson, AZ, USA, February 26 - March 6, 2025 , pp.\ 6634--6645. IEEE , 2025. doi:10.1109/W...

    2025

  75. [85]

    T., Balles, L., and Zappella, G

    Wistuba, M., Sivaprasad, P. T., Balles, L., and Zappella, G. Continual learning with low rank adaptation, 2023. URL https://arxiv.org/abs/2311.17601

    2023 arXiv

  76. [86]

    Transolver: A fast transformer solver for pdes on general geometries

    Wu, H., Luo, H., Wang, H., Wang, J., and Long, M. Transolver: A fast transformer solver for pdes on general geometries. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024 a . URL https://openreview.n...

    2024

  77. [87]

    Wu, P., Li, K., Wang, T., Dong, Y., Leung, V. C. M., and Wang, F. Fedfmsl: Federated learning of foundation models with sparsely activated lora. IEEE Transactions on Mobile Computing, 23 0 (12): 0 15167--15181, 2024 b . doi:10.1109/TMC.2024.3454634

    2024

  78. [88]

    Transfer learning enhanced deeponet for long-time prediction of evolution equations

    Xu, W., Lu, Y., and Wang, L. Transfer learning enhanced deeponet for long-time prediction of evolution equations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 10629--10636, 2023

    2023

  79. [89]

    Yang, L., Liu, S., Meng, T., and Osher, S. J. In-context operator learning with data prompts for differential equation problems. Proceedings of the National Academy of Sciences, 120 0 (39): 0 e2310142120, 2023 a . doi:10.1073/pnas.2310142120. URL https://www.pnas.org/doi/abs/1...

    2023 doi

  80. [90]

    Low-rank adaptation for foundation models: A comprehensive review, 2024

    Yang, M., Chen, J., Zhang, Y., Liu, J., Zhang, J., Ma, Q., Verma, H., Zhang, Q., Zhou, M., King, I., and Ying, R. Low-rank adaptation for foundation models: A comprehensive review, 2024. URL https://arxiv.org/abs/2501.00365

    2024

  81. [91]

    Foundation models for decision making: Problems, methods, and opportunities, 2023 b

    Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., and Schuurmans, D. Foundation models for decision making: Problems, methods, and opportunities, 2023 b . URL https://arxiv.org/abs/2303.04129

    2023 arXiv

  82. [92]

    Florence: A new foundation model for computer vision, 2021

    Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., Liu, C., Liu, M., Liu, Z., Lu, Y., Shi, Y., Wang, L., Wang, J., Xiao, B., Xiao, Z., Yang, J., Zeng, M., Zhou, L., and Zhang, P. Florence: A new foundation model for computer visi...

    2021 arXiv

  83. [93]

    Parameter-efficient fine-tuning on large protein language models improves signal peptide prediction

    Zeng, S., Wang, D., Jiang, L., and Xu, D. Parameter-efficient fine-tuning on large protein language models improves signal peptide prediction. Genome research, 34 0 (9): 0 1445--1454, 2024

    2024

  84. [94]

    and Lee, K

    Zeng, Y. and Lee, K. The expressive power of low-rank adaptation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=likXVjmh3E

    2024

  85. [95]

    and Sennrich, R

    Zhang, B. and Sennrich, R. Root mean square layer normalization. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d'Alch \' e - Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process...

    2019

  86. [96]

    P., Norman, M., Stefanski, D., and Gounley, J

    Zhang, P., Laiu, M. P., Norman, M., Stefanski, D., and Gounley, J. MATEY : multiscale adaptive foundation models for spatiotemporal physical systems. arXiv preprint arXiv:2412.20601, 2024 a

    2024 arXiv

  87. [97]

    Fed-piLot : Optimizing lora assignment for efficient federated foundation model fine-tuning, 2024 b

    Zhang, Z., Xu, J., Liu, P., and Hu, R. Fed-piLot : Optimizing lora assignment for efficient federated foundation model fine-tuning, 2024 b . URL https://arxiv.org/abs/2410.10200

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.