Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Jigsaw: Training Multi-Billion-Parameter AI Weather Models with Optimized Model Parallelism

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Jigsaw splits data and model to train billion-parameter weather models

desk verdict Jigsaw is a clean, well-measured parallelization scheme for large-input ML training, but the 'exceeds SOTA' claim leans on an apples-to-oranges Megatron-LM comparison. read the letter →

arxiv 2507.05753 v1 pith:7HQHIIKO submitted 2025-07-08 cs.LG

Deifilia Kieckhefen , Markus Götz , Lars H. Heyen , Achim Streit , Charlotte Debus This is my paper
classification cs.LG
keywords WeatherMixerJigsawparallelismmodeldomaintensorforecastingrooflineanalysisMLP-Mixer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces two coupled pieces: WeatherMixer, an MLP-based weather model whose workload scales linearly with input size, and Jigsaw, a model-parallel scheme that shards both the model weights and each data sample across GPUs so no accelerator ever holds a full copy. The claim is that Jigsaw removes memory redundancy and the data-loading bottleneck, allowing efficient training of billion-parameter weather models on up to 256 GPUs. At that scale the authors report 9 and 11 PFLOPs, 23% and 28% of theoretical peak, with scaling efficiencies of 68% and 72% versus 51% for the non-model-parallel baseline. If correct, this tackles the two constraints that block higher-resolution, longer-lead AI weather forecasting: accelerator memory and I/O bandwidth.

What carries the argument

Jigsaw parallelism is the central mechanism: a linear layer's matrix-matrix multiplication $XW^T$ is partitioned so that in 2-way parallelism both data and weights are split along the channel dimension, and in 4-way parallelism along both longitude and channel dimensions, with processes exchanging only the partial sums $X_i W_j^T$ via non-blocking point-to-point MPI while computing local terms concurrently. Because no process ever gathers all parameters, memory redundancy is zero and the data loader can read only each process's partition, which is what turns I/O bandwidth from a bottleneck into a source of superscalar weak scaling. WeatherMixer supports this because it reduces the model to a sequence of matrix-matrix multiplications whose workload scales linearly with the number of spatial tokens, unlike the quadratic cost of self-attention.

What would settle it

Measure the actual FLOPs of the backward pass and layer norms on a small model at 0.25 TFLOPs per forward pass with TF32 and see whether the workload is still classified as I/O-bound; if the backward FLOPs differ from twice the forward pass, the superscalar weak-scaling result may be an artifact of the FLOPs model.

Watch

Extended reading notes

Core claim

The central discovery is that combining domain parallelism with tensor parallelism in a distributed matrix-matrix multiplication removes the allgather and broadcast steps and the memory duplication of standard model-parallel training while also partitioning the data-loading work. The paper reports that this scheme exceeds the strong-scaling results of Megatron-LM in computation-communication-limited settings and shows superscalar weak scaling when training is I/O-bandwidth-limited, because each GPU reads only its own slice of the data. At the largest scale, training WeatherMixer on 256 NVIDIA A100 GPUs reaches 9 PFLOPs with 2-way Jigsaw and 11 PFLOPs with 4-way Jigsaw, 23% and 28% of theoretical peak, with scaling efficiencies of 68% and 72% versus 51% for the non-model-parallel baseline. In addition, the 1-billion-parameter WeatherMixer trained with Jigsaw reaches forecast RMSE below the IFS ENS-1 ensemble member on key variables, using only a 6-hour subsample of ERA5.

Load-bearing premise

The scaling analysis assumes that the backward pass has exactly twice the FLOPs of the forward pass and that layer norms, reductions, and dropout contribute negligibly, so the roofline classification into I/O-bound versus computation-communication-bound regimes depends on that FLOPs accounting being accurate for all model sizes.

Editorial extensions

If this is right

  • Training multi-billion-parameter weather models no longer requires model copies on every GPU, so larger models fit on the same hardware.
  • Partitioned data loading makes I/O-bound training faster with more parallelism, a regime that currently dominates operational training at 0.25 degree resolution.
  • Because Jigsaw uses smaller global batch sizes at fixed compute budget, it mitigates large-batch effects and can improve final forecast skill.
  • Jigsaw can be combined with conventional data parallelism across nodes and with pipeline parallelism, giving a route to scale beyond 256 GPUs.
  • The roofline analysis provides a method to decide beforehand whether a given model size and precision will be I/O-bound or computation-communication-bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same domain-plus-tensor partitioning could apply to other scientific ML workloads with gigabyte-sized samples, such as climate simulations, where I/O bandwidth is the bottleneck.
  • A natural testable extension is arbitrary n-way Jigsaw, for example 8-way splitting latitude, longitude, and channels, to scale within nodes with more than four GPUs.
  • If data loading is moved to faster storage or pre-staged in memory, the superscalar weak-scaling advantage shrinks and communication overhead at high parallelism should dominate; the paper's roofline framework predicts where that crossover lies.
  • The FLOPs-accounting assumption could be checked directly: if layer norms, reductions, and dropout contribute non-negligibly in smaller models, some I/O-bound classifications may shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents two contributions: WeatherMixer, an MLP-Mixer-based weather forecasting architecture whose compute grows linearly with input size, and Jigsaw, a model-parallelization scheme that combines domain parallelism with tensor parallelism to shard data, weights, and optimizer states within a node. The authors derive the communication pattern for 2- and 4-way Jigsaw parallelism, implement it in PyTorch with MPI point-to-point communication, and evaluate it on the ERA5/WeatherBench task. They report forecast skill above the IFS ENS-1 ensemble member, roofline analyses for single and mixed precision, strong- and weak-scaling measurements on A100 GPUs, and a system-wide weak-scaling study up to 256 GPUs reaching 9-11 PFLOPs (23%-28% of theoretical peak) with scaling efficiencies of 68%-72% for the model-parallel variants versus 51% for the non-model-parallel baseline.

Significance. If the claims hold, the paper makes a useful systems contribution for training large, data-heavy scientific ML models: Jigsaw's point-to-point sharded communication with overlapped computation is a clean design, the open-source implementation and energy reporting are strengths, and the linear-scaling WeatherMixer is a sensible alternative to Transformers for spatiotemporal data. The internal scaling results on A100s appear internally consistent and are valuable as a demonstration. However, the headline claim of exceeding state-of-the-art strong scaling is not supported by the uncontrolled comparison to Megatron-LM, and the 'zero memory redundancy' claim is too strong as stated. The FLOPs-model assumptions underlying the roofline classification also need verification before the superscalar weak-scaling conclusion can be fully accepted.

major comments (3)
  1. [Abstract and §6.3.2] The claim that Jigsaw 'exceeds state-of-the-art performance in strong scaling' is not established by the evidence presented. Section 6.3.2 compares measured speedups of 1.9 and 2.7 to Megatron-LM's published speedups of 1.6 and 2.3, but the latter were obtained on NVIDIA V100 hardware with a Transformer architecture and column/row-parallel allreduce communication, whereas Jigsaw is measured on A100s with an MLP-Mixer and point-to-point sharded communication. Strong-scaling speedups depend on hardware, architecture, batch size, and precision, so this is not a controlled comparison. The internal numbers are interesting, but the abstract and conclusion should not claim general superiority; they should be rephrased as 'competitive with published Megatron-LM results' or supplemented with a same-system, same-architecture baseline against an established model-parallel method.
  2. [§4 and §5] The paper's repeated claim of 'zero memory redundancy' (Section 4, abstract, conclusion) is contradicted by the 4-way layer norm design described in Section 5. There, ranks 0 and 2, and ranks 1 and 3, hold normalization parameters for the same channels and must perform pair-wise gradient reductions to keep those parameters synchronized. That means duplicated layer-norm parameters and optimizer states exist, which is a form of memory redundancy beyond 'necessary buffers for communication.' The claim should be qualified to linear-layer weights or the layer-norm replication should be accounted for in the memory model.
  3. [§6.3] The roofline classification into I/O-bound versus computation-communication-bound regimes, and hence the superscalar weak-scaling claim, rests on the FLOPs model stated in Section 6.3: the backward pass is assumed to be exactly twice the forward pass, and FLOPs in layer norms, reductions, and dropout are assumed negligible. No profiler measurements or sensitivity analysis are provided to verify these assumptions across the model sizes in Table 1. For the smallest models (0.25-1 TFLOPs per forward pass), non-matmul operations could be a non-negligible share of the runtime and FLOPs, which would shift the operational intensity on the roofline and potentially change the I/O-bound classification. The authors should verify the FLOPs count with a profiler or demonstrate that the classification is robust to a plausible range of FLOPs underestimation.
minor comments (5)
  1. [§6.3.3] The text first states that weak-scaling experiments use 1, 4, and 16 TFLOPs per forward pass per GPU, but later refers to 'the largest model (64 TFLOPs/forward pass)'; Table 2 lists 64 TFLOPs for the 4-way case. Please clarify which model sizes were actually used in the weak-scaling measurements.
  2. [References] References [20] and [21] are the same FourCastNet paper and should be merged to avoid duplication.
  3. [§6.2.1] The claimed 2-9% RMSE improvement of the 2-way model over the naive model is reported without error bars or multiple seeds; given that only a single training run is shown in Figure 4, the reader cannot assess whether this difference is significant. Adding run-to-run variance or at least stating the number of seeds would strengthen the large-batch-effect discussion.
  4. [§5] There is a typo: 'beetween' should be 'between' in the layer-norm description.
  5. [§6.2.2] The sentence 'this represents an upper-bound of the RMSE values achievable by this model architecture as per neural scaling laws' appears to state the opposite of what is meant; more training data should lower RMSE, so this should be a lower bound on achievable RMSE or should be rephrased to say the reported RMSE is an upper bound on skill.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: central scaling and forecast-skill claims are measured against external references; self-citations are peripheral.

full rationale

The paper's central claims are empirical measurements against external benchmarks, not derivations from their own assumptions. Strong and weak scaling numbers are measured wall-clock times on HoreKa A100 GPUs; the 9/11 PFLOPs figures follow from a stated FLOPs model (forward pass, backward pass at 2x, norms/reductions/dropout neglected) and measured runtimes. That FLOPs model is an assumption that could affect roofline classification and efficiency claims, but it is not fitted to the target results and does not make any prediction equal to an input by construction. The 'zero memory redundancy' property follows from the paper's own definition of an n-way parallel model, but it is a design definition rather than a derived prediction, so it is not a circular step in the sense of a claim that reduces to its input. The comparison to Megatron-LM's published 1.6/2.3 strong-scaling speedups on different hardware and a different architecture is uncontrolled and weakens the 'exceeds state-of-the-art' phrasing, but an uncontrolled external comparison is a correctness/validity concern, not circularity. Self-citations appear: ref. [9] for reporting energy consumption and ref. [37] for prior evidence of large-batch effects in atmospheric models. Ref. [37] supports an explanation of why model-parallel training with smaller global batch sizes converges to lower loss, but the paper also provides its own direct empirical evidence in Figure 4, and the scaling and performance claims do not rest on this citation. The forecast-skill comparisons use WeatherBench, Pangu-Weather, IFS HRES, and IFS ENS-1 as external references, and the pressure-level loss weights are hand-set rather than fitted to the validation targets. No load-bearing step reduces to a self-citation or to a fitted parameter renamed as a prediction. The claimed superscalar weak scaling is an observed consequence of sharding data loading in an I/O-bound regime, which is a real mechanism rather than a tautology. Overall, the derivation chain is self-contained apart from minor, non-load-bearing self-citations; score 1.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central scaling claims rest on a FLOPs model that neglects certain layers and on the chosen model dimensions; the forecast-skill claims rest on hand-set loss weights and the ERA5 subset. No new physical entities are introduced.

free parameters (2)
  • Pressure-level loss weights = [1,1,1,1,1,1,0.9,0.8,0.7,0.6,0.5,0.4,0.3]
    Hand-adapted weighting of pressure-level variables in the loss, based on meteorological feedback (Section 6). Affects the forecast-skill comparison but not the scaling analysis.
  • Scaling model dimensions (d_emb, d_tok, d_ch) = Model-specific, e.g., model 7: d_emb=4896, d_tok=8640, d_ch=4896 (Table 1)
    Chosen by hand to realize target FLOPs per forward pass for the roofline and scaling experiments (Section 6.3). These choices define the workload values and the memory-bound limits.
assumptions (5)
  • standard math Block matrix multiplication partition in Eq. (3)-(4) correctly computes XW^T with the described communication pattern
    Standard block matrix product identity; the communication pattern follows from the equation.
  • domain assumption Token-mixing MLP applied across all spatial tokens learns global weather patterns
    WeatherMixer's architectural premise (Section 3): an MLP on the token dimension gives global spatial mixing without quadratic attention.
  • ad hoc to paper FLOPs in layer norms, reductions, and dropout are negligible; backward pass = 2x forward pass FLOPs
    Stated in Section 6.3; this FLOPs model drives the roofline x-axis and the I/O-bound vs compute-bound classification.
  • domain assumption ERA5 0.25-degree data with the selected variables and 6-hour subsampling is sufficient to train a competitive forecasting model
    Dataset choice in Section 6; underlies the forecast-skill claims.
  • ad hoc to paper Megatron-LM's published strong-scaling speedups (1.6, 2.3) are a valid state-of-the-art baseline
    Used in Section 6.3.2 to claim Jigsaw exceeds SOTA; baseline is from different hardware, model, and framework.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Jigsaw: Training Multi-Billion-Parameter AI Weather Models with Optimized Model Parallelism." pith.science (2026). https://pith.science/paper/7HQHIIKO

@misc{pith2026250705753,
  author       = {Pith},
  title        = {Pith review of: Jigsaw: Training Multi-Billion-Parameter AI Weather Models with Optimized Model Parallelism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7HQHIIKO}},
  note         = {Machine review of arXiv:2507.05753}
}
read the original abstract

AI-based methods have revolutionized atmospheric forecasting, with recent successes in medium-range forecasting spurring the development of climate foundation models. Accurate modeling of complex atmospheric dynamics at high spatial resolutions and longer lead times requires large neural networks and gigabyte-sized data samples, making accelerator memory and I/O-bandwidth the bottlenecks for model training. We introduce WeatherMixer, a multi-layer-perceptron-based architecture whose workload scales linearly with input size, allowing the model to learn global weather phenomena at accuracies similar to numerical weather prediction. To cope with the computational demand, we propose Jigsaw, a novel model parallelization scheme that employs both domain and tensor parallelism, eliminating memory redundancy. Jigsaw exceeds state-of-the-art performance in strong scaling in compute-communication-limited systems and achieves superscalar weak scaling in I/O-bandwidth-limited systems. We scale training to 256 GPUs, reaching peak performances of 9 and 11 PFLOPs, 23% and 28% of theoretical peaks, achieving 68% and 72% scaling efficiency versus 51% without model parallelism.

Figures

Figures reproduced from arXiv: 2507.05753 by the authors.

Figure 1
Figure 1. Jigsaw parallelism applied to WeatherMixer model. Jigsaw applies domain parallelism over individual data samples, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. 4-way MLP-Mixer schematic. The MLP-Mixer consists of two MLPs: First, an MLP is applied across the spatial tokens, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Validation loss of three model sizes, trained on [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Validation RMSE of best-performing WM model [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Roofline plot for training models with 1-, 2-, 4-way [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Weak scaling experiments for training models [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Efficiency for weak scaling experiments combining [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 17 canonical work pages

  1. [9]

    Charlotte Debus, Marie Piraud, Achim Streit, Fabian Theis, and Markus Götz

    work page

  2. [37]

    D. To, J. Quinting, G. A. Hoshyaripour, M. Götz, A. Streit, and C. Debus. 2024. Architectural insights into and training methodology optimization of Pangu- Weather. Geoscientific Model Development 17, 23 (2024), 8873–8884. doi:10.5194/ gmd-17-8873-2024

    work page 2024

  3. [1]

    Tal Ben-Nun and Torsten Hoefler. 2019. Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. ACM Computing Surveys (CSUR) 52, 4 (2019), 1–43. doi:10.1145/3320060

    doi:10.1145/3320060 2019

  4. [2]

    Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. 2023. Accurate medium-range global weather forecasting with 3D neural networks. Nature 619, 7970 (7 2023), 533—-538. doi:10.1038/s41586-023-06185-3

    doi:10.1038/s41586-023-06185-3 2023

  5. [3]

    Bruinsma, Ana Lucic, Megan Stanley, Anna Vaughan, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan A

    Cristian Bodnar, Wessel P. Bruinsma, Ana Lucic, Megan Stanley, Anna Vaughan, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan A. Weyn, Haiyu Dong, Jayesh K. Gupta, Kit Thambiratnam, Alexander T. Archibald, Chun-Chieh Wu, Elizabeth Heider, Max Welling, Richard E. Turner, and Paris Perdikaris. 2024. A Foundation Model for the Earth System. arX...

    arXiv 2024

  6. [4]

    Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. 2023. Spherical Fourier neural operators: learning stable dynamics on the sphere. In Proceedings of the 40th International Conference on Machine Learning (ICML’23) . JMLR.org, Hon- olulu, Hawaii, USA, Article 117, 18 pages. https://proc...

    work page 2023

  7. [5]

    Lei Chen, Xiaohui Zhong, Feng Zhang, Yuan Cheng, Yinghui Xu, Yuan Qi, and Hao Li. 2023. FuXi: A cascade machine learning forecasting system for 15-day global weather forecast. arXiv:2306.12873 [physics.ao-ph] https://arxiv.org/abs/ 2306.12873

    arXiv 2023

  8. [6]

    Arik, and Tomas Pfis- ter

    Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O. Arik, and Tomas Pfis- ter. 2023. TSMixer: An All-MLP Architecture for Time Series Forecasting. arXiv:2303.06053 [cs.LG] https://arxiv.org/abs/2303.06053

    arXiv 2023

Show all 48 references
  1. [7]

    Daniel Coquelin, Charlotte Debus, Markus Götz, Fabrice von der Lehr, James Kahn, Martin Siggel, and Achim Streit. 2022. Accelerating neural network train- ing with distributed asynchronous and selective optimization (DASO). Journal of Big Data 9, 1 (2022), 14. doi:10.1186/s405...

    2022 doi

  2. [8]

    Christopher De Sa, Matthew Feldman, Christopher Ré, and Kunle Olukotun

  3. [10]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognit...

    2021 arXiv

  4. [11]

    Peter D Dueben and Peter Bauer. 2018. Challenges and design choices for global weather and climate models based on machine learning. Geoscientific Model Development 11, 10 (2018), 3999–4009. doi:10.5194/gmd-11-3999-2018

    2018 doi

  5. [12]

    Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’23) . ACM, L...

    2023

  6. [13]

    Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gian- paolo Balsamo, Peter Bechtold, Gionata Biavati, Jean...

    2020

  7. [14]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al . 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3. https://openreview.net/forum? id=nZeVKeeFYf9

    2022

  8. [15]

    Petra Icha, Thomas Lauf, and Gunter Kuhs. 2021. Entwicklung der spezifis- chen Kohlendioxid - Emissionen des deutschen Strommix in den Jahren 1990–

    2021

  9. [16]

    Inness, M

    A. Inness, M. Ades, A. Agustí-Panareda, J. Barré, A. Benedictow, A.-M. Blech- schmidt, J. J. Dominguez, R. Engelen, H. Eskes, J. Flemming, V. Huijnen, L. Jones, Z. Kipling, S. Massart, M. Parrington, V.-H. Peuch, M. Razinger, S. Remy, M. Schulz, and M. Suttie. 2019. The CAMS r...

    2019 doi

  10. [17]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs.LG] https: //arxiv.org/abs/2001.08361

    2020 arXiv

  11. [18]

    Ryan Keisler. 2022. Forecasting Global Weather with Graph Neural Networks. arXiv:2202.07575 [physics.ao-ph] https://arxiv.org/abs/2202.07575

    2022 arXiv

  12. [19]

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2017. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. arXiv:1609.04836 [cs.LG] https://arxiv. org/abs/1609.04836

    2017 arXiv

  13. [21]

    Thorsten Kurth, Shashank Subramanian, Peter Harrington, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Miele, Karthik Kashinath, and Anima Anandkumar. 2023. FourCastNet: Accelerating Global High-Resolution Weather Forecasting Using Adaptive Fourier Neural Operators. In Pr...

    2023

  14. [22]

    Thorsten Kurth, Sean Treichler, Joshua Romero, Mayur Mudigonda, Nathan Luehr, Everett Phillips, Ankur Mahesh, Michael Matheson, Jack Deslippe, Massi- miliano Fatica, Prabhat, and Michael Houston. 2018. Exascale deep learning for climate analytics. In Proceedings of the Interna...

    2018

  15. [23]

    Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and Pe- ter Battaglia

    Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton- Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Kieckhefen et al. Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Sh...

    2023 doi

  16. [24]

    Simon Lang, Mihai Alexe, Matthew Chantry, Jesper Dramsch, Florian Pin- ault, Baudouin Raoult, Mariana C. A. Clare, Christian Lessig, Michael Maier- Gerber, Linus Magnusson, Zied Ben Bouallègue, Ana Prieto Nemesio, Peter D. Dueben, Andrew Brown, Florian Pappenberger, and Floren...

    2024 arXiv

  17. [25]

    Robertson, and José A

    Hyuk-Jae Lee, James P. Robertson, and José A. B. Fortes. 1997. Generalized Cannon’s algorithm for parallel matrix multiplication. In Proceedings of the 11th International Conference on Supercomputing (Vienna, Austria) (ICS ’97) . Association for Computing Machinery, New York, ...

    1997

  18. [26]

    Christian Lessig, Ilaria Luise, Bing Gong, Michael Langguth, Scarlet Stadtler, and Martin Schultz. 2023. AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learning. arXiv:2308.13280 [physics.ao-ph] https://arxiv.org/abs/2308.13280

    2023 arXiv

  19. [27]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE Press, virtual, 9992–10002. doi:...

    2021

  20. [28]

    Gupta, and Aditya Grover

    Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K. Gupta, and Aditya Grover. 2023. ClimaX: a foundation model for weather and climate. In Proceedings of the 40th International Conference on Machine Learning (ICML’23) . JMLR.org, Honolulu, Hawaii, USA, Article 1078, 3...

    2023

  21. [29]

    Tung Nguyen, Rohan Shah, Hritik Bansal, Troy Arcomano, Romit Maulik, Veerab- hadra Kotamarthi, Ian Foster, Sandeep Madireddy, and Aditya Grover. 2024. Scal- ing transformer neural networks for skillful and reliable medium-range weather forecasting. arXiv:2312.03876 [physics.ao...

    2024 arXiv

  22. [30]

    Thomas Nils Nipen, Håvard Homleid Haugen, Magnus Sikora Ingstad, Even Mar- ius Nordhagen, Aram Farhad Shafiq Salihi, Paulina Tedesco, Ivar Ambjørn Seierstad, Jørn Kristiansen, Simon Lang, Mihai Alexe, Jesper Dramsch, Bau- douin Raoult, Gert Mertes, and Matthew Chantry. 2024. R...

    2024 arXiv

  23. [31]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des- maison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, L...

    2019

  24. [32]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory optimizations Toward Training Trillion Parameter Models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Press, virtual, 1–16. doi:10.1...

    2020 arXiv

  25. [33]

    Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russel, Alvaro Sanchez-Gonzalez, Vivian Yang, Rob Carver, Shreya Agrawal, Matthew Chantry, Zied Ben Bouallegue, Peter Dueben, Carla Bromberg, Jared Sisk, Luke Barrington, Aaron Bell, and Fei Sh...

    2024 arXiv

  26. [34]

    Stephan Rasp and Nils Thuerey. 2021. Data-driven medium-range weather prediction with a resnet pretrained on climate simulations: A new model for weatherbench. Journal of Advances in Modeling Earth Systems 13, 2 (2021), e2020MS002405. doi:10.1029/2020MS002405

    2021 doi

  27. [35]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053 [cs.CL] https: //arxiv.org/abs/1909.08053

    2020 arXiv

  28. [36]

    Taylor, Ronald J

    Karl E. Taylor, Ronald J. Stouffer, and Gerald A. Meehl. 2012. An Overview of CMIP5 and the Experiment Design. Bulletin of the American Meteorological Society 93, 4 (2012), 485 – 498. doi:10.1175/BAMS-D-11-00094.1

    2012 doi

  29. [38]

    Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszko- reit, Mario Lucic, and Alexey Dosovitskiy. 2021. MLP-mixer: an all-MLP archi- tecture for vision. In Proceedings of t...

    2021

  30. [39]

    van de Geijn and Jerrell Watts

    Robert A. van de Geijn and Jerrell Watts. 1995.SUMMA: Scalable Universal Matrix Multiplication Algorithm. Technical Report. Computer Science Dept. Taylor Hall 2.124 Austin, TX, USA

    1995

  31. [40]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, ...

    2017

  32. [41]

    Aji, Ming Fan, Wei Zhang, Junqi Yin, Moetasim Ashfaq, Dan Lu, and Prasanna Balaprakash

    Xiao Wang, Siyan Liu, Aristeidis Tsaris, Jong-Youl Choi, Ashwin M. Aji, Ming Fan, Wei Zhang, Junqi Yin, Moetasim Ashfaq, Dan Lu, and Prasanna Balaprakash

  33. [42]

    Hamrud, Christian Kuehnlein, S

    Nils Wedi, Peter Bauer, Willem Deconinck, Michail Diamantakis, M. Hamrud, Christian Kuehnlein, S. Malardel, Kristian Mogensen, G. Mozdzynski, and Piotr Smolarkiewicz. 2015. The modelling infrastructure of the Integrated Forecasting System: Recent advances and future challenges...

    2015 doi

  34. [43]

    Masafumi Yamazaki, Akihiko Kasagi, Akihiro Tabuchi, Takumi Honda, Masahiro Miwa, Naoto Fukumoto, Tsuguchika Tabaru, Atsushi Ike, and Kohta Nakashima

  35. [44]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences...

    2023

  36. [2017]

    SIGARCH Comput

    Understanding and Optimizing Asynchronous Low-Precision Stochastic Gradient Descent. SIGARCH Comput. Archit. News 45, 2 (6 2017), 561––574. doi:10.1145/3140659.3080248

    2017

  37. [2019]

    arXiv:1903.12650 [cs.LG] https://arxiv.org/abs/1903.12650

    Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds. arXiv:1903.12650 [cs.LG] https://arxiv.org/abs/1903.12650

    1903 arXiv

  38. [2020]

    Umweltbundesamt

    Technical Report. Umweltbundesamt. https://www.umweltbundesamt.de/ publikationen/entwicklung-der-spezifischen-kohlendioxid-7 [in German]

  39. [2023]

    Nature Machine Intelligence 5, 11 (2023), 1176–1178

    Reporting electricity consumption is essential for sustainable AI. Nature Machine Intelligence 5, 11 (2023), 1176–1178. doi:10.1038/s42256-023-00750-1

    2023 doi

  40. [2024]

    In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC ’24)

    ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC ’24) . IEEE Press, Atlanta, GA, USA, Article 1, 11 pages. doi:10.1109/SC41406.2024.00007

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.