Lightweight Metadata-Aware Mixture-of-Experts Masked Autoencoder for Earth Observation

Mohanad Albughdadi

REVIEW 5 major objections 5 minor 1 cited by

Lightweight Metadata-Aware Mixture-of-Experts Masked Autoencoder for Earth Observation

T0 review · 5 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 2.5M-parameter metadata-aware MoE-MAE competes with EO foundation models hundreds of times larger.

desk verdict A plausible compact EO model with a real result buried under an untested attribution claim; the paper deserves review but needs controlled ablations. read the letter →

arxiv 2509.10919 v1 pith:OWLNSEGO submitted 2025-09-13 cs.CV cs.LG

Mohanad Albughdadi This is my paper · ORCID

classification cs.CVcs.LG

keywords EarthObservationMaskedAutoencoderMixture-of-ExpertsMetadataconditioningGeo-temporalencodingSelf-supervisedpretrainingLinearprobingLandsat

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a very small Earth-observation model—about 2.5 million parameters, roughly two orders of magnitude smaller than recent EO foundation models—can learn transferable image representations when pretrained as a masked autoencoder with mixture-of-experts layers and geo-temporal metadata conditioning. The author evaluates frozen encoder embeddings with linear probes: on BigEarthNet-Landsat the model reaches 0.767 micro mAP, and on EuroSAT-Landsat 84.2% accuracy, performance that is competitive with models that have hundreds of millions of parameters. The reason to care is practical: if true, high-quality pretrained EO representations do not require massive compute, lowering the barrier for resource-constrained users. The claim depends on the metadata tokens and sparse routing carrying the performance, which is not isolated by a same-size ablated control.

What carries the argument

The central mechanism is the metadata token: latitude, longitude, week-of-year, and hour-of-day are each encoded as (sin,cos) pairs, projected to the embedding dimension, and concatenated with class and patch tokens so the encoder can exploit spatio-temporal regularities. On top of that, every transformer block replaces its feed-forward network with a mixture-of-experts module—SwiGLU experts with shared projection matrices, a NoisyTop-k router, and two coefficient-of-variation balancing penalties—so each token activates only a small subset of expert weights. The combination lets a small parameter budget act like a larger network while keeping active compute low.

What would settle it

Train an identical encoder with the same parameter count and pretraining data but remove the four metadata tokens (or replace MoE with a dense feed-forward of the same budget) and compare linear-probe mAP on BigEarthNet-LS. If the score does not drop materially, the 'metadata-aware' component is not carrying the claimed benefit.

Watch

Extended reading notes

Core claim

The core discovery is that a 2.5M-parameter masked autoencoder, whose encoder is about 2.3M parameters, can produce embeddings that compete with much larger architectures on two Landsat benchmarks. The model fuses four sinusoidal metadata tokens—latitude, longitude, week-of-year, and hour-of-day—with image patch tokens at the encoder input, and replaces dense feed-forward blocks with sparsely routed SwiGLU experts under a NoisyTop-k gate with load-balancing regularization. Pretrained on BigEarthNet-Landsat with 75% masking, the frozen encoder yields linear-probe micro mAP 0.767 on BigEarthNet-LS and 84.2% accuracy on EuroSAT-LS, despite EuroSAT providing no metadata at test time. The paper i

Load-bearing premise

The load-bearing premise is that the gains over external baselines come from the metadata tokens and MoE routing, but no same-size control without metadata (and without MoE) is trained, so the causal attribution to metadata is assumed rather than demonstrated.

Editorial extensions

If this is right

Lightweight, metadata-aware MoE-MAEs are a viable alternative to large EO foundation models for classification and land-cover tasks.
Pretraining compact models on a single dataset like BigEarthNet-Landsat can transfer across datasets even when metadata is absent.
Sparse MoE routing plus geo-temporal conditioning can compensate for small size, so parameter count is not the only driver of transfer quality.
The demonstrated expert specialization suggests interpretable routing—experts stable assigned to vegetation, water/shadow, and textured regions.
Scaling this compact design to larger multi-sensor pretraining corpora is a promising, low-cost path toward general-purpose EO models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not isolate the contribution of metadata: no same-size dense model and no MoE-MAE without metadata tokens are trained, so the headline attribution is an inference from external baselines rather than a controlled result.
EuroSAT-LS has no metadata, yet Section 4.5 never explains how the four pretrained metadata tokens are omitted at test time; a robust design would treat metadata as optional (e.g., dropout or separate masks) and evaluate with and without.
If metadata conditioning is the causal driver, the model should show degraded transfer when evaluated on tiles outside the lat/lon distribution of pretraining; that is a testable prediction beyond the paper's current benchmarks.
The reported numbers are 'competitive' against selective baselines; against the strongest listed models (e.g., Prithvi-EO-2.0 600M at 91.9% EuroSAT OA) the gap is sizable, so the practical claim is about efficiency per parameter, not parity with the largest models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a legitimate new combination — a 2.5M-parameter MoE-MAE with latitude/longitude and time tokens pretrained on BigEarthNet-Landsat — and it ships code and weights. The transfer numbers (BEN mAP 0.767, EuroSAT 84.2) are competitive with much larger models, and the expert specialization maps in Section 4.7 are a nice qualitative check. If you work on efficient EO representation learning, this is worth reading.

The problem is that the headline attribution outruns the experiments. The abstract says 'metadata-aware pretraining improves transfer and label efficiency,' but no experiment isolates either component. There is no no-metadata pretraining control, no same-size dense model, and no few-shot or reduced-label probe. The comparisons in Table 7 are against external baselines that differ in architecture, parameter count, pretraining data, and objective; they tell you that a compact MoE-MAE can reach parity with larger models, not why. The 'why' is the entire selling point.

There are also a few smaller issues. Section 4.5 calls 84.2% the 'CLS-token accuracy' when Table 6 and Table 7 both show 84.2% for all tokens and 78.4% for CLS. That's a clear mislabel. And the EuroSAT evaluation uses no metadata at test time, but the paper never says how the four metadata tokens in Eq. (1) are omitted or replaced. If the encoder simply tolerates their absence, that's useful information; if they're zeroed, that should be stated. Finally, there are no error bars or repeated seeds, which is a minor issue given the small margins.

None of this kills the central finding. A 2.3M-parameter encoder that reaches near-foundation-model accuracy on two Landsat benchmarks is a real result, even if the mechanism is not yet explained. The authors built on mLiT/mmLiT and added metadata conditioning; that's modest but honest novelty, and the code release makes it easy to verify.

I'd send this to peer review, but with a strong request: run a no-metadata variant and a same-size dense model, add a few-shot curve or remove the label-efficiency claim, fix the EuroSAT mislabel, and explain how metadata tokens are handled at test time. A reviewer who doesn't insist on those controls isn't doing their job. For a reading group, it's a good case study in how missing ablations can undermine an otherwise plausible system paper.

Referee Report

5 major / 5 minor

Summary. The paper proposes a compact metadata-aware Mixture-of-Experts Masked Autoencoder (MoE-MAE) for Earth observation. The encoder (~2.3M parameters) prepends four sinusoidal metadata tokens (week, hour, latitude, longitude) to patch/class tokens and replaces feed-forward blocks with NoisyTop-k MoE layers. Pretraining on BigEarthNet-Landsat uses masked reconstruction, an auxiliary reconstruction term on visible patches, and MoE load-balancing losses. Frozen-encoder linear probes are evaluated on BigEarthNet-Landsat multi-label and EuroSAT-Landsat classification, reporting BEN mAP micro 0.767 and EuroSAT OA 84.2% with all tokens, competitive with much larger baselines. The authors conclude that metadata-aware pretraining and MoE routing improve transfer and label efficiency.

Significance. If the attribution were established, this would be a useful efficiency result: a 2.5M-parameter model competing with 100M+ parameter EO foundation models would lower the compute barrier for representation learning. The paper releases code and weights and largely follows the LandsatBench protocol, which supports reproducibility. However, the reported comparisons do not currently isolate the claimed causes; the central attribution and the label-efficiency statement are not supported by controlled experiments. The contribution is therefore promising but requires additional experiments to be convincing.

major comments (5)

[§4.4, Table 7] The headline claim that 'metadata-aware pretraining improves transfer and label efficiency' is not tested. Table 7 compares against external baselines that differ simultaneously in architecture, parameter count, pretraining corpus, and objective. The only in-model ablation (§4.7) measures expert specialization, not downstream performance. To support the attribution, the paper should pretrain at least (i) a dense same-size transformer with identical data and objective, (ii) a MoE-MAE with metadata tokens removed, and (iii) a MoE-MAE with metadata tokens but dense feed-forward networks, and report the same linear-probe metrics. Without these, the parity with larger models could be due to the MAE objective or the small architecture rather than metadata conditioning or MoE routing.
[Abstract; §4.4, Table 4] The abstract states that the model demonstrates 'label efficiency,' but no label-efficiency experiment is reported. All linear probes in §4.4–4.5 are trained on the full training splits described in Table 4; there is no few-shot or reduced-label curve. A label-efficiency claim requires at least accuracy/mAP as a function of training-set fraction. Add such curves or remove the claim.
[§4.5, Table 6] The EuroSAT result is misreported. Section 4.5 first says 'all-token embeddings achieve 84.2% accuracy' and then says 'Our CLS-token accuracy of 84.2% outperforms many larger models,' but Table 6 reports CLS OA = 78.4 and All-tokens OA = 84.2. This conflation affects the comparison narrative and must be corrected; the text and Table 7 should consistently distinguish CLS-token and all-token results.
[§4.5, Eq. (1)] The EuroSAT-LS evaluation says metadata is 'absent at test time' and 'only imagery tokens are available,' but the architecture in Eq. (1) always prepends four metadata tokens. The paper never states whether these tokens are dropped, zero-padded, replaced with a learned placeholder, or handled by modifying positional embeddings. This is essential for reproducing the transfer result and for interpreting whether the metadata pathway is active. Specify the exact inference-time construction and, ideally, ablate it.
[Tables 5–7] All results appear to come from a single run, with no error bars or repeated seeds. The key comparisons involve small differences (e.g., BEN mAP 0.767 vs. 0.761 in Table 7), and without variance estimates the competitive claims are not quantitatively grounded. At minimum, repeat the linear probes over several seeds and report mean ± std; ideally also pretrain with a second seed.

minor comments (5)

[Abstract; §3.1; §5] The paper alternates between 'with only 2.5M parameters' and 'encoder with only ~2.3M parameters.' Clarify which number refers to the full model versus the encoder at each occurrence.
[§4.5] Typo: 'acqusition' should be 'acquisition.'
[Eq. (9)] The equation formatting is broken in the text; also define p_t explicitly and use consistent spacing.
[§4.5, Figure 4] The text says CLS-token geometry is 'consistent with its stronger cross-dataset accuracy,' but Table 6 shows CLS accuracy (78.4) is lower than all-token accuracy (84.2). Rephrase to avoid contradiction.
[§4.7] The analysis is described as a 'quantitative ablation study,' but the only quantitative performance metric is sparsity; the expert-specialization analysis is qualitative. Consider renaming to 'MoE analysis' or adding a quantitative routing measure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's results are empirical benchmark measurements, not quantities forced by construction or by self-citation.

full rationale

The paper contains no derivation chain whose output equals its input by construction. The pretraining objective in Eq. (8) is a masked-reconstruction loss supplemented by unmasked reconstruction and MoE load-balancing penalties; it does not define any of the reported metrics (mAP, accuracy, F1), and none of the headline numbers is a fitted parameter renamed as a prediction. Linear probes are trained on held-out training splits and evaluated on test splits, which is a standard external evaluation protocol. The paper cites prior work for architectural components (mLiT/mmLiT [9] for staged expert configuration and loss weights, NoisyTop-k routing [14], etc.), but these are external and independent choices, not self-citations that carry the paper's claims. The main weakness is the absence of a controlled ablation that isolates metadata tokens or MoE routing from the many other differences with external baselines, and the unexplained handling of metadata tokens when evaluating EuroSAT-LS, which lacks metadata. However, missing controls and underdetermined attribute are methodological/correctness concerns, not circularity: the results do not reduce by definition to the inputs. The Section 4.5 statement 'Our CLS-token accuracy of 84.2%' contradicts Table 6, which reports 84.2% for all tokens and 78.4% for the CLS token; this is a reporting inconsistency that does not affect the circularity assessment.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical ML study; the ledger contains only standard training choices and domain assumptions. No new physical entities or ad hoc mathematical devices are introduced. The main unaccounted element is the assumed benefit of metadata tokens, which is not isolated experimentally.

free parameters (3)

masking_ratio = 0.75
Chosen by hand following MAE convention (Section 4.2), not fitted to the reported benchmarks.
loss_weight_alpha = 0.1
Fixed as in mLiT [9]; not fitted to the data.
loss_weight_beta = 0.5
Fixed as in mLiT [9]; not fitted to the data.

assumptions (3)

domain assumption BigEarthNet-Landsat training split is a sufficient source for pretraining transferable EO representations.
The entire pretraining and evaluation rely on BEN-LS as the only pretraining corpus (Section 4.1).
domain assumption Linear probing of frozen embeddings is an adequate measure of representation quality.
All conclusions about transfer are based on logistic-regression probes (Section 4.2).
standard math The masked autoencoder reconstruction objective, Eq. 8, drives the encoder to learn useful semantics.
Standard practice in MAE literature, adopted rather than derived here (Section 3.3).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Lightweight Metadata-Aware Mixture-of-Experts Masked Autoencoder for Earth Observation." pith.science (2026). https://pith.science/paper/OWLNSEGO

@misc{pith2026250910919,
  author       = {Pith},
  title        = {Pith review of: Lightweight Metadata-Aware Mixture-of-Experts Masked Autoencoder for Earth Observation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWLNSEGO}},
  note         = {Machine review of arXiv:2509.10919}
}

read the original abstract

Recent advances in Earth Observation have focused on large-scale foundation models. However, these models are computationally expensive, limiting their accessibility and reuse for downstream tasks. In this work, we investigate compact architectures as a practical pathway toward smaller general-purpose EO models. We propose a Metadata-aware Mixture-of-Experts Masked Autoencoder (MoE-MAE) with only 2.5M parameters. The model combines sparse expert routing with geo-temporal conditioning, incorporating imagery alongside latitude/longitude and seasonal/daily cyclic encodings. We pretrain the MoE-MAE on the BigEarthNet-Landsat dataset and evaluate embeddings from its frozen encoder using linear probes. Despite its small size, the model competes with much larger architectures, demonstrating that metadata-aware pretraining improves transfer and label efficiency. To further assess generalization, we evaluate on the EuroSAT-Landsat dataset, which lacks explicit metadata, and still observe competitive performance compared to models with hundreds of millions of parameters. These results suggest that compact, metadata-aware MoE-MAEs are an efficient and scalable step toward future EO foundation models.

Figures

Figures reproduced from arXiv: 2509.10919 by the authors.

**Figure 2.** Pretraining dynamics of MoE-MAE on BEN-LS. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Qualitative reconstruction examples on BEN-LS. Each row shows a different sample: original input (left), [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: t-SNE (2D) of EuroSAT-LS embeddings extracted from the frozen encoder. Colors denote class labels. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Visualization of expert specialization at the first MoE layer across five examples. Each row corresponds to [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

From Surface Forecasting to Observability Forecasting: A Latent World Model for Cloud-Aware EO Monitoring
cs.CV 2026-07 accept novelty 6.0 of 10

A JEPA world model forecasts cloud-induced observation usability and recovery timing on EarthNet2021, outperforming persistence and competing with LightGBM on most splits.

Reference graph

Works this paper leans on

20 extracted references · cited by 1 Pith paper

[1]

Lobell, and Stefano Ermon

Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, and Stefano Ermon. SatMAE: Pre-training transformers for temporal and multi-spectral satellite imagery, 2023

2023
[2]

Johannes Jakubik, Sujit Roy, C. E. Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniela Szwarcman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, Daiki Kimura, Naomi Simumba, Linsong Chu, S. Karthik Mukkavilli, Devyani Lambhate, Kamal Das, Ranjini Bangalore, Dario Oliveira, Michal Muszynski, Kumar Ankur, Muthukumaran Ramasubramanian, Iksha Gurun...

2023
[3]

TerraMind: Large-scale generative multimodality for Earth Observation, 2025

Johannes Jakubik, Felix Yang, Benedikt Blumenstiel, Erik Scheurer, Rocco Sedona, Stefano Maurogiovanni, Jente Bosmans, Nikolaos Dionelis, Valerio Marsocci, Niklas Kopp, Rahul Ramachandran, Paolo Fraccaro, Thomas Brunschwiler, Gabriele Cavallaro, Juan Bernabe-Moreno, and Nicolas Longépé. TerraMind: Large-scale generative multimodality for Earth Observation, 2025

2025
[4]

Clay foundation model

Clay Foundation. Clay foundation model. https://clay-foundation.github.io/model/index.html. Accessed: 2025-09-01

2025
[5]

Albrecht, and Xiao Xiang Zhu

Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M. Albrecht, and Xiao Xiang Zhu. SSL4EO-S12: A large-scale multi-modal, multi-temporal dataset for self-supervised learning in Earth Observation, 2023

2023
[6]

Stewart, Nils Lehmann, Isaac A

Adam J. Stewart, Nils Lehmann, Isaac A. Corley, Yi Wang, Yi-Chia Chang, Nassim Ait Ali Braham, Shradha Sehgal, Caleb Robinson, and Arindam Banerjee. SSL4EO-L: Datasets and foundation models for Landsat imagery, 2023

2023
[7]

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

2022
[8]

Scaling vision with sparse Mixture of Experts, 2021

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse Mixture of Experts, 2021

2021

Show all 20 references

[9]

How lightweight can a Vision Transformer be, 2024

Jen Hong Tan. How lightweight can a Vision Transformer be, 2024

2024
[10]

Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell

Colorado J. Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. Scale-MAE: A scale-aware masked autoencoder for multiscale geospatial representation learning, 2023

2023
[11]

Senpa-MAE: Sensor parameter aware masked autoencoder for multi-satellite self-supervised pretraining, 2024

Jonathan Prexl and Michael Schmitt. Senpa-MAE: Sensor parameter aware masked autoencoder for multi-satellite self-supervised pretraining, 2024

2024
[12]

Masked autoencoders are scalable vision learners, 2021

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. 11 APREPRINT- SEPTEMBER16, 2025

2021
[13]

GLU variants improve transformer, 2020

Noam Shazeer. GLU variants improve transformer, 2020

2020
[14]

Outrageously large neural networks: The sparsely-gated Mixture-of-Experts layer, 2017

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated Mixture-of-Experts layer, 2017

2017
[15]

Landsat-Bench: Datasets and benchmarks for Landsat foundation models, 2025

Isaac Corley, Lakshay Sharma, and Ruth Crasto. Landsat-Bench: Datasets and benchmarks for Landsat foundation models, 2025

2025
[16]

BigEarthNet: A large-scale benchmark archive for remote sensing image understanding

Gencer Sumbul, Marcela Charfuelan, Begum Demir, and V olker Markl. BigEarthNet: A large-scale benchmark archive for remote sensing image understanding. InIGARSS 2019 – 2019 IEEE International Geoscience and Remote Sensing Symposium. IEEE, July 2019

2019
[17]

EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification, 2019

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification, 2019

2019
[18]

Stewart, Joëlle Hanna, Damian Borth, Ioannis Papoutsis, Bertrand Le Saux, Gustau Camps-Valls, and Xiao Xiang Zhu

Zhitong Xiong, Yi Wang, Fahong Zhang, Adam J. Stewart, Joëlle Hanna, Damian Borth, Ioannis Papoutsis, Bertrand Le Saux, Gustau Camps-Valls, and Xiao Xiang Zhu. Neural plasticity-inspired multimodal foundation model for Earth Observation, 2024

2024
[19]

SatlasPretrain: A large-scale dataset for remote sensing image understanding, 2023

Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi. SatlasPretrain: A large-scale dataset for remote sensing image understanding, 2023

2023
[20]

Prithvi-EO-2.0: A versatile multi-temporal foundation model for Earth Observation applications

Daniela Szwarcman, Sujit Roy, Paolo Fraccaro, Þorsteinn Elí Gíslason, Benedikt Blumenstiel, Rinki Ghosal, Pedro Henrique de Oliveira, Joao Lucas de Sousa Almeida, Rocco Sedona, Yanghui Kang, Srija Chakraborty, Sizhe Wang, Carlos Gomes, Ankur Kumar, Myscon Truong, Denys Godwin,...

2025

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Lobell, and Stefano Ermon

Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, and Stefano Ermon. SatMAE: Pre-training transformers for temporal and multi-spectral satellite imagery, 2023

2023

[2] [2]

Johannes Jakubik, Sujit Roy, C. E. Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniela Szwarcman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, Daiki Kimura, Naomi Simumba, Linsong Chu, S. Karthik Mukkavilli, Devyani Lambhate, Kamal Das, Ranjini Bangalore, Dario Oliveira, Michal Muszynski, Kumar Ankur, Muthukumaran Ramasubramanian, Iksha Gurun...

2023

[3] [3]

TerraMind: Large-scale generative multimodality for Earth Observation, 2025

Johannes Jakubik, Felix Yang, Benedikt Blumenstiel, Erik Scheurer, Rocco Sedona, Stefano Maurogiovanni, Jente Bosmans, Nikolaos Dionelis, Valerio Marsocci, Niklas Kopp, Rahul Ramachandran, Paolo Fraccaro, Thomas Brunschwiler, Gabriele Cavallaro, Juan Bernabe-Moreno, and Nicolas Longépé. TerraMind: Large-scale generative multimodality for Earth Observation, 2025

2025

[4] [4]

Clay foundation model

Clay Foundation. Clay foundation model. https://clay-foundation.github.io/model/index.html. Accessed: 2025-09-01

2025

[5] [5]

Albrecht, and Xiao Xiang Zhu

Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M. Albrecht, and Xiao Xiang Zhu. SSL4EO-S12: A large-scale multi-modal, multi-temporal dataset for self-supervised learning in Earth Observation, 2023

2023

[6] [6]

Stewart, Nils Lehmann, Isaac A

Adam J. Stewart, Nils Lehmann, Isaac A. Corley, Yi Wang, Yi-Chia Chang, Nassim Ait Ali Braham, Shradha Sehgal, Caleb Robinson, and Arindam Banerjee. SSL4EO-L: Datasets and foundation models for Landsat imagery, 2023

2023

[7] [7]

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

2022

[8] [8]

Scaling vision with sparse Mixture of Experts, 2021

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse Mixture of Experts, 2021

2021