Pith. sign in

REVIEW 3 major objections 5 minor 15 references

This paper establishes that the internal representation spaces of Earth observation foundation models are organized more by sensor architecture than by scene content: a simple classifier can identify the source satellite from a frozen embed

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

EOFM embeddings are strongly partitioned by sensor architecture, so matching spectral bands is not enough to make cross-sensor embedding search reliable.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful, novel empirical result with a real temporal-alignment confound; worth a serious referee. the 3 major comments →

arxiv 2510.03316 v1 pith:AUXAVHCC submitted 2025-10-01 cs.CV cs.AIcs.LG

The View From Space: Navigating Instrumentation Differences with EOFMs

classification cs.CV cs.AIcs.LG
keywords Earth observation foundation modelssensor architectureembedding spacemodality classificationsimilarity searchremote sensingPrithviDOFA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests a question the Earth-observation machine-learning community has not directly asked: when a foundation model converts a satellite image into an embedding, does the vector encode what the image shows or which instrument captured it? Using paired images of the same locations in Indiana taken by multiple satellite sensors, the authors show that sensor identity is recoverable from frozen embeddings with 88–91% accuracy using simple classifiers, and that the nearest neighbors an embedding retrieves are usually from the same sensor. They conclude that current EOFMs are highly sensitive to sensor architecture, that matching spectral bands by wavelength is not enough to align modalities, and that users should match their data collection regime to a model's pretraining data. This matters because embeddings from these models are increasingly used for similarity search and few-shot learning, where sensor-identity dominance would silently corrupt results.

Core claim

The central claim is that the representation space of EOFMs is highly sensitive to sensor architecture. The authors demonstrate this with two prominent models—Prithvi, pretrained solely on HLS six-band data, and DOFA, which accepts any band configuration via a wavelength-based dynamic embedding layer and was trained on multiple modalities. With frozen weights, both models produce embeddings of the same geographic patches seen through four optical sensors (HLS, Landsat-8, Landsat-9, Sentinel-2). t-SNE plots show clear modality-based clusters; local neighborhood agreement between any two sensors averages below 30% for cls tokens; a Random Forest classifier predicts the source sensor with 90.7%

What carries the argument

The load-bearing object is a purpose-built paired-sample dataset: 600 randomly sampled locations in Indiana, each rendered as 224×224 pixel two-month mosaic composites at 30-meter resolution from four optical satellite data streams (HLS, Landsat-8, Landsat-9, Sentinel-2) plus a Sentinel-1 SAR stream, with clouds masked using each sensor's QA layer and a 20% occlusion threshold. Because the same ground locations are observed by different instruments, any systematic separation in the embeddings can be attributed to the instruments themselves—provided the temporal composites are well aligned. This dataset is fed through the frozen encoders of Prithvi and DOFA, and the resulting 768-dimensional

Load-bearing premise

The claim rests on the assumption that the two-month composite images of the same location are temporally aligned well enough that the only systematic difference among modalities is the sensor itself; if acquisition time, atmospheric conditions, or crop stage vary systematically by sensor, those differences could explain the embedding separation.

What would settle it

Run the same random-forest sensor-classification test on patches of a single sensor (e.g., Landsat-8) partitioned into two sets by acquisition date, or compare accuracies from same-day versus two-month-mosaic pairs. If classifier accuracy remains near 90% for date-split single-sensor data, the reported sensor-identity signal could be a temporal artifact; if it drops sharply with tighter temporal alignment, the sensor-architecture interpretation is supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Similarity search and retrieval across sensors will be systematically biased: an embedding from one sensor will mostly return neighbors from the same sensor, not semantically similar scenes from other sensors.
  • The common practice of matching spectral bands by wavelength to apply a single-modality model to other sensors is insufficient; the observed embedding shifts arise from subtler distributional differences in sensor architectures.
  • Users of EOFMs should align their input data's acquisition and calibration regime with the pretraining data of the chosen model to avoid silent domain gaps.
  • The high classification accuracy provides a diagnostic: sensor identity is a recoverable attribute of frozen embeddings, which could be used for sensor attribution or as a metric for model bias.
  • The DOFA architecture, with its wavelength-based dynamic input layer and multimodal pretraining, shows slightly lower sensor sensitivity than Prithvi, suggesting architecture and training diversity can mitigate but not eliminate the effect.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's causal claim would be to repeat the sensor-classification experiment on data from the same sensor split by acquisition date; if temporally separated mosaics of the same sensor can be classified at comparable accuracy, then part of the reported 88–91% may be attributed to time/atmosphere rather than instrument architecture.
  • Future EOFM development could explicitly optimize for sensor-invariance—for example, by augmenting pretraining with cross-sensor paired samples—and the 88–91% classification metric could serve as a benchmark for measuring progress.
  • The paired-sample setup itself is a promising evaluation template: any new foundation model could be probed with the same 600-location multi-sensor dataset to quantify its sensor entanglement before deployment.
  • If the sensor-identity dominance persists even under perfect temporal alignment, it would imply that instrument-specific noise patterns and calibration differences are large enough that cross-sensor few-shot learning will require explicit domain adaptation, not just spectral band matching.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether Earth Observation Foundation Model (EOFM) embeddings are organized by sensor identity rather than by semantic content. The authors build a paired dataset of 600 random points in Indiana, each represented as 224x224 30-meter 2-month mosaic composites from HLS, Landsat-8, Landsat-9, Sentinel-2, and Sentinel-1. They freeze the encoders of two EOFMs (Prithvi and DOFA), extract embeddings, and analyze them with t-SNE visualization, local-neighborhood agreement, semantic-neighbor agreement using USDA Crop Data Layer labels, and simple classifiers that predict the input modality. They report strong modality-based clustering in t-SNE, low cross-modal neighborhood overlap (Table 1), moderate shifts in semantic neighbor agreement (Table 2), and 88–91% accuracy in classifying the source modality from frozen embeddings (Table 3). The paper concludes that EOFM representation spaces are highly sensitive to sensor architecture and that this must be considered when using embeddings for retrieval and downstream tasks.

Significance. If established, the claim would be practically important for the growing use of EOFM embeddings in similarity search, few-shot classification, and cross-sensor transfer, and it would motivate more explicit treatment of sensor modality in evaluation benchmarks. The paper has notable strengths: it uses frozen pretrained models, external reference labels (CDL), a held-out test set for the classifier experiment, and it promises to release code and data. The core direction is interesting and the evidence does show that embeddings are separable by source in the evaluated setting. However, the central claim that sensor architecture is the organizing principle is not yet isolated from acquisition-time effects and from the trivial optical-versus-SAR distinction. These gaps are addressable within the manuscript's scope, so the result is not overturned but the conclusion needs to be either strengthened with controls or narrowed.

major comments (3)
  1. [§2, Dataset construction] The acquisition-time confound is not controlled. The text says 'each modality will have a different acquisition period and therefore some work will need to be done to align these as best as possible,' but the actual procedure describes only 2-month mosaic composites with a 20% cloud threshold and gives no description of the alignment performed. Since each sensor can select different clear-sky dates within the mosaic window, systematic differences in phenology, atmosphere, and solar geometry can separate embeddings. The t-SNE clusters, the low neighborhood overlap in Table 1, and the high modality-classification accuracy in Table 3 are all consistent with a timing confound. A same-date or same-acquisition-window control, or at least a report of the per-sensor acquisition-date distributions, is needed to support the claim that sensor architecture is the organizing axis.
  2. [§4, Table 3] The modality-classification experiment does not separate the trivial optical-versus-SAR distinction from sensor architecture among optical instruments. The dataset includes Sentinel-1 (VV/VH), but Table 3 reports only aggregate accuracy, so it is unclear whether the 88–91% figures include Sentinel-1. If they do, much of the separability is expected from radar versus optical input statistics, not from architecture differences among HLS, Landsat, and Sentinel-2. Report per-class accuracies, rerun the optical-only analysis, and provide confidence intervals for the 150-point held-out test set.
  3. [§4, Tables 1–3 and §1] The strength of the claim is out of proportion to the sample. Six hundred points from a single US state, with an unnamed two-month mosaic window and no multi-season or multi-region evaluation, support the statement that these embeddings are separable by source under these conditions, but not the broad claim that 'the representation space of EOFMs is highly sensitive to sensor architecture.' Please either restrict the conclusion to the evaluated setting or add multi-season and multi-region evidence.
minor comments (5)
  1. [§1] Typo: 'ofthen times' should be 'often times' or 'often'. Also 'dataset to dataset' in §2 appears to be a duplicated phrase.
  2. [§4, Table 3] The caption and column labels should specify whether Sentinel-1 is included and define 'patch embed'. The Random and Difference rows in Table 2 are also unclear and should be labeled more explicitly.
  3. [§6, Data Availability] The data availability statement says the dataset can be accessed at 'the following GitHub repository,' but no URL is provided. Please include the actual link.
  4. [§3.1, Figures 1 and 2] The t-SNE plots need legends or color keys, axis labels, and a statement of the number of points plotted. Without these, the visual clustering claim is hard to assess from the figures.
  5. [References] The reference formatting is inconsistent (e.g., 'u. a.' instead of 'et al.'), and some entries lack complete metadata. Please standardize.

Circularity Check

0 steps flagged

No circularity: empirical comparison of frozen embeddings from external EOFMs.

full rationale

The paper's claim chain is an empirical measurement, not a derivation. It constructs paired mosaic composites, extracts frozen embeddings from external pretrained models (Prithvi and DOFA), and then measures t-SNE cluster separation, local-neighborhood overlap, and held-out modality-classification accuracy. Table 3's RandomForest/KNN accuracies are predictive evaluations on held-out patches; the modality labels are the target being predicted, not a quantity used to fit the embeddings or a parameter of the models. The models themselves are pretrained externally (Prithvi, DOFA/PANGAEA), and the Crop Data Layer labels are an external reference, so the central observations are not defined in terms of the conclusions. Section 2's statement that 'each modality will have a different acquisition period and therefore some work will need to be done to align these as best as possible' is an acknowledged temporal-alignment caveat; this is a validity/confounding concern about isolating sensor architecture from acquisition time, not a circular step. No load-bearing self-citation or imported uniqueness theorem appears. The finding therefore has independent empirical content and receives a circularity score of 0.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The paper makes no fitted derivation; all quantitative claims come from empirical comparisons. The free parameters are dataset-construction and analysis thresholds chosen by hand: the 2-month mosaic window, the 20% cloud-mask threshold, the 600-point sample size, and the neighborhood K values. These affect the strength and generality of the central claim but are not fitted to the target result. The main axioms are domain assumptions: frozen-encoder embeddings are meaningful, wavelength-based band matching preserves comparability, CDL labels are a valid semantic reference, and t-SNE structure reflects latent-space structure. No new entities are introduced.

free parameters (4)
  • Cloud occlusion threshold = 20%
    Images more than 20% occluded by clouds are masked; chosen by authors, affects paired sample composition and could bias comparisons if cloud patterns correlate with sensor.
  • Mosaic composite window = 2 months
    Acquisition periods differ per modality; 2-month composites chosen to align, but temporal mismatch remains a confound.
  • Random sample size = 600 points
    600 random points from Indiana; sample size affects power and geographic generality.
  • Neighborhood sizes K = 5, 10, 20
    Reported overlap percentages are computed at these three K values; the pattern is consistent but no error bars are provided.
axioms (6)
  • domain assumption Frozen EOFM encoders produce embeddings that faithfully capture image patch and CLS semantics.
    Invoked throughout §3–4 when comparing Prithvi and DOFA outputs without fine-tuning.
  • domain assumption Matching spectral bands by wavelength category and discarding extra bands is a valid way to input different sensors into Prithvi.
    §3: "The six input bands from each data stream can be accepted by matching the spectral bands by wavelength categories and discarding the additional bands."
  • domain assumption USDA Crop Data Layer majority class is a valid semantic reference for 30 m patches.
    §2: "we align the USDA Crop Data Layer as a reference to explore image semantics"; used as ground truth in Table 2 without uncertainty.
  • standard math t-SNE embeddings preserve cluster structure relevant to modality separation.
    §3.1 relies on t-SNE to conclude modality clusters; t-SNE can distort global distances.
  • domain assumption HLS is a harmonized synthesis of Landsat and Sentinel-2, so it can serve as an independent optical modality.
    §1 and dataset construction; based on Claverie et al. (2018), but processing differences may still make it non-neutral.
  • domain assumption Two-month cloud-masked mosaics from different sensors at the same points are aligned enough to treat sensor architecture as the main varying factor.
    §2 acknowledges differing acquisition periods but does not control or quantify the temporal/atmospheric confound.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of The View From Space: Navigating Instrumentation Differences with EOFMs." pith.science (2026). https://pith.science/paper/AUXAVHCC

@misc{pith2026251003316,
  author       = {Pith},
  title        = {Pith review of: The View From Space: Navigating Instrumentation Differences with EOFMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUXAVHCC}},
  note         = {Machine review of arXiv:2510.03316}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Earth Observation Foundation Models (EOFMs) have exploded in prevalence as tools for processing the massive volumes of remotely sensed and other earth observation data, and for delivering impact on the many essential earth monitoring tasks. An emerging trend posits using the outputs of pre-trained models as 'embeddings' which summarize high dimensional data to be used for generic tasks such as similarity search and content-specific queries. However, most EOFM models are trained only on single modalities of data and then applied or benchmarked by matching bands across different modalities. It is not clear from existing work what impact diverse sensor architectures have on the internal representations of the present suite of EOFMs. We show in this work that the representation space of EOFMs is highly sensitive to sensor architecture and that understanding this difference gives a vital perspective on the pitfalls of current EOFM design and signals for how to move forward as model developers, users, and a community guided by robust remote-sensing science.

Figures

Figures reproduced from arXiv: 2510.03316 by Karis Tenneson, Nicholas LaHaye, Ryan P. DeMilt.

Figure 1
Figure 1. Figure 1: Prithvi TSNE Plots optical modality embeddings [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 8 linked inside Pith

  1. [1]

    2024 Blumenstiel , Benedikt ; Moor , Viktoria ; Kienzler , Romeo ; Brunschwiler , Thomas: Multi-Spectral Remote Sensing Image Retrieval Using Geospatial Foundation Models

    Blumenstiel u.\,a. 2024 Blumenstiel , Benedikt ; Moor , Viktoria ; Kienzler , Romeo ; Brunschwiler , Thomas: Multi-Spectral Remote Sensing Image Retrieval Using Geospatial Foundation Models. In: IGARSS 2024 - 2024 IEEE International Geoscience and Remote Sensing Symposium (2024), S. 7286--7291. -- URL https://api.semanticscholar.org/CorpusID:268247915

  2. [2]

    2018 Claverie , Martin ; Ju , Junchang ; Masek , Jeffrey G

    Claverie u.\,a. 2018 Claverie , Martin ; Ju , Junchang ; Masek , Jeffrey G. ; Dungan , Jennifer L. ; Vermote , Eric F. ; Roger , Jean-Claude ; Skakun , Sergii V. ; Justice , Christopher: The Harmonized Landsat and Sentinel-2 surface reflectance data set. In: Remote Sensing of Environment 219 (2018), S. 145--161. -- URL https://www.sciencedirect.com/scienc...

  3. [3]

    https://clay-foundation.github.io/model/index.html

    Clay 2024 Clay , Foundation: Clay Foundation Model. https://clay-foundation.github.io/model/index.html. 2024. -- [Accessed 11-01-2025]

  4. [4]

    Dosovitskiy u.\,a. 2021 Dosovitskiy , Alexey ; Beyer , Lucas ; Kolesnikov , Alexander ; Weissenborn , Dirk ; Zhai , Xiaohua ; Unterthiner , Thomas ; Dehghani , Mostafa ; Minderer , Matthias ; Heigold , Georg ; Gelly , Sylvain ; Uszkoreit , Jakob ; Houlsby , Neil: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. 2021. -- URL http...

  5. [5]

    2023 Duderstadt , Brandon ; Helm , Hayden S

    Duderstadt u.\,a. 2023 Duderstadt , Brandon ; Helm , Hayden S. ; Priebe , Carey E.: Comparing Foundation Models using Data Kernels. In: ArXiv abs/2305.05126 (2023). -- URL https://api.semanticscholar.org/CorpusID:258564434

  6. [6]

    2023 Fuller , Anthony ; Millard , Koreen ; Green , James R.: CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders

    Fuller u.\,a. 2023 Fuller , Anthony ; Millard , Koreen ; Green , James R.: CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders. 2023. -- URL https://arxiv.org/abs/2311.00566

  7. [7]

    2023 Jakubik , Johannes ; Roy , Sujit ; Phillips , C

    Jakubik u.\,a. 2023 Jakubik , Johannes ; Roy , Sujit ; Phillips , C. E. ; Fraccaro , Paolo ; Godwin , Denys ; Zadrozny , Bianca ; Szwarcman , Daniela ; Gomes , Carlos ; Nyirjesy , Gabby ; Edwards , Blair ; Kimura , Daiki ; Simumba , Naomi ; Chu , Linsong ; Mukkavilli , S. K. ; Lambhate , Devyani ; Das , Kamal ; Bangalore , Ranjini ; Oliveira , Dario ; Mus...

  8. [8]

    Jakubik u.\,a. 2025 Jakubik , Johannes ; Yang , Felix ; Blumenstiel , Benedikt ; Scheurer , Erik ; Sedona , Rocco ; Maurogiovanni , Stefano ; Bosmans , Jente ; Dionelis , Nikolaos ; Marsocci , Valerio ; Kopp , Niklas ; Ramachandran , Rahul ; Fraccaro , Paolo ; Brunschwiler , Thomas ; Cavallaro , Gabriele ; Bernabe-Moreno , Juan ; Longépé , Nicolas: TerraM...

  9. [9]

    2023 Lacoste , Alexandre ; Lehmann , Nils ; Rodriguez , Pau ; Sherwin , Evan D

    Lacoste u.\,a. 2023 Lacoste , Alexandre ; Lehmann , Nils ; Rodriguez , Pau ; Sherwin , Evan D. ; Kerner , Hannah ; Lütjens , Björn ; Irvin , Jeremy A. ; Dao , David ; Alemohammad , Hamed ; Drouin , Alexandre ; Gunturkun , Mehmet ; Huang , Gabriel ; Vazquez , David ; Newman , Dava ; Bengio , Yoshua ; Ermon , Stefano ; Zhu , Xiao X.: GEO-Bench: Toward Found...

  10. [10]

    2019 Ma , Lei ; Liu , Yu ; Zhang , Xueliang ; Ye , Yuanxin ; Yin , Gaofei ; Johnson , Brian A.: Deep learning in remote sensing applications: A meta-analysis and review

    Ma u.\,a. 2019 Ma , Lei ; Liu , Yu ; Zhang , Xueliang ; Ye , Yuanxin ; Yin , Gaofei ; Johnson , Brian A.: Deep learning in remote sensing applications: A meta-analysis and review. In: ISPRS Journal of Photogrammetry and Remote Sensing 152 (2019), S. 166--177. -- URL https://www.sciencedirect.com/science/article/pii/S0924271619301108. -- ISSN 0924-2716

  11. [11]

    In: Journal of Machine Learning Research 9 (2008), Nr

    van der Maaten und Hinton 2008 Maaten , Laurens van der ; Hinton , Geoffrey: Visualizing Data using t-SNE. In: Journal of Machine Learning Research 9 (2008), Nr. 86, S. 2579--2605. -- URL http://jmlr.org/papers/v9/vandermaaten08a.html

  12. [12]

    2024 Marsocci , Valerio ; Jia , Yuru ; Bellier , Georges L

    Marsocci u.\,a. 2024 Marsocci , Valerio ; Jia , Yuru ; Bellier , Georges L. ; Kerekes , David ; Zeng , Liang ; Hafner , Sebastian ; Gerard , Sebastian ; Brune , Eric ; Yadav , Ritu ; Shibli , Ali ; Fang , Heng ; Ban , Yifang ; Vergauwen , Maarten ; Audebert , Nicolas ; Nascetti , Andrea: PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation ...

  13. [13]

    2024 Tavares , Tiago F

    Tavares u.\,a. 2024 Tavares , Tiago F. ; Ayres , Fabio ; Smaragdis , Paris: Measuring similarity between embedding spaces using induced neighborhood graphs. 2024. -- URL https://arxiv.org/abs/2411.08687

  14. [14]

    2023 Wang , Yi ; Braham , Nassim Ait A

    Wang u.\,a. 2023 Wang , Yi ; Braham , Nassim Ait A. ; Xiong , Zhitong ; Liu , Chenying ; Albrecht , Conrad M. ; Zhu , Xiao X.: SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation. 2023. -- URL https://arxiv.org/abs/2211.07044

  15. [15]

    2024 Xiong , Zhitong ; Wang , Yi ; Zhang , Fahong ; Stewart , Adam J

    Xiong u.\,a. 2024 Xiong , Zhitong ; Wang , Yi ; Zhang , Fahong ; Stewart , Adam J. ; Hanna , Joëlle ; Borth , Damian ; Papoutsis , Ioannis ; Saux , Bertrand L. ; Camps-Valls , Gustau ; Zhu , Xiao X.: Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation. 2024. -- URL https://arxiv.org/abs/2403.15356

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.