Pith. sign in

REVIEW 2 major objections 5 minor 47 references

SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation

T0 review · 2 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pre-trained vision transformers have redundant deep layers that representation similarity on unlabeled task data can remove before any fine-tuning.

desk verdict Solid pre-adaptation depth selection for EO ViTs: useful engineering with honest limits, not a new theory of pruning. read the letter →

arxiv 2603.19873 v2 pith:WFXECTBN submitted 2026-03-20 cs.CV

Víctor Barreiro , Johannes Jakubik , Francisco Argüello , Dora B. Heras This is my paper · ORCID
classification cs.CV
keywords representationsimilaritymodelcompressionfoundationmodelsefficientfine-tuningEarthObservationvisiontransformersstructuredpruningCKA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning large Earth Observation foundation models is expensive in training time and still leaves the full model depth at inference, which blocks satellite and edge use. The paper argues that deeper layers of pre-trained vision transformers produce nearly identical representations on a task’s own unlabeled samples, so those layers are already redundant for that distribution. SIMPLER measures layer-wise similarity with Centered Kernel Alignment, then automatically chooses a cutoff that keeps diverse early layers and drops stable late ones, without gradients, magnitude rules, or hyperparameter search. On Prithvi-EO-2 it removes up to 79% of parameters while retaining about 94% of baseline performance and speeding both training and inference; the same procedure works on TerraMind and ImageNet-pretrained ViT-MAE across segmentation, multi-label classification, and time-series tasks. The result is a single pre-fine-tuning step that cuts both adaptation cost and deployment cost at once.

What carries the argument

SIMPLER’s cutoff score: partition the CKA similarity matrix at candidate depth c into a top-left (kept) block and bottom-right (pruned) block, compute each block’s mean absolute consecutive-row difference, and choose c that maximizes diversity of the kept block minus stability of the pruned block—no thresholds or tuning.

What would settle it

On a new EO task, fully fine-tune both the CKA-selected shallow model and the full model (and a same-size random or magnitude-chosen depth); if the CKA model loses far more accuracy than the alternatives while deeper, highly similar layers still improve linear probes or fine-tuning, the claim that pre-trained similarity predicts post-adaptation redundancy fails.

Watch

Extended reading notes

Core claim

Layer-wise representation similarity computed on a small set of unlabeled downstream samples from a frozen pre-trained vision transformer predicts which deep layers will be redundant after fine-tuning. An automated score on the similarity matrix therefore selects an effective model depth before any adaptation begins, delivering simultaneous training and inference savings while retaining most of the full model’s task performance.

Load-bearing premise

High similarity between consecutive deep layers on a few hundred unlabeled task images is assumed to mean those layers stay redundant after fine-tuning, and a simple block-difference score is assumed to pick a near-best depth without task-specific tuning.

Editorial extensions

If this is right

  • A foundation model can be thinned for a target EO task before fine-tuning starts, cutting GPU hours and deployment latency together.
  • Depth selection plus LoRA compounds: fewer layers and fewer trainable parameters in one pipeline.
  • One large pre-training plus SIMPLER-style reduction can outperform training several smaller foundation models independently.
  • When progressive representation stabilization is present, the same pre-fine-tuning selection applies to multispectral EO and RGB ViTs.
  • The output is a dense shorter model that runs on ordinary frameworks, without sparse-inference libraries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same pre-fine-tuning similarity signal holds for language or other multimodal transformers, architecture selection could become a default first step before any PEFT method.
  • Models with explicit collapse-prevention (oscillating CKA, no deep stabilization) mark a clear boundary: pre-training recipe, not just architecture, decides whether deep redundancy is exploitable this way.
  • On-board satellite and edge pipelines could treat a SIMPLER-style cutoff as a per-mission capacity dial, trading a few accuracy points for large memory and power savings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes SIMPLER, a pre-fine-tuning depth-selection method for vision-transformer foundation models in Earth Observation. It computes a layer-wise CKA similarity matrix on a small unlabeled task sample set from the frozen pre-trained model, partitions the matrix at candidate cutoffs, and selects the cutoff maximizing score(c)=Δ_TL−Δ_BR (mean absolute consecutive-row difference of the retained vs. pruned CKA blocks). The reduced architecture is then fine-tuned (optionally with LoRA). On Prithvi-EO-2 the method reports up to ~79% parameter reduction while retaining ~94% of baseline mIoU on MADOS, with simultaneous training and inference speedups; similar compression–performance trade-offs are shown on BigEarthNetv2 and Sen4Map, and the approach is validated on TerraMind (L/S/T) and ImageNet-pretrained ViT-MAE/CIFAR-100. Ablations cover similarity metrics, from-scratch capacity, random-noise controls, sample-size stability, and linear probing; Appendix C notes failure on DINOv3-style models without progressive stabilization.

Significance. If the central claim holds, SIMPLER fills a genuine gap: PEFT reduces training cost but not inference depth, while structured pruning is post-hoc and requires full fine-tuning first. A pre-adaptation, gradient-free, hyperparameter-light depth selector that simultaneously cuts training and deployment cost is practically valuable for EO edge/satellite settings. Strengths include multi-task and multi-model evaluation, 5-run statistics, from-scratch and noise ablations, linear-probe corroboration, public code, and an explicit applicability check (CKA heatmap / DINOv3 counter-example). The contribution is primarily empirical and systems-oriented rather than a new theoretical guarantee, but the experimental package is substantial for the EO foundation-model efficiency literature.

major comments (2)
  1. §3.3 and Alg. S1: the automated score score(c)=Δ_TL−Δ_BR is load-bearing for the “no hyperparameter tuning / optimal depth” claim, yet it is not compared to other pre-adaptation selectors that use the same CKA matrix (e.g., first consecutive-CKA threshold, argmax of Δ_TL alone, cumulative-similarity knee, or the linear-probe peak depth, which Appendix E places at Block 6 vs SIMPLER’s Block 5). Tab. 5 only shows CKA vs Jaccard/SVCCA under this fixed score. Without that comparison, “near-optimal automated selection” is established only relative to other metrics and to post-hoc magnitude pruning, not relative to the space of pre-adaptation depth criteria. A short ablation on MADOS (and one other task) would make the claim proportionate.
  2. §3.2–3.3 and Appendix C: the premise that high consecutive-layer CKA on unlabeled task samples implies post-adaptation redundancy is architecture-dependent (DINOv3 fails). The main claims and abstract present the method as generally applicable to “pre-trained vision transformers” / foundation models; the limitation is only in the discussion and appendix. The paper should state the applicability condition (clear block-diagonal deep-layer structure on ~500 task samples) up front and, if feasible, report one additional EO backbone that does or does not exhibit the pattern, so readers can judge scope without reading the supplement.
minor comments (5)
  1. Fig. 1a radar axes and normalization of “efficiency” (1/time scaled to [0,1]) should be stated explicitly in the caption so the plot is self-contained.
  2. Tables 2–4: clarify whether reported training times for post-hoc pruning include the full fine-tuning + prune + retrain pipeline (text says yes; table headers alone do not).
  3. §4.1 / Appendix F: sample size |S|=500 and Jaccard k=20 are free parameters; a one-sentence sensitivity note in the main text (beyond Fig. 3) would help reproducibility claims.
  4. Notation: score(c; S, F_pre) in Eq. (1) vs score(c)=Δ_TL−Δ_BR in §3.3; keep a single definition.
  5. Qualitative MADOS figures (S6–S9) are useful; a brief pointer in the main text to the oil-spill failure mode of magnitude pruning would strengthen the comparison to post-hoc methods.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: pre-adaptation CKA scoring does not embed held-out task metrics; performance claims are independent empirical outcomes.

full rationale

SIMPLER’s load-bearing chain is: (i) extract pre-trained layer representations on unlabeled task samples S; (ii) form similarity matrix Z (CKA by default); (iii) choose cutoff c* = arg max_c (Δ_TL − Δ_BR) from Z alone (Eq. 1, §3.3, Alg. S1); (iv) fine-tune the truncated encoder; (v) report mIoU/mAP/accuracy on held-out labels. Step (v) is not a function of the score in (iii), so the reported retention (e.g., 94% mIoU on MADOS) is not forced by construction. The score uses only consecutive-row differences of CKA blocks and never fits to task labels, magnitude, or gradients. Ablations (from-scratch capacity, random-noise CKA collapse, linear probes peaking near but not identical to the CKA cutoff, DINOv3 failure mode) are independent checks rather than tautologies. Citations for CKA/stabilization (Kornblith, Raghu, etc.) and for the evaluated FMs (Prithvi, TerraMind, ViT-MAE) are external or model sources, not self-authored uniqueness theorems that forbid alternatives. Design choice of the particular score form is a methodological limitation, not circular reduction of prediction to input. The paper is self-contained against external benchmarks; no step reduces Eq. X to Eq. Y by definition or fit-as-prediction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central claim rests on standard ViT/transfer-learning practice plus a few paper-specific design choices: CKA as the default similarity, the Δ_TL−Δ_BR cutoff score, unlabeled sample count, and the hypothesis that deep-layer stabilization on task data marks prunable redundancy. No new physical entities are postulated; the invented pieces are methodological constructs. Free parameters are few and mostly procedural rather than fitted performance targets.

free parameters (4)
  • CKA sample size |S|
    Number of unlabeled task images used to build the similarity matrix; paper selects 500 after sensitivity analysis (Fig. 3). Affects cutoff stability and compute.
  • Jaccard neighbor count k
    Set to 20 for the Jaccard baseline metric; influences alternative-metric cutoffs though CKA is default.
  • Cutoff search bounds {2,...,L-2}
    Hard-coded exclusion of extreme depths so neither a single block nor nearly full depth is considered; shapes feasible architectures.
  • Scoring function form score(c)=Δ_TL−Δ_BR
    Hand-designed combination of retained-layer diversity and pruned-layer stability; not learned from labels but chosen as the selection rule the method depends on.
assumptions (5)
  • domain assumption Deep pre-trained ViT layers often produce highly similar representations on downstream inputs due to early global attention and residual propagation.
    Invoked in §1 and §3.2 with citations to prior ViT representation studies; required for progressive stabilization to exist.
  • ad hoc to paper High inter-layer representation similarity on unlabeled task data indicates redundancy for that task under subsequent gradient fine-tuning.
    Core predictive hypothesis of §3.2; supported empirically by ablations but not proven generally.
  • domain assumption CKA (and related kernel/CCA/neighborhood similarities) are valid measures of representational redundancy for layer selection.
    Standard representational-similarity toolkit (§2–3.2); paper treats CKA as default after comparison.
  • domain assumption Fine-tuning only layers 1..c* with a task head is a valid adaptation of a pre-trained encoder for EO segmentation/classification/time-series tasks.
    Standard transfer-learning practice assumed throughout experiments.
  • standard math Linear algebra / kernel definitions of CKA, SVCCA, and Jaccard as stated in Appendix A.2 are correctly applied.
    Background math for similarity matrices; not re-derived.
invented entities (2)
  • SIMPLER pre-fine-tuning depth-selection pipeline independent evidence
    purpose: Select reduced ViT depth from pre-trained representations before any task fine-tuning to cut train and inference cost together.
    The paper’s named method; evaluated empirically rather than derived from a uniqueness theorem.
  • Automated cutoff score Δ_TL − Δ_BR on CKA blocks independent evidence
    purpose: Choose c* without magnitude thresholds, gradients, or manual compression ratios.
    Paper-specific scoring construct in §3.3 / Algorithm S1; success is judged by downstream task metrics, which provides an external handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation." pith.science (2026). https://pith.science/paper/WFXECTBN

@misc{pith2026260319873,
  author       = {Pith},
  title        = {Pith review of: SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFXECTBN}},
  note         = {Machine review of arXiv:2603.19873}
}
read the original abstract

Fine-tuning foundation models for Earth Observation is computationally expensive, with high training time and memory demands for both training and deployment. Parameter-efficient methods reduce training cost but retain full inference complexity, while post-hoc compression optimizes inference only after costly full fine-tuning. We introduce SIMPLER, a pre-fine-tuning architecture selection method that reduces inference and deployment costs by identifying an effective model depth before adaptation. SIMPLER exploits stabilization of representations in deeper layers of pre-trained vision transformers: it computes layer-wise representation similarity on unlabeled task data and applies an automated scoring function to select redundant layers, with no gradients, magnitude heuristics, or hyperparameter tuning required. On Prithvi-EO-2, SIMPLER prunes up to 79% of parameters while retaining 94% of baseline performance, yielding a 2.1x training speedup and 2.6x inference speedup. The method generalizes to TerraMind (a multimodal EO foundation model) and ImageNet-pretrained ViT-MAE, demonstrating applicability across tasks, architectures, and spectral modalities. Code is available at https://gitlab.citius.gal/hpc4rs/simpler.

Figures

Figures reproduced from arXiv: 2603.19873 by the authors.

Figure 1
Figure 1. Overview of the proposed SIMPLER method. The upper radar plot summa￾rizes the trade-off between training/inference efficiency and mIoU performance on the MADOS dataset. The lower panel illustrates the complete pipeline, from layer repre￾sentation extraction to fine-tuning of the reduced model. applications including disaster response and precision agriculture [22]. These constraints motivate efficiency research: red… view at source ↗
Figure 2
Figure 2. Similarity metrics comparison (CKA, Jaccard, SVCCA) on MADOS dataset (weak semantic segmentation) with Prithvi-EO-2 300M. Higher values (yellow) indi￾cate greater similarity between layer representations, while lower values (blue) indicate greater divergence. optimal cutoff maximizes c ∗ = arg maxc∈{2,...,L−2} (∆T L − ∆BR), requiring no hyperparameter tuning. Procedure: (1) Extract representations {h1, . . . , hL} f… view at source ↗
Figure 3
Figure 3. Sample size sensitivity analysis for CKA computation on MADOS (Prithvi￾300M). Left: Cutoff selection stabilizes at 500 samples (std=0.53, 5.6× reduction from 10 samples) with acceptable computation time (218s vs. 678s for 1000 samples). Right: Memory consumption scales linearly (36.8GB at 500 samples), while CKA variance remains stable. The 500-sample configuration provides optimal balance between cutoff stability, … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: CKA similarity for random noise (Gaussian, Uniform). Uniformly high simi￾larity (0.998-1.000) with 115× narrower range vs. real data confirms learned features, not architectural artifacts [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 4 canonical work pages

  1. [1]

    arXiv preprint arXiv:1610.01644 (2016).https://doi.org/10.48550/ arXiv.1610.01644

    Alain, G., Bengio, Y.: Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 (2016).https://doi.org/10.48550/ arXiv.1610.01644

    arXiv 2016

  2. [2]

    Neural Networks153, 461–473 (2022)

    Bakhtiarnia, A., Zhang, Q., Iosifidis, A.: Single-layer vision transformers for more accurate early exits with less overhead. Neural Networks153, 461–473 (2022). https://doi.org/10.1016/j.neunet.2022.06.038

    work page doi:10.1016/j.neunet.2022.06.038 2022

  3. [3]

    arXiv preprint arXiv:2108.07258 (2021).https://doi.org/10.48550/arXiv.2108.07258

    Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).https://doi.org/10.48550/arXiv.2108.07258

    arXiv doi:10.48550/arxiv.2108.07258 2021

  4. [4]

    In: IEEE/CVF International Conference on Computer Vision (ICCV)

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9650–9660 (2021). https://doi.org/10.1109/ICCV48922.2021.00951

    doi:10.1109/iccv48922.2021.00951 2021

  5. [5]

    In: Advances in Neu- ral Information Processing Systems (NeurIPS) (2022),https://arxiv.org/abs/ 2205.13535

    Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J., Luo, P.: AdaptFormer: Adapting vision transformers for scalable visual recognition. In: Advances in Neu- ral Information Processing Systems (NeurIPS) (2022),https://arxiv.org/abs/ 2205.13535

    arXiv 2022

  6. [6]

    In: IEEE Inter- national Geoscience and Remote Sensing Symposium (IGARSS) (2025).https: //doi.org/10.5281/zenodo.10891137

    Clasen, K.N., Hackel, L., Burgert, T., Sumbul, G., Demir, B., Markl, V.: reBEN: Refined BigEarthNet dataset for remote sensing image analysis. In: IEEE Inter- national Geoscience and Remote Sensing Symposium (IGARSS) (2025).https: //doi.org/10.5281/zenodo.10891137

    work page doi:10.5281/zenodo.10891137 2025

  7. [7]

    In: Proceedings of the 36th International Conference on Neural Information Processing Systems

    Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D.B.,Ermon,S.:Satmae:pre-trainingtransformersfortemporalandmulti-spectral satellite imagery. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS ’22 (2022)

    2022

  8. [8]

    In: International Conference on Learning Representations (ICLR) (2021),https: //openreview.net/forum?id=YicbFdNTTy

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021),https: //openreview.net/forum?id=YicbFdNTTy

    2021

Show all 47 references
  1. [9]

    In: International Conference on Learning Representations (2020), https://openreview.net/forum?id=SylO2yStDr

    Fan, A., Grave, E., Joulin, A.: Reducing transformer depth on demand with struc- tured dropout. In: International Conference on Learning Representations (2020), https://openreview.net/forum?id=SylO2yStDr

    2020

  2. [10]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR)

    Fang, G., Ma, X., Song, M., Mi, M.B., Wang, X.: DepGraph: Towards any struc- tural pruning. In: IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). pp. 16091–16101 (2023).https://doi.org/10.1109/CVPR52729. 2023.01544 16 V. Barreiro et al

    2023 doi

  3. [11]

    In: International Conference on Learning Representations (ICLR) (2019),https://openreview.net/forum?id=rJl-b3RcF7

    Frankle, J., Carbin, M.: The lottery ticket hypothesis: Finding sparse, trainable neural networks. In: International Conference on Learning Representations (ICLR) (2019),https://openreview.net/forum?id=rJl-b3RcF7

    2019

  4. [12]

    In: IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR)

    He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR). pp. 15979–15986 (2022).https://doi.org/10.1109/ CVPR52688.2022.01553

    2022

  5. [13]

    arXiv preprint arXiv:1503.02531 (2015).https://doi.org/10.48550/arXiv

    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).https://doi.org/10.48550/arXiv. 1503.02531

    2015 doi arXiv

  6. [14]

    In: Chaudhuri, K., Salakhutdinov, R

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Ges- mundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for NLP. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning...

    2019

  7. [15]

    In: International Conference on Learning Representations (ICLR) (2022),https://openreview.net/forum?id= nZeVKeeFYf9

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: International Conference on Learning Representations (ICLR) (2022),https://openreview.net/forum?id= nZeVKeeFYf9

    2022

  8. [16]

    In: International Conference on Machine Learning (ICML) (2024), https://proceedings.mlr.press/v235/huh24a.html

    Huh, M., Cheung, B., Wang, T., Isola, P.: Position: The platonic representation hypothesis. In: International Conference on Machine Learning (ICML) (2024), https://proceedings.mlr.press/v235/huh24a.html

    2024

  9. [17]

    In: IEEE/CVF International Conference on Computer Vision (ICCV) (2025)

    Jakubik, J., Yang, F., Blumenstiel, B., Scheurer, E., Sedona, R., Maurogiovanni, S., Bosmans, J., Dionelis, N., Marsocci, V., Kopp, N., et al.: TerraMind: Large- scale generative multimodality for earth observation. In: IEEE/CVF International Conference on Computer Vision (ICC...

    2025

  10. [18]

    In: European Conference on Computer Vision (ECCV)

    Jia, M., Tang, L., Chen, B.C., Cardie, C., Belongie, S., Hariharan, B., Lim, S.N.: Visual prompt tuning. In: European Conference on Computer Vision (ECCV). pp. 709–727. Springer (2022).https://doi.org/10.1007/978-3-031-19827-4_41

    2022 doi

  11. [19]

    Kikaki, K., Kakogeorgiou, I., Hoteit, I., Karantzalos, K.: Detecting marine pol- lutants and sea surface features with deep learning in Sentinel-2 imagery. IS- PRS Journal of Photogrammetry and Remote Sensing210, 39–57 (2024).https: //doi.org/10.1016/j.isprsjprs.2024.02.017, m...

    2024 doi

  12. [20]

    ACM Comput

    Klabunde, M., Schumacher, T., Strohmaier, M., Lemmerich, F.: Similarity of neural network models: A survey of functional and representational measures. ACM Comput. Surv.57(9) (May 2025).https://doi.org/10.1145/3728458, https://doi.org/10.1145/3728458

    2025 doi

  13. [21]

    In: Chaudhuri, K., Salakhutdinov, R

    Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network representations revisited. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceed- ings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 3...

    2019

  14. [22]

    In: Advances in Neural Information Pro- cessing Systems (NeurIPS) (2023),https://arxiv.org/abs/2306.03831

    Lacoste, A., Lehmann, N., Rodriguez, P., Sherwin, E.D., Kerner, H., Lütjens, B., Irvin, J.A., Dao, D., Alemohammad, H., Drouin, A., et al.: GEO-Bench: Toward foundation models for earth monitoring. In: Advances in Neural Information Pro- cessing Systems (NeurIPS) (2023),https:...

    2023 arXiv

  15. [23]

    In: Proceedings of the 2021 Conference on Empirical Methods SIMPLER 17 in Natural Language Processing

    Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. In: Proceedings of the 2021 Conference on Empirical Methods SIMPLER 17 in Natural Language Processing. pp. 3045–3059. Association for Computational Linguistics, Online and Punta Ca...

    2021 doi

  16. [24]

    In: International Conference on Learning Representations (2017), https://openreview.net/forum?id=rJqFGTslg

    Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for effi- cient ConvNets. In: International Conference on Learning Representations (2017), https://openreview.net/forum?id=rJqFGTslg

    2017

  17. [25]

    arXiv preprint arXiv:2101.00190 (2021),https://arxiv.org/abs/2101.00190

    Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021),https://arxiv.org/abs/2101.00190

    2021 arXiv

  18. [26]

    In: Advances in Neural Information Pro- cessing Systems (2022),https://proceedings.neurips.cc/paper_files/paper/ 2022/hash/deb921bff461a7b0a5c344a4871e7101-Abstract-Conference.html

    Li, Y., Xu, S., Zhang, B., Cao, X., Gao, P., Guo, G.: Q-ViT: Accurate and fully quantized low-bit vision transformer. In: Advances in Neural Information Pro- cessing Systems (2022),https://proceedings.neurips.cc/paper_files/paper/ 2022/hash/deb921bff461a7b0a5c344a4871e7101-Abs...

    2022

  19. [27]

    In: Raedt, L.D

    Lin, Y., Zhang, T., Sun, P., Li, Z., Zhou, S.: FQ-ViT: Post-training quantization for fully quantized vision transformer. In: Raedt, L.D. (ed.) Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. pp. 1173–1179. International Joi...

    2022 doi

  20. [28]

    Michel, P., Levy, O., Neubig, G.: Are sixteen heads really better than one? In: Advances in Neural Information Processing Systems (NeurIPS). pp. 14014–14024 (2019),https : / / proceedings . neurips . cc / paper / 2019 / hash / 2c601ad9d2ff9bc8b282670cdd54f69f-Abstract.html

    2019

  21. [29]

    In: International Conference on Learning Representations (ICLR) (2021),https: //openreview.net/forum?id=KJNcAkY8tY4

    Nguyen, T., Raghu, M., Kornblith, S.: Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth. In: International Conference on Learning Representations (ICLR) (2021),https: //openreview.net/forum?id=KJNcAkY8tY4

    2021

  22. [30]

    In: Neural Information Processing Systems (2017).https://doi.org/10.48550/ arXiv.1706.05806

    Raghu, M., Gilmer, J., Yosinski, J., Sohl-Dickstein, J.N.: SVCCA: Singular vec- tor canonical correlation analysis for deep learning dynamics and interpretability. In: Neural Information Processing Systems (2017).https://doi.org/10.48550/ arXiv.1706.05806

    2017 arXiv

  23. [31]

    (eds.) Advances in Neural Information Processing Systems (2021),https://openreview.net/forum?id=Gl8FHfMVTZu

    Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A.: Do vision transformers see like convolutional neural networks? In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021),https://openreview.n...

    2021

  24. [32]

    Research, F.: fvcore: A light-weight core library for pytorch.https://github.com/ facebookresearch/fvcore(2019), accessed: 2024

    2019

  25. [33]

    IEEE Journal of Selected Topics in Ap- plied Earth Observations and Remote Sensing17, 13893–13907 (2024).https: //doi.org/10.1109/JSTARS.2024.3435081

    Sharma, S., Sedona, R., Riedel, M., Cavallaro, G., Paris, C.: Sen4Map: Advanc- ing mapping with Sentinel-2 by providing detailed semantic descriptions and cus- tomizable land-use and land-cover data. IEEE Journal of Selected Topics in Ap- plied Earth Observations and Remote Se...

    2024 doi

  26. [34]

    Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., Massa, F., Haziza, D., Wehrstedt, L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, ...

    2025 arXiv

  27. [35]

    arXiv preprint arXiv:1906.02243 (2019).https://doi.org/10

    Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243 (2019).https://doi.org/10. 48550/arXiv.1906.02243, seminal work on carbon footprint of training large mod- els 18 V. Barreiro et al

    1906 arXiv

  28. [36]

    Szwarcman, D., Roy, S., Fraccaro, P., Þorsteinn Elí Gíslason, Blumenstiel, B., Ghosal, R., de Oliveira, P.H., de Sousa Almeida, J.L., Sedona, R., Kang, Y., Chakraborty, S., Wang, S., Gomes, C., Kumar, A., Truong, M., Godwin, D., Lee, H., Hsu, C.Y., Asanjan, A.A., Mujeci, B., S...

    2025

  29. [37]

    ACM Computing Surveys55(6), 1–28 (2022).https://doi.org/10.1145/3530811

    Tay, Y., Dehghani, M., Bahri, D., Metzler, D.: Efficient transformers: A survey. ACM Computing Surveys55(6), 1–28 (2022).https://doi.org/10.1145/3530811

    2022 doi

  30. [38]

    In: International Conference on Pattern Recognition (ICPR)

    Teerapittayanon, S., McDanel, B., Kung, H.T.: BranchyNet: Fast inference via early exiting from deep neural networks. In: International Conference on Pattern Recognition (ICPR). pp. 2464–2469 (2016).https://doi.org/10.1109/ICPR. 2016.7900006

    2016 doi

  31. [39]

    In: Meila, M., Zhang, T

    Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jegou, H.: Training data-efficient image transformers and distillation through attention. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine...

    2021

  32. [40]

    In: European Conference on Computer Vision (ECCV)

    Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., Yuan, L.: TinyViT: Fast pretraining distillation for small vision transformers. In: European Conference on Computer Vision (ECCV). pp. 68–85. Springer (2022).https://doi.org/10. 1007/978-3-031-19803-8_5

    2022

  33. [41]

    org/10.1109/MGRS.2025.3576766

    Xiao, A., Xuan, W., Wang, J., Huang, J., Tao, D., Lu, S., Yokoya, N.: Foundation models for remote sensing and earth observation: A survey (2025).https://doi. org/10.1109/MGRS.2025.3576766

    2025 doi

  34. [42]

    In: Proceedings of the 31st ACM International Conference on Multimedia

    Xu, G., Hao, J., Shen, L., Hu, H., Luo, Y., Lin, H., Shen, J.: LGViT: Dynamic early exiting for accelerating vision transformer. In: Proceedings of the 31st ACM International Conference on Multimedia. p. 9103–9114. MM ’23, Association for Computing Machinery, New York, NY, USA...

    2023

  35. [43]

    In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR)

    Yang, H., Yin, H., Shen, M., Molchanov, P., Li, H., Kautz, J.: Global vision trans- former pruning with hessian-aware saliency. In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR). pp. 18547–18557 (June 2023)

    2023

  36. [44]

    In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

    Yang,Z.,Li,Z.,Zeng,A.,Li,Z.,Yuan,C.,Li,Y.:ViTKD:Feature-basedknowledge distillation for vision transformers. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1379–1388 (2024). https://doi.org/10.1109/CVPRW63382.2024.00145

    2024 doi

  37. [45]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR)

    Yu, L., Xiang, W.: X-pruner: explainable pruning for vision transformers. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR). pp. 24355–24363 (June 2023)

    2023

  38. [46]

    SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation

    Yuan, Z., Xue, C., Chen, Y., Wu, Q., Sun, G.: PTQ4ViT: Post-training quan- tization for vision transformers with twin uniform quantization. In: European Conference on Computer Vision (ECCV). pp. 191–207. Springer (2022).https: //doi.org/10.1007/978-3-031-19775-8_12 Supplementa...

    2022 doi

  39. [47]

    Strong augmentation (Mixup 0.8, CutMix 1.0, DropPath 0.1) with 300 epochs for from-scratch training

    to compensate for limited depth; full models use conservative rates (2e-4). Strong augmentation (Mixup 0.8, CutMix 1.0, DropPath 0.1) with 300 epochs for from-scratch training. Post-hoc Pruning Baselines:Magnitude-based (L2 norm) pruning at 20%/40% compression on trained model...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.