Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EarthDial is claimed to be the first unified vision-language model for multi-resolution, multi-spectral, and multi-temporal Earth observation imagery, reporting higher accuracy than generic and domain-specific models across 44 downstream…

desk verdict A genuinely big instruction dataset and a plausible multi-modal RS VLM, but the zero-shot evaluation is unauditable and the paper's own appendix undermines the headline generalization claim. read the letter →

arxiv 2412.15190 v2 pith:CL7UZVEQ submitted 2024-12-19 cs.CV

Sagar Soni , Akshay Dudhane , Hiyam Debary , Mustansar Fiaz , Muhammad Akhtar Munir , Muhammad Sohail Danish , Paolo Fraccaro , Campbell D Watson
show 3 more authors
Levente J Klein Fahad Shahbaz Khan Salman Khan
This is my paper · ORCID
classification cs.CV
keywords Earthobservationvision-languagemodelinstructiontuningremotesensingmulti-spectralimagerymulti-temporalanalysissyntheticapertureradarchangedetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EarthDial is a vision-language model built specifically for Earth observation data. The paper claims it is the first unified assistant that can take in multi-resolution, multi-spectral, and multi-temporal satellite and aerial imagery and answer in natural language. To make this work, the authors assembled an instruction-tuning dataset of more than 11.11 million question-answer pairs covering optical, SAR, infrared, and multispectral channels, and trained a 4-billion-parameter model in three stages. Across 44 downstream benchmarks, EarthDial is reported to outperform both generic vision-language models and earlier remote-sensing-specific models. If these results hold, a single lightweight assistant could serve tasks from disaster assessment to methane-plume detection without retraining per sensor.

What carries the argument

The load-bearing pieces are the EarthDial-Instruct dataset (11.11M QA pairs across modalities) and two architectural modules. Adaptive High Resolution tiles images into 448x448 patches plus a global thumbnail so variable resolutions enter the vision transformer without distortion, while Data Fusion processes multispectral or temporal inputs three channels at a time, encodes each group through the ViT, aggregates features via the AnyRes bilinear-interpolation block, and concatenates the result with text embeddings for the LLM. The three-stage training schedule (multi-sensor caption pretraining, then RGB and temporal instruction tuning, then multispectral and SAR tuning) is what carries the generalization claim.

What would settle it

Run a membership test comparing the test images of each of the 44 downstream datasets against the 11.11M instruction-tuning pairs; any overlap in a benchmark that the paper labels zero-shot would show the reported generalization is inflated.

Watch

Extended reading notes

Core claim

EarthDial's central claim is that one model can jointly process the three axes that make Earth observation data hard for generic vision-language models: resolution, spectral band, and time. The paper demonstrates this by building the largest remote-sensing instruction dataset to date (11.11M pairs) and a three-stage training recipe on a 4B InternVL/Phi-3 backbone, with an adaptive high-resolution tiling module and a data-fusion module that aggregates features from groups of spectral or temporal channels. On 44 downstream tasks spanning scene classification, referred object detection, region captioning, grounding, VQA, image captioning, change detection, disaster assessment, methane-plume detection, tree-species classification, local-climate-zone classification, and urban-heat-island classification, EarthDial is reported to beat GPT-4o, InternVL2, and GeoChat, including on zero-shot evaluations. The authors also report that the multi-spectral version of BigEarthNet benefits from the fusion module, and that full fine-tuning outperforms LoRA adaptation for zero-shot detection.

Load-bearing premise

The evaluations are fair: none of the 11.11M training instructions were built from the same images or labels used to test the 44 downstream benchmarks, and all baselines were prompted and scored under the same conditions.

Editorial extensions

If this is right

  • A single 4-billion-parameter model can serve classification, detection, captioning, VQA, grounding, and change detection on RGB, SAR, multispectral, infrared, and hyperspectral inputs, removing the need for separate per-task or per-sensor systems.
  • The 11.11M-instruction EarthDial-Instruct dataset is the largest remote-sensing instruction set to date and is the resource that enables the model's multi-sensor, multi-resolution, multi-temporal behavior.
  • EarthDial reports large gains over GPT-4o, InternVL2, and GeoChat on the reported 44 tasks, including zero-shot detection and captioning benchmarks.
  • The bilinear-interpolation fusion strategy is found to outperform average and max pooling for multispectral classification, and full fine-tuning is found to outperform LoRA for zero-shot referred-object detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the zero-shot results reproduce under a stricter protocol, the same three-stage recipe could be transferred to other sensor families (e.g., LiDAR or drone video) by generating instruction pairs from existing annotated datasets.
  • Because the data-fusion module processes channels in groups of three, the architecture may extend to arbitrary multispectral sensors without per-sensor retraining, provided the ViT can encode the channel subset.
  • A useful stress test would be to evaluate EarthDial on disasters or regions absent from its 44 benchmarks; performance there would clarify whether the model generalizes to novel geography and events.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. EarthDial is a 4B-parameter vision-language model for Earth observation, trained in three stages on a claimed 11.11M instruction pairs spanning RGB, SAR, multispectral (including NIR and infrared), and multi-temporal imagery. The model adapts InternVL with adaptive high-resolution tiling and a data-fusion module that processes multispectral channels in groups of three, and is evaluated on scene classification, detection, region captioning, grounding, VQA, image captioning, change detection, disaster assessment, and methane-plume detection. The paper reports consistent outperformance over generic (GPT-4o, InternVL2) and domain-specific (GeoChat, LHRS-Bot, EarthGPT) baselines across these tasks, and attributes this to the large instruction dataset and the multi-stage training recipe. The main claims are that EarthDial is the first unified EO VLM supporting multi-resolution, multi-spectral, and multi-temporal inputs, and that it generalizes better than existing models across 44 downstream tasks.

Significance. If the evaluation is trustworthy, a single 4B assistant that handles multi-spectral, multi-temporal, and multi-resolution observations would be a meaningful step for Earth-observation foundation models, and the scale of the instruction dataset (claimed 11.11M pairs) is itself a contribution. The paper provides a clear architecture, a three-stage training strategy, ablations showing the benefit of multi-stage pretraining and bilinear fusion, and useful qualitative and failure-case analyses. However, the central generalization claim is only as strong as the zero-shot evaluation protocol, and that protocol is currently not auditable: the paper does not provide a split manifest, does not enumerate which instruction datasets were used in each stage, and does not state explicitly whether QA pairs generated from evaluation test splits were excluded from training. The significance of the contribution therefore hinges on a verification that readers cannot perform with the information given.

major comments (5)
  1. [Section 5 and Appendix Table A.1] The zero-shot evaluations may be contaminated by training on test-split QA pairs. Table A.1 lists AID, UCMerced, WHU-RS19, RSVQA, NWPU-Captions, and many other evaluation datasets with their test splits, and the table note states 'we generate QA-pairs for each split separately.' Since the Stage 2 and Stage 3 instruction data in Table 2 are not enumerated by source dataset, the paper does not rule out that QA pairs generated from evaluation test-split images were included in the 11.11M-instruction training set. This ambiguity directly affects the validity of the 'zero-shot' labels in Tables 4, 6, 7, 8, 9, 10, and 12, and hence the abstract's claim of 'better generalization.' The authors must provide a split manifest that lists, for every evaluation dataset, which splits were used for generating training instructions and which were held out, and must explicitly state that no evaluation-split QA pairs appear in the instruction-tuning data; if that is not the case, the experiments must be rerun without such pairs.
  2. [Section 5, Tables 6-12] The protocol for marking zero-shot results is inconsistent across tables. Table 6 appends '(ZS)' to some dataset columns but not others (e.g., GeoChat-Instruct is not marked, while NWPU VHR-10 is); Table 7 marks HIT-UAV, NWPU-VHR10, Swimming Pool, UCAS AOD, and Urban Tree Crown as ZS but leaves the remaining columns unmarked; Table 9 marks only RSITMD as zero-shot; Table 10 labels RSVQA-HRBEN as zero-shot in the text; and Table 11 marks only SYSU as zero-shot. Section 5 explicitly identifies only three supervised rows (BigEarthNet, xBD Set 1, fMoW in Table 4). Without a table-level enumeration of which datasets were used in any training stage, the reader cannot interpret the numbers or the 'zero-shot' designations. The authors should add a status column (or footnote) to every evaluation table indicating whether EarthDial was instruction-tuned on that dataset, and should avoid presenting supervised and zero-shot results in a single table without clear visual separation.
  3. [Abstract, Section 5, and Conclusion] The headline claims that EarthDial 'outperforms existing generic and domain-specific models' across 44 downstream tasks and 'achieves better generalization' are not supported by the reported protocol because a substantial fraction of the benchmarks are datasets on which EarthDial was instruction-tuned. For example, GeoChat-Instruct appears in Tables 6-8, several captioning datasets (NWPU-Captions, RSCID, Sydney, UCM Captions) appear in Table 9, and many change-detection datasets (LEVIR-MCI, Dubai-CC, MUDS) appear in Table 11 without zero-shot markers, while Section 5 explicitly labels BigEarthNet, xBD Set 1, and fMoW as supervised. The generalization claim should be restated only for the zero-shot subset, or the paper should separately report fine-tuned and zero-shot aggregate results. As written, the abstract and conclusion conflate in-domain fine-tuning performance with generalization.
  4. [Section 5, Tables 5, 9-12] No error bars, standard deviations, or significance tests are reported. Although the margins are often large, several evaluations involve relatively small test sets (e.g., UHI-AD, STARCOP, and the xBD sub-tasks), and the results are sensitive to prompt phrasing and decoding randomness in generative models. The authors should report the test-set size for each benchmark and, where feasible, the standard deviation over multiple decoding runs or a paired significance test against the strongest baseline. This is needed to establish that the reported differences, especially on the smaller benchmarks, are not due to chance or to a favorable prompt template.
  5. [Section 4, Table 2, and Appendix Table A.1] The claimed dataset size of 11.11M instruction pairs is inconsistent with the sums in Table 2. Summing the rows gives approximately 7.67M (Stage 1) + 1.85M (Stage 2) + 2.48M (Stage 3) = 12.0M, which does not match the '11.11M' figure used in the Abstract and Section 1. The authors should resolve this discrepancy and state exactly how an 'instruction pair' is counted (e.g., whether each attribute-specific prompt is a separate pair, and whether the values in Table 2 are unique samples or prompt occurrences).
minor comments (7)
  1. [Section 3.1] The data fusion module groups multispectral channels 'in groups of three' without a stated justification; please explain why three channels were chosen and whether the grouping order (e.g., band ordering) affects the results.
  2. [Section 4 and Appendix A] The label-based and image-based filtering thresholds are described qualitatively ('sparse labels (<3)', 'luminance and coverage thresholds') but the exact numerical values are not given; please provide the precise thresholds for reproducibility.
  3. [Table 5] The table caption says 'referred object detection tasks on SAR imagery,' but the table also reports classification results for BigEarthNet (MS), SoSAT-LCZ42 (MS), and TreeSatAI (RGBI), which are not SAR datasets; please correct the caption to describe all the settings shown.
  4. [Section 6, Table 13] The text states that bilinear fusion 'enhances the average classification accuracy by 9.5%,' but the table shows a 19.35-point gain on BigEarthNet and a 7.84-point gain on TreeSatAI (67.01 vs 47.66 and 56.93 vs 49.09); please clarify how the 9.5% figure is computed.
  5. [Section 5 (Methane Plume Classification)] There is a typo in the last sentence: 'EarDial' should be 'EarthDial.'
  6. [Figure 2] The architecture figure is too dense to read at publication size; please increase the font size of labels and separate the text boxes visually.
  7. [Section 1 and Conclusion] The paper refers collectively to '44 downstream tasks,' but no list of the 44 datasets is provided in the main text; please include a table or list mapping each task and dataset to the evaluation tables.

Circularity Check

1 steps flagged · score 6.0 of 10

Zero-shot evaluations are trained on test-split QA pairs from the same benchmarks, so the headline 'better generalization' partly reduces to retrieval of the instruction set.

  1. fitted input called prediction [Appendix A, Table A.1; Section 5 (Scene classification); Tables 4 and 6-11]
    "For zero-shot evaluation, we compare EarthDial with RGB datasets in Tab. 4, whereas BigEarthNet (RGB), xBD Set 1, and fMoW are supervised. ... Here, split means that we generate QA-pairs for each split separately."

    Appendix Table A.1 lists AID, UCMerced, WHU-RS19, RSVQA-HRBEN, NWPU-VHR10, UCAS-AOD, HIT-UAV, Swimming Pool, Urban Tree Crown, SYSU-CC, and other benchmarks under the downstream instruction-generation pipeline, explicitly with 'test' splits, and states that QA-pairs are generated for each split separately. Section 5 and Tables 4-11 then report these same benchmarks as zero-shot evaluations. Since Table 2 shows Stages 2-3 finetuning on the EarthDial-Instruct dataset, the zero-shot rows are not independent tests of generalization: the model was trained on QA pairs derived from the same images and labels used for scoring. The claimed 'better generalization' is therefore partly forced by the training set rather than demonstrated by prediction on unseen data.

full rationale

EarthDial's architecture and dataset construction are internally coherent: the adaptive-resolution and data-fusion modules are described concretely, and the multi-stage training recipe is not derived from the evaluation numbers. The central problem is the boundary between training and evaluation. Section 5 labels AID, UCMerced, and WHU-19 as zero-shot in Table 4, and Tables 6-11 label NWPU-VHR10, UCAS-AOD, HIT-UAV, Swimming Pool, Urban Tree Crown, SYSU-CC, and RSVQA-HRBEN as zero-shot. Appendix Table A.1 lists all of these datasets under the downstream instruction-generation pipeline, with 'test' splits, and states 'we generate QA-pairs for each split separately.' If those generated test-split QA pairs entered EarthDial-Instruct (Table 2, Stages 2-3), then the zero-shot rows measure the model's ability to reproduce pairs it was trained on, not generalization to unseen data. The paper does not release the instruction dataset or a split manifest, so this cannot be audited. The supervised rows, such as BigEarthNet, xBD, and fMoW, are not circular because training on a benchmark's train split and testing on its test split is standard supervised evaluation; the circularity is specifically the zero-shot labeling of benchmarks whose test-split QA pairs appear in the training pipeline. No load-bearing self-citation chain is present; citations to GeoChat and CDChat are ordinary baseline and dataset references.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on data curation choices, architecture assumptions, and evaluation protocol assumptions rather than on mathematical axioms. The main load-bearing choices are the instruction-generation model, the spectral fusion scheme, and the assumption that benchmark splits cleanly separate training from evaluation. No new physical entities or forces are postulated; the special modality and task tokens are formatting conventions, not independent entities.

free parameters (5)
  • Label filter threshold = >=3 labels
    Appendix A, Step I: samples with fewer than 3 labels are filtered out; this hand-chosen threshold shapes the instruction dataset.
  • Cloud and coverage filter thresholds = Unspecified
    Appendix A, Step II: luminance and coverage thresholds remove cloudy or low-coverage images, but the actual threshold values are not reported.
  • Adaptive patch size range = 1 to 12 tiles in training, up to 40 at inference
    Section 3.1: the dynamic resolution strategy is adopted from InternVL 1.5 with these hand-chosen ranges.
  • Number of training epochs per stage = 1
    Table 3: each stage uses one epoch, a hand-chosen training budget.
  • LoRA rank (ablation only) = 128
    Appendix B: used in the LoRA comparison; the rank is set by hand and influences the adaptation capacity.
assumptions (5)
  • domain assumption InternVL 1.5 dynamic high-resolution tiling improves fine-grained understanding without losing global context
    Section 3.1: the adaptive high-resolution block is adopted from InternVL 1.5; if tiling degrades spectral or global context, the multi-spectral and multi-temporal claims weaken.
  • domain assumption InternLM-XComposer2-generated instruction pairs are accurate enough to train a trustworthy VLM
    Section 4 and Appendix A: all pretraining QA pairs are generated by InternLM-XComposer2 from labels, with only random manual verification; label or generation errors propagate into the model.
  • ad hoc to paper Processing multi-spectral channels in groups of three and fusing with bilinear interpolation preserves band-specific discriminative information
    Section 3.1, Data Fusion: this design choice is introduced for this paper; if grouping destroys spectral distinctions, the reported multispectral gains over RGB are unsupported.
  • domain assumption Evaluation test splits are disjoint from instruction-tuning imagery
    Section 5: 'zero-shot' labels imply no overlap, but no geographic or tile deduplication is described; overlap would inflate reported accuracies.
  • domain assumption Phi-3-mini and InternViT-300M are adequate backbones for Earth observation understanding
    Section 3.1: model capacity and pretrained weights are taken as given from prior work, with no analysis of backbone-induced bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues." pith.science (2026). https://pith.science/paper/CL7UZVEQ

@misc{pith2026241215190,
  author       = {Pith},
  title        = {Pith review of: EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CL7UZVEQ}},
  note         = {Machine review of arXiv:2412.15190}
}
read the original abstract

Automated analysis of vast Earth observation data via interactive Vision-Language Models (VLMs) can unlock new opportunities for environmental monitoring, disaster response, and {resource management}. Existing generic VLMs do not perform well on Remote Sensing data, while the recent Geo-spatial VLMs remain restricted to a fixed resolution and few sensor modalities. In this paper, we introduce EarthDial, a conversational assistant specifically designed for Earth Observation (EO) data, transforming complex, multi-sensory Earth observations into interactive, natural language dialogues. EarthDial supports multi-spectral, multi-temporal, and multi-resolution imagery, enabling a wide range of remote sensing tasks, including classification, detection, captioning, question answering, visual reasoning, and visual grounding. To achieve this, we introduce an extensive instruction tuning dataset comprising over 11.11M instruction pairs covering RGB, Synthetic Aperture Radar (SAR), and multispectral modalities such as Near-Infrared (NIR) and infrared. Furthermore, EarthDial handles bi-temporal and multi-temporal sequence analysis for applications like change detection. Our extensive experimental results on 44 downstream datasets demonstrate that EarthDial outperforms existing generic and domain-specific models, achieving better generalization across various EO tasks. Our source codes and pre-trained models are at https://github.com/hiyamdebary/EarthDial.

Figures

Figures reproduced from arXiv: 2412.15190 by the authors.

Figure 1
Figure 1. EarthDial is the first domain-specific VLM for earth [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. EarthDial Architecture: The model can take a diverse set of inputs ranging from RGB to multi-spectral and time-series images. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. EarthDial training Strategy for different RS modali [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Illustration of our versatile EarthDial model that performs across multi-modalities, multi-resolution, multispectral, and multi￾temporal data from diverse remote sensing applications. EarthDial extends its capabilities to a range of tasks such as scene classification, …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing

    cs.CV 2025-01 conditional novelty 6.0 of 10

    GeoPixel brings pixel-level grounding to remote sensing large multimodal models for the first time, with a new dataset and benchmark built from iSAID.

Reference graph

Works this paper leans on

75 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 4

    arXiv 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  4. [4]

    Omnisat: Self-supervised modality fusion for earth observation

    Guillaume Astruc, Nicolas Gonthier, Clement Mallet, and Loic Landrieu. Omnisat: Self-supervised modality fusion for earth observation. arXiv preprint arXiv:2404.08351 , 2024. 6, 12

    arXiv 2024

  5. [5]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 7

    arXiv 2023

  6. [6]

    Satlaspretrain: A large- scale dataset for remote sensing image understanding

    Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdi- nando, and Aniruddha Kembhavi. Satlaspretrain: A large- scale dataset for remote sensing image understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16772–16782, 2023. 5, 10

    work page 2023

  7. [7]

    Nwpu vhr-10, 2023

    Luxuan Bian. Nwpu vhr-10, 2023. 7

    work page 2023

  8. [8]

    QuakeSet: A Dataset and Low-Resource Models to Monitor Earthquakes through Sentinel-1

    Daniele Rege Cambrin and Paolo Garza. Quakeset: A dataset and low-resource models to monitor earthquakes through sentinel-1. arXiv preprint arXiv:2403.18116, 2024. 6, 8, 12

    work page Pith review arXiv 2024

Show all 75 references
  1. [9]

    Visualgpt: Data-efficient adaptation of pretrained language models for image captioning

    Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed El- hoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030–18040, 2022. 2

    2022

  2. [10]

    K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal llm’s referential di- alogue magic. arXiv preprint arXiv:2306.15195, 2023. 1

    2023 arXiv

  3. [11]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 1, 2, 3, 4

    2024 arXiv

  4. [12]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  5. [13]

    Multi-class geospatial object detection and geographic im- age classification based on collection of part detectors

    Gong Cheng, Junwei Han, Peicheng Zhou, and Lei Guo. Multi-class geospatial object detection and geographic im- age classification based on collection of part detectors. IS- PRS Journal of Photogrammetry and Remote Sensing , 98: 119–132, 2014. 7, 12

    2014

  6. [14]

    Remote sens- ing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sens- ing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, 2017. 7, 12

    2017

  7. [15]

    Nwpu-captions dataset and mlca-net for remote sensing image captioning

    Qimin Cheng, Haiyan Huang, Yuan Xu, Yuzhuo Zhou, Huanying Li, and Zhongyuan Wang. Nwpu-captions dataset and mlca-net for remote sensing image captioning. IEEE Transactions on Geoscience and Remote Sensing , 60:1–19,

  8. [16]

    Meerkat: Audio-visual large language model for grounding in space and time

    Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Di- nesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. arXiv preprint arXiv:2407.01851, 2024. 2

    2024 arXiv

  9. [17]

    Functional map of the world

    Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6172–6180, 2018. 5, 6, 12

    2018

  10. [18]

    Satellite image classification via two-layer sparse coding with biased image representa- tion

    Dengxin Dai and Wen Yang. Satellite image classification via two-layer sparse coding with biased image representa- tion. IEEE Geoscience and remote sensing letters, 8(1):173– 176, 2010. 6, 12

    2010

  11. [19]

    Vg4d: Vision-language model goes 4d video recognition

    Zhichao Deng, Xiangtai Li, Xia Li, Yunhai Tong, Shen Zhao, and Mengyuan Liu. Vg4d: Vision-language model goes 4d video recognition. arXiv preprint arXiv:2404.11605, 2024. 2

    2024 arXiv

  12. [20]

    A public dataset for fine-grained ship classification in optical remote sensing images

    Yanghua Di, Zhiguo Jiang, and Haopeng Zhang. A public dataset for fine-grained ship classification in optical remote sensing images. Remote Sensing, 13(4):747, 2021. 12

    2021

  13. [21]

    Object detection in aerial im- ages: A large-scale benchmark and challenges

    Jian Ding, Nan Xue, Gui-Song Xia, Xiang Bai, Wen Yang, Michael Ying Yang, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, et al. Object detection in aerial im- ages: A large-scale benchmark and challenges. IEEE trans- actions on pattern analysis and machine intellige...

    2021

  14. [22]

    Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model. arXiv preprint arXiv:2401.1...

    2024 arXiv

  15. [23]

    Phys- ically grounded vision-language models for robotic manipu- lation

    Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Phys- ically grounded vision-language models for robotic manipu- lation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462–12469. IEE...

    2024

  16. [24]

    Creating xbd: A dataset for assessing building damage from satellite imagery

    Ritwik Gupta, Bryce Goodman, Nirav Patel, Ricky Hosfelt, Sandra Sajeev, Eric Heim, Jigar Doshi, Keane Lucas, Howie Choset, and Matthew Gaston. Creating xbd: A dataset for assessing building damage from satellite imagery. In Pro- ceedings of the IEEE/CVF conference on computer ...

    2019

  17. [25]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 12

    2019

  18. [26]

    Rsgpt: A remote sensing vision language model and benchmark

    Yuan Hu, Jianlong Yuan, Congcong Wen, Xiaonan Lu, and Xiang Li. Rsgpt: A remote sensing vision language model and benchmark. arXiv preprint arXiv:2307.15266, 2023. 2

    2023 arXiv

  19. [27]

    Language is not all you need: Aligning perception with language mod- els

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language mod- els. Advances in Neural Information Processing Systems , 36:72096–721...

    2023

  20. [28]

    Geochat: Grounded large vision-language model for remote sensing

    Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad Shahbaz Khan. Geochat: Grounded large vision-language model for remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27831– 2784...

    2024

  21. [29]

    Srsdd-v1

    Songlin Lei, Dongdong Lu, Xiaolan Qiu, and Chibiao Ding. Srsdd-v1. 0: A high-resolution sar rotation ship detection dataset. Remote Sensing, 13(24):5104, 2021. 7, 12

    2021

  22. [30]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 4

    2024 arXiv

  23. [31]

    Rsi-cb: A large-scale remote sensing image classification benchmark using crowd- sourced data

    Haifeng Li, Xin Dou, Chao Tao, Zhixiang Wu, Jie Chen, Jian Peng, Min Deng, and Ling Zhao. Rsi-cb: A large-scale remote sensing image classification benchmark using crowd- sourced data. Sensors, 20(6):1594, 2020. 12

    2020

  24. [32]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  25. [33]

    Object detection in optical remote sensing images: A survey and a new benchmark

    Ke Li, Gang Wan, Gong Cheng, Liqiu Meng, and Junwei Han. Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS journal of photogram- metry and remote sensing, 159:296–307, 2020. 12

    2020

  26. [34]

    Vrsbench: A versatile vision-language benchmark dataset for remote sens- ing image understanding

    Xiang Li, Jian Ding, and Mohamed Elhoseiny. Vrsbench: A versatile vision-language benchmark dataset for remote sens- ing image understanding. arXiv preprint arXiv:2406.12384,

    arXiv

  27. [35]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2025. 2

    2025

  28. [36]

    Vlm2scene: Self-supervised image-text-lidar learning with foundation models for autonomous driving scene understanding

    Guibiao Liao, Jiankun Li, and Xiaoqing Ye. Vlm2scene: Self-supervised image-text-lidar learning with foundation models for autonomous driving scene understanding. InPro- ceedings of the AAAI Conference on Artificial Intelligence , pages 3351–3359, 2024. 2

    2024

  29. [37]

    Change-agent: Towards interactive comprehensive remote sensing change interpre- tation and analysis

    Chenyang Liu, Keyan Chen, Haotian Zhang, Zipeng Qi, Zhengxia Zou, and Zhenwei Shi. Change-agent: Towards interactive comprehensive remote sensing change interpre- tation and analysis. IEEE Transactions on Geoscience and Remote Sensing, 2024. 7, 12

    2024

  30. [38]

    Re- moteclip: A vision language foundation model for remote sensing

    Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Re- moteclip: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024. 2

    2024

  31. [39]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 1

    2024

  32. [40]

    Rsvqa: Visual question answering for remote sensing data

    Sylvain Lobry, Diego Marcos, Jesse Murray, and Devis Tuia. Rsvqa: Visual question answering for remote sensing data. IEEE Transactions on Geoscience and Remote Sensing , 58 (12):8555–8566, 2020. 12

    2020

  33. [41]

    Exploring models and data for remote sensing im- age caption generation

    Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xue- long Li. Exploring models and data for remote sensing im- age caption generation. IEEE Transactions on Geoscience and Remote Sensing, 56(4):2183–2195, 2017. 7, 12

    2017

  34. [42]

    Skysensegpt: A fine-grained in- struction tuning dataset and model for remote sensing vision- language understanding

    Junwei Luo, Zhen Pang, Yongjun Zhang, Tingzhu Wang, Linlin Wang, Bo Dang, Jiangwei Lao, Jian Wang, Jingdong Chen, Yihua Tan, et al. Skysensegpt: A fine-grained in- struction tuning dataset and model for remote sensing vision- language understanding. arXiv preprint arXiv:2406.10100,

    arXiv

  35. [43]

    Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model

    Dilxat Muhtar, Zhenshi Li, Feng Gu, Xueliang Zhang, and Pengfeng Xiao. Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model. arXiv preprint arXiv:2402.02544, 2024. 2, 7

    2024 arXiv

  36. [44]

    Cdchat: A large multimodal model for remote sensing change description

    Mubashir Noman, Noor Ahsan, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, and Fa- had Shahbaz Khan. Cdchat: A large multimodal model for remote sensing change description. arXiv preprint arXiv:2409.16261, 2024. 7, 12

    2024 arXiv

  37. [45]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint, abs/2303.08774, 2023. Available at https://doi.org/10.48550/arXiv.2303.08774. 11, 14

    2023 doi arXiv

  38. [46]

    C. Pang, X. Weng, J. Wu, J. Li, Y . Liu, J. Sun, W. Li, S. Wang, L. Feng, G.S. Xia, and C. He. VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis. arXiv, 2024. 2, 3

    2024

  39. [47]

    Deep semantic understanding of high resolution remote sensing image

    Bo Qu, Xuelong Li, Dacheng Tao, and Xiaoqiang Lu. Deep semantic understanding of high resolution remote sensing image. In 2016 International conference on computer, in- formation and telecommunication systems (Cits), pages 1–5. IEEE, 2016. 7, 12

    2016

  40. [48]

    Floodnet: A high resolution aerial imagery dataset for post flood scene understanding

    Maryam Rahnemoonfar, Tashnim Chowdhury, Argho Sarkar, Debvrat Varshney, Masoud Yari, and Robin Rober- son Murphy. Floodnet: A high resolution aerial imagery dataset for post flood scene understanding. IEEE Access , 9:89644–89654, 2021. 12

    2021

  41. [49]

    Se- mantic segmentation of methane plumes with hyperspectral machine learning models

    Vit Ruzicka, Gonzalo Mateo-Garcia, Luis Gomez-Chova, Anna Vaughan, Luis Guanter, and Andrew Markham. Se- mantic segmentation of methane plumes with hyperspectral machine learning models. Scientific Reports, 13(1):19999,

  42. [50]

    Bigearthnet: A large-scale benchmark archive for remote sensing image understanding

    Gencer Sumbul, Marcela Charfuelan, Beg ¨um Demir, and V olker Markl. Bigearthnet: A large-scale benchmark archive for remote sensing image understanding. In IGARSS 2019- 2019 IEEE International Geoscience and Remote Sensing Symposium, pages 5901–5904. IEEE, 2019. 5, 6, 12

    2019

  43. [51]

    Fair1m: A benchmark dataset for fine- grained object recognition in high-resolution remote sens- ing imagery

    Xian Sun, Peijin Wang, Zhiyuan Yan, Feng Xu, Ruiping Wang, Wenhui Diao, Jin Chen, Jihao Li, Yingchao Feng, Tao Xu, et al. Fair1m: A benchmark dataset for fine- grained object recognition in high-resolution remote sens- ing imagery. ISPRS Journal of Photogrammetry and Remote Se...

    2022

  44. [52]

    Visual grounding in remote sensing images

    Yuxi Sun, Shanshan Feng, Xutao Li, Yunming Ye, Jian Kang, and Xu Huang. Visual grounding in remote sensing images. In Proceedings of the 30th ACM International Con- ference on Multimedia, pages 404–412, 2022. 6, 12

    2022

  45. [53]

    Hit-uav: A high-altitude in- frared thermal dataset for unmanned aerial vehicle-based ob- ject detection

    Jiashun Suo, Tianyi Wang, Xingzhou Zhang, Haiyang Chen, Wei Zhou, and Weisong Shi. Hit-uav: A high-altitude in- frared thermal dataset for unmanned aerial vehicle-based ob- ject detection. Scientific Data, 10(1):227, 2023. 7, 12

    2023

  46. [54]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  47. [55]

    A comprehensive review of multimodal large language models: Performance and challenges across different tasks

    Jiaqi Wang, Hanqi Jiang, Yiheng Liu, Chong Ma, Xu Zhang, Yi Pan, Mengyuan Liu, Peiran Gu, Sichen Xia, Wenjun Li, et al. A comprehensive review of multimodal large language models: Performance and challenges across different tasks. arXiv preprint arXiv:2408.01319, 2024. 2

    2024 arXiv

  48. [56]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  49. [57]

    A sar dataset of ship detection for deep learn- ing under complex backgrounds

    Yuanyuan Wang, Chao Wang, Hong Zhang, Yingbo Dong, and Sisi Wei. A sar dataset of ship detection for deep learn- ing under complex backgrounds. remote sensing, 11(7):765,

  50. [58]

    Skyscript: A large and seman- tically diverse vision-language dataset for remote sensing

    Zhecheng Wang, Rajanie Prabha, Tianyuan Huang, Jiajun Wu, and Ram Rajagopal. Skyscript: A large and seman- tically diverse vision-language dataset for remote sensing. In Proceedings of the AAAI Conference on Artificial Intel- ligence, pages 5805–5813, 2024. 5, 10

    2024

  51. [59]

    Solving robotics problems in zero-shot with vision-language models

    Zidan Wang, Rui Shen, and Bradly Stadie. Solving robotics problems in zero-shot with vision-language models. arXiv preprint arXiv:2407.19094, 2024. 2

    2024 arXiv

  52. [60]

    Aid: A benchmark data set for performance evaluation of aerial scene classification

    Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xiaoqiang Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7):3965–3981, 2017. 6, 12

    2017

  53. [61]

    Vlm-grounder: A vlm agent for zero-shot 3d visual grounding

    Runsen Xu, Zhiwei Huang, Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Vlm-grounder: A vlm agent for zero-shot 3d visual grounding. arXiv preprint arXiv:2410.13860, 2024. 2

    2024 arXiv

  54. [62]

    Made to order: Discovering monotonic temporal changes via self-supervised video ordering

    Charig Yang, Weidi Xie, and Andrew Zisserman. Made to order: Discovering monotonic temporal changes via self-supervised video ordering. arXiv preprint arXiv:2404.16828, 2024. 6, 7, 12

    2024 arXiv

  55. [63]

    Bag-of-visual-words and spa- tial extensions for land-use classification

    Yi Yang and Shawn Newsam. Bag-of-visual-words and spa- tial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pages 270–279, 2010. 6, 12

    2010

  56. [64]

    Exploring a fine-grained multiscale method for cross-modal remote sensing image re- trieval

    Zhiqiang Yuan, Wenkai Zhang, Kun Fu, Xuan Li, Chubo Deng, Hongqi Wang, and Xian Sun. Exploring a fine-grained multiscale method for cross-modal remote sensing image re- trieval. arXiv preprint arXiv:2204.09868, 2022. 7, 12

    2022 arXiv

  57. [65]

    Bench- marking anchor-based and anchor-free state-of-the-art deep learning methods for individual tree detection in rgb high- resolution images

    Pedro Zamboni, Jos ´e Marcato Junior, Jonathan de Andrade Silva, Gabriela Takahashi Miyoshi, Edson Takashi Matsub- ara, Keiller Nogueira, and Wesley Nunes Gonc ¸alves. Bench- marking anchor-based and anchor-free state-of-the-art deep learning methods for individual tree detect...

    2021

  58. [66]

    Rsvg: Exploring data and models for visual grounding on remote sensing data

    Yang Zhan, Zhitong Xiong, and Yuan Yuan. Rsvg: Exploring data and models for visual grounding on remote sensing data. IEEE Transactions on Geoscience and Remote Sensing , 61: 1–13, 2023. 6, 12

    2023

  59. [67]

    Skyeyegpt: Unifying remote sensing vision-language tasks via instruc- tion tuning with large language model

    Yang Zhan, Zhitong Xiong, and Yuan Yuan. Skyeyegpt: Unifying remote sensing vision-language tasks via instruc- tion tuning with large language model. arXiv preprint arXiv:2401.09712, 2024. 2, 3

    2024 arXiv

  60. [68]

    Good at captioning, bad at counting: Benchmarking gpt-4v on earth observation data

    Chenhui Zhang and Sherrie Wang. Good at captioning, bad at counting: Benchmarking gpt-4v on earth observation data. arXiv preprint arXiv: 2401.17600, 2024. 1

    2024 arXiv

  61. [69]

    Multi-step question- driven visual question answering for remote sensing

    Meimei Zhang, Fang Chen, and Bin Li. Multi-step question- driven visual question answering for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2023. 12

    2023

  62. [70]

    Earthgpt: A universal multi-modal large lan- guage model for multi-sensor image comprehension in re- mote sensing domain

    Wei Zhang, Miaoxin Cai, Tong Zhang, Yin Zhuang, and Xuerui Mao. Earthgpt: A universal multi-modal large lan- guage model for multi-sensor image comprehension in re- mote sensing domain. IEEE Transactions on Geoscience and Remote Sensing, 2024. 2, 3, 7

    2024

  63. [71]

    Rs5m and georsclip: A large scale vision-language dataset and a large vision-language model for remote sensing

    Zilun Zhang, Tiancheng Zhao, Yulong Guo, and Jianwei Yin. Rs5m and georsclip: A large scale vision-language dataset and a large vision-language model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024. 3

    2024

  64. [72]

    Patternnet: A benchmark dataset for performance evaluation of remote sensing image retrieval

    Weixun Zhou, Shawn Newsam, Congmin Li, and Zhenfeng Shao. Patternnet: A benchmark dataset for performance evaluation of remote sensing image retrieval. ISPRS journal of photogrammetry and remote sensing, 145:197–209, 2018. 12

    2018

  65. [73]

    Openannotate3d: Open- vocabulary auto-labeling system for multi-modal 3d data

    Yijie Zhou, Likun Cai, Xianhui Cheng, Zhongxue Gan, Xi- angyang Xue, and Wenchao Ding. Openannotate3d: Open- vocabulary auto-labeling system for multi-modal 3d data. In 2024 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 9086–9092. IEEE, 2024. 2

    2024

  66. [74]

    Orientation robust object detection in aerial images using deep convolutional neural network

    Haigang Zhu, Xiaogang Chen, Weiqun Dai, Kun Fu, Qixi- ang Ye, and Jianbin Jiao. Orientation robust object detection in aerial images using deep convolutional neural network. In 2015 IEEE international conference on image processing (ICIP), pages 3735–3739. IEEE, 2015. 7, 12

    2015

  67. [75]

    So2sat lcz42: A bench- mark dataset for global local climate zones classification

    Xiao Xiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi, Jian Kang, Lichao Mou, Hossein Bagheri, Matthias H¨aberle, Yuansheng Hua, Rong Huang, et al. So2sat lcz42: A bench- mark dataset for global local climate zones classification. arXiv preprint arXiv:1912.12171, 2019. 5, 6, 12

    1912 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.