Pith. sign in

REVIEW 4 major objections 7 minor 2 cited by

Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 210K-image dataset with 1.3M detailed captions, built by fusing rule-based, MLLM, and LLM descriptions, lets a CLIP model beat previous remote-sensing state of the art using 4.2% of the training data.

desk verdict A strong data-centric result—210K LLM-generated pairs beat 5M-pair baselines—with a real confound in the new long-text benchmark that should be fixed before the SOTA claim on that task is taken at face value. read the letter →

arxiv 2507.16716 v1 pith:B4LCQ3OV submitted 2025-07-22 cs.CV

Yiguo He , Junjie Zhu , Yiying Li , Xiaoyu Zhang , Chunping Qiu , Jun Wang , Qiangjuan Huang , Ke Yang This is my paper
classification cs.CV
keywords remotesensingvision-languagemodelsimage-textdatasetgenerationCLIPCoCaMLLMLLMintegrationlong-textretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the main bottleneck for remote-sensing vision-language models is not data volume but caption quality, and that high-quality captions can be manufactured cheaply by combining complementary description sources. It proposes a two-stage pipeline, MpGI, that first generates detailed multi-perspective descriptions for each image using rule-based relays, Kosmos-2, and LLaVA-1.6, then uses LLaMA-3 to fuse those descriptions into diverse, comprehensive captions. The resulting dataset, HQRS-IT-210K with about 210K images and 1.26M captions, is shown to train a CLIP model that outperforms prior RS CLIP models on zero-shot retrieval, few-shot classification, semantic localization, and long-text retrieval while using only 4.2% of GeoRSCLIP's training data. The same dataset fine-tunes a CoCa model, RS-CoCa, whose generated captions are judged comparable or superior to human annotations. A sympathetic reader would care because the paper suggests that data quality, not scale, is the lever for advancing remote-sensing vision-language models.

What carries the argument

The key machinery is the two-stage Multi-Perspective Generation and Integration (MpGI) pipeline. Stage one generates three complementary description streams: rule-based Annotation-to-Description (A2D) and ChatGPT-4V relay expansion for classification and detection datasets; Kosmos-2 with grounding instructions converted via Annotation-to-Instruction (A2I); and LLaVA-1.6 with a detail prompt. Stage two uses LLaMA-3-8B-Instruct with two distinct CO-STAR-framed prompts to fuse the three descriptions into multiple styles of comprehensive captions, with a probability-based fusion (α ≈ 0.5) to mix caption styles. This pipeline turns 23 public datasets with class labels, boxes, and masks into a dense, diverse, and aligned image-text corpus.

What would settle it

A test that would settle the claim: evaluate HQRS-CLIP and RS-CoCa on a held-out suite of RS images that were never part of the 23 source datasets, with captions written by domain experts, and compare retrieval and captioning quality. If the models trained on HQRS-IT-210K underperform on genuinely unseen scenes, the caption quality would be overfit to the source distribution. Additionally, checking whether randomly mixing captions from a model with known hallucinations (e.g., swapping Kosmos-2 or LLaVA-1.6 for a weaker MLLM) degrades performance would isolate whether fusion or the specific MLLMs is load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that multi-perspective, LLM-fused captions are a far more efficient training signal for remote-sensing vision-language models than the rule-based or BLIP2-generated captions used in prior datasets. The paper demonstrates that a CLIP model fine-tuned on only 210K images with six captions per image, selected via a probability-based fusion strategy, outperforms GeoRSCLIP (trained on 5M+ pairs), SkyCLIP (1.2M+), and RemoteCLIP (820K) on RSCTIR zero-shot and fine-tuned retrieval, on the AIR-SLT semantic localization task, and on zero-shot and few-shot classification, while using a fraction of the training data. It also introduces the first RS long-text image retrieval benchmark, LongRET3, and shows that its CLIP model handles long captions better than prior models. For generation, the same dataset fine-tunes CoCa (RS-CoCa) to produce captions that rival or exceed human annotations in manual evaluation and CLIP-similarity distribution.

Load-bearing premise

The generated captions are accurate and comprehensive enough to serve as training signal, which depends on the reliability of Kosmos-2, LLaVA-1.6, and LLaMA-3, plus the manual and regex filtering being sufficient to remove hallucinations.

Editorial extensions

If this is right

  • If caption quality is the dominant factor, then remote-sensing vision-language models can be improved with far less data than previously thought, reducing compute and annotation cost substantially (e.g., 1.5 hours vs 233.4 hours to train ViT-L-14).
  • Dataset construction shifts from collecting more data to designing complementary caption-generation sources; the ablations indicate that each additional perspective (rule-based, Kosmos-2, LLaVA-1.6) adds performance, and that fusing styles with probability α ≈ 0.5 further boosts retrieval and semantic localization.
  • Long-text retrieval becomes a viable evaluation axis for RS vision-language models; the LongRET3 benchmark shows that longer, LLM-rewritten captions reduce category ambiguity and yield higher image-text similarity.
  • Generative RS captioning can reach human-level quality with synthetic training data; RS-CoCa's captions average 48.88 words versus 10.34 for human annotations and score higher on image-caption similarity, suggesting generative models can serve as data-labeling engines for downstream tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The framework likely transfers to other data-scarce vision-language domains (e.g., medical imaging, agriculture, or wildlife surveys) where accurate labels are hard to obtain but where a rule-based signal plus two diverse MLLMs plus an LLM integrator could be assembled at low cost; the paper's ablations show each perspective adds value, so the recipe may be broadly applicable.
  • The probability-based caption fusion (α ≈ 0.5) is essentially a cheap regularizer that prevents overfitting to one caption style; this suggests that synthetic caption diversity, not just caption accuracy, is a trainable asset, and future work could actively search over prompt styles and fusion ratios per domain.
  • Because the dataset sources include drone, detection, and segmentation datasets, the HQRS-CLIP image encoder likely carries stronger object-level and spatial grounding than encoders trained only on classification captions; a testable extension is whether the same dataset improves open-vocabulary detection or grounding tasks beyond retrieval and localization.
  • The long-text retrieval benchmark (LongRET3) may become the standard way to evaluate RS VLMs' comprehension, since short-caption benchmarks saturate and fail to distinguish models that handle detailed scene composition; a practical extension is to validate LongRET3 against human judgments of caption-image alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The manuscript proposes a two-stage caption-generation framework, MpGI, for remote sensing images. In Stage 1 it combines rule-based annotation-to-description conversion (A2D), ChatGPT-4V expansion of class-level captions, and instruction-guided MLLM generation with Kosmos-2 and LLaVA-1.6; in Stage 2 it uses LLaMA-3-8B-Instruct to summarize the multi-perspective drafts into several caption styles. The result is HQRS-IT-210K, with about 210K images and 1.26M captions. The authors fine-tune CLIP and CoCa on this dataset to obtain HQRS-CLIP and RS-CoCa, and report state-of-the-art or competitive results on zero-shot and fine-tuned retrieval, few-shot classification, semantic localization, and captioning, using substantially less training data than GeoRSCLIP. They also introduce LongRET3-test, a long-text retrieval benchmark built by LLaMA-3 rewriting of existing short captions.

Significance. The central data-efficiency claim - that a carefully generated 210K-pair dataset can outperform 5M-pair rule- and BLIP-2-based datasets - is supported by standard external benchmarks (RSITMD, RSICD, UCM, AID, RESISC45, EuroSAT) and by careful ablations of each construction stage. The planned release of the dataset, models, and code, the explicit exclusion of overlapping data in the classification evaluations, and the systematic study of caption length, caption count, and fusion ratio are genuine strengths. However, two headline claims need additional support: the long-text retrieval benchmark is confounded by the fact that the same LLaMA-3 family produced both the training captions and the test captions, and the claim that RS-CoCa captions rival or exceed human annotations is not backed by a controlled human evaluation. These issues affect the novelty and strength of two contributions but do not invalidate the main data-efficiency results on the standard benchmarks.

major comments (4)
  1. [Section III-E / Table V] The LongRET3-test benchmark is constructed by using LLaMA-3-8B-Instruct to summarize the five short captions of RSITMD, RSICD, and UCM, while the HQRS-CLIP training captions were themselves generated by LLaMA-3-8B-Instruct in Stage 2 (Section III-C). HQRS-CLIP therefore sees test text whose lexical choices, sentence structure, and information organization match its training distribution, whereas GeoRSCLIP (trained on BLIP-2 captions) and RemoteCLIP (trained on rule-based captions) do not. The reported gains of 6.94 and 8.03 percentage points in mean recall can be explained by train/test covariate alignment rather than by better image-text correspondence. The paper's own caveat that these results are 'preliminary only' and that HQRS-CLIP was 'trained on longer texts' acknowledges the asymmetry but does not remove the confound. A control set using human-written long captions, or long captions generated by a different model family, is needed before claiming state-of-the-art long-text retrieval or a validated first benchmark.
  2. [Section IV-C6 / Figures 9-10 / Table IX] The conclusion that RS-CoCa generates captions that 'rival or even exceed manual annotations' is supported only by qualitative examples and by a CLIP-similarity score distribution that may simply favor longer, more fluent text. No controlled human evaluation with multiple annotators (e.g., pairwise preference, adequacy and fluency ratings, or correctness of visual details) is reported, and the standard captioning metrics in Table IX are computed with human captions as references, which answers a different question. The claim should either be weakened to 'comparable to human annotations on standard reference-based metrics' or be backed by a human study with inter-annotator agreement.
  3. [Tables III-V] No variance or statistical significance information is reported for any of the retrieval results. Several of the differences that support the headline claims are modest in absolute terms (e.g., Table III RSICD mean recall 29.37 vs 26.18 for GeoRSCLIP; Table V I2T R@1 14.70 vs 11.40 for GeoRSCLIP), and without multiple runs or a paired significance test it is unclear whether these margins are stable. I recommend reporting mean and standard deviation over at least three training runs, or a paired test, for the main comparisons with GeoRSCLIP and RemoteCLIP.
  4. [Section III-B2 / III-C] The hallucination analysis is based on 200 sampled first-stage captions, and the manual sentence-by-sentence review is described only for the ChatGPT-4V relay captions. No quantitative audit (e.g., human-verified factual accuracy or object-presence consistency) is reported for the final LLaMA-3 summaries that actually compose HQRS-IT-210K, even though the paper acknowledges that hallucinations 'cannot be entirely eliminated.' A small human audit of the final captions with agreement statistics would substantiate the 'high-quality' designation for the released dataset.
minor comments (7)
  1. [Section III-A] The sentence 'Ultimately, about 1 5 images were removed' is ambiguous; it should state the exact number or fraction of removed images.
  2. [Section III-B1] The citation '(See Figure XII)' refers to an experimental results table, not a figure; it should read 'Table XII'.
  3. [Figure 5] The caption uses 'Llama-3' while the text uses 'LLaMA-3-8B-Instruct'; the spelling and capitalization should be unified throughout.
  4. [Section IV-B] The phrase 'The modulal interaction' appears to be a typo for 'modal interaction'.
  5. [Figures 8 and 12] The axes in these figures are unlabeled; adding axis labels and units (e.g., caption length in words, CLIP similarity score) would make the distributions interpretable.
  6. [Table IV] No GeoRSCLIP row is reported for the UCM test set; the text should state explicitly whether those numbers are unavailable or were omitted, so readers can compare across all three benchmarks.
  7. [Table III] The rows labeled 'Ours (ViT-L-14)' should be explicitly named 'HQRS-CLIP (ViT-L-14)' to avoid ambiguity with the default HQRS-CLIP ViT-B-32.

Circularity Check

1 steps flagged · score 4.0 of 10

The LongRET3-test long-text retrieval claim is confounded by using the same LLaMA-3-8B-Instruct generator that produced the HQRS-IT-210K training captions, while the paper's central data-efficiency and standard-benchmark results remain independently supported.

  1. other [Section III-C (Stage 2 caption summarization) and Section III-E / Section IV-C2 (LongRET3-test, Table V)]
    "we use LLaMA-3-8B-Instruct to extract comprehensive visual information from the three captions and condense lengthy ones ... We utilized LLaMA-3-8B-Instruct to combine and rewrite the five captions for each image in the test sets of three widely used benchmark datasets (RSITMD, RSICD, and UCM) into a single long-text caption."

    The text distribution seen by HQRS-CLIP during training (Stage 2 captions summarized by LLaMA-3-8B-Instruct) is the same generator family used to construct the LongRET3-test captions from the five short human captions. GeoRSCLIP and RemoteCLIP were not trained on LLaMA-3-generated text, so their lower Table V scores can be explained by train/test distribution mismatch rather than by inferior image-text alignment. The paper itself states the results 'are intended for preliminary evaluation only and might not be fully conclusive, as HQRS-CLIP were trained on longer texts,' and no control long-text set from human-written or differently-generated captions is provided.

full rationale

The dataset construction itself is not circular: captions are produced by a defined two-stage pipeline (Rule-MLLM relay, Kosmos-2, LLaVA-1.6, then LLaMA-3 summarization) and the resulting HQRS-CLIP is evaluated on external benchmarks with human-annotated captions (RSITMD, RSICD, UCM), on AID/RESISC45/EuroSAT for zero-shot classification, and on AIR-SLT for semantic localization. These evaluations are independent of the training data and support the central data-efficiency claim. The fine-tuning protocols exclude overlapping training images before zero-shot classification and few-shot tests, so the reported gains are not merely re-evaluations of training data. No load-bearing uniqueness theorem is imported by self-citation, and the few-shot benchmark reference [70] is a methodological citation rather than a circular premise. The main circularity concern is isolated to the newly proposed LongRET3-test: its captions are generated by the same LLaMA-3-8B-Instruct model family used to create HQRS-IT-210K training captions, so HQRS-CLIP's Table V margin (6.94% mean recall over GeoRSCLIP, 8.03% over RemoteCLIP(FT)) may reflect stylistic and distributional alignment rather than semantic superiority. The paper's explicit caveat that these results are preliminary and that HQRS-CLIP was trained on longer texts confirms the limitation. A secondary concern is that RS-CoCa's 'higher image-caption similarity' evidence (Figure 9) likely uses the same style-matched CLIP scorer, though this is backed by manual evaluation and standard captioning metrics. Overall, the long-text retrieval novelty is partially circular, but the core model claims rest on independent benchmarks, so the circularity score is moderate.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the reliability of several pre-trained models and on the correctness of source annotations. These are domain assumptions rather than new mathematical axioms. The only explicit fitted parameter is the fusion ratio alpha, and the center/edge threshold in the rule-based captions is an ad hoc design choice.

free parameters (2)
  • alpha (a) = 0.5
    Fusion ratio between Prompt-1 and Prompt-2 generated captions, chosen by sweeping alpha and selecting the value with best retrieval and SeLo performance.
  • central area threshold = 1/4 to 3/4 of width and height
    Defines center vs edge in A2D rule-based captions; a hand-chosen geometry, not fitted to data but ad hoc.
assumptions (4)
  • domain assumption The 23 source datasets have accurate class labels, bounding boxes, and segmentation masks.
    The pipeline relies on these annotations for rule-based captions and for grounding MLLM prompts.
  • domain assumption Kosmos-2, LLaVA-1.6, LLaMA-3, and ChatGPT-4V generate captions that are accurate enough for training; residual hallucinations are mostly filtered by prompts, regex, and manual review.
    No independent verification of caption accuracy beyond qualitative samples and CLIP similarity scores.
  • domain assumption Deduplication via p-hash and URL prevents train-test leakage.
    The authors state strict deduplication but do not show cross-check against the test benchmarks.
  • standard math Contrastive and generative losses used in CLIP and CoCa are standard and correctly implemented.
    The paper relies on OpenCLIP implementations; no new mathematical claims are made.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation." pith.science (2026). https://pith.science/paper/B4LCQ3OV

@misc{pith2026250716716,
  author       = {Pith},
  title        = {Pith review of: Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4LCQ3OV}},
  note         = {Machine review of arXiv:2507.16716}
}
read the original abstract

The application of Vision-language foundation models (VLFMs) to remote sensing (RS) imagery has garnered significant attention due to their superior capability in various downstream tasks. A key challenge lies in the scarcity of high-quality, large-scale, image-text paired training data. Recently, several works introduced extensive image-text datasets for RS and trained their VLFMs. However, due to the rudimentary methods used for generating captions, the quality of datasets is suboptimal, requiring larger volumes of training data, while only yielding modest performance improvements. In this paper, we propose a two-stage method named MpGI(Multi-Perspective Generation and Integration) for generating high-quality text captions for RS images. Firstly, we generate distinct and detailed descriptions from different perspectives using Rule-MLLM(Multimodal Large Language Model) Relay Generation and MLLMs generation methods. Next, we utilize Large Language Models (LLMs) to integrate these diverse descriptions into comprehensive captions, capturing details from multiple perspectives. Finally, we have created the HQRS-IT-210K dataset, including about 210,000 RS images and 1.3 million captions. We fine-tuned two VLFMs using our dataset: CLIP, a discriminative model, and CoCa, an image-to-text generative model. This process resulted in our proposed HQRS-CLIP and RS-CoCa models. Experimental results demonstrate that HQRS-CLIP surpassed the previous SOTA RS CLIP model in various downstream tasks while using only 4.2\% of the training data. RS-CoCa outperforms other advanced approaches across benchmark datasets and can generate captions for RS images that rival or even exceed manual annotations. Dataset, pre-trained models, and codes will be released at https://github.com/YiguoHe/HQRS-210K-and-HQRS-CLIP.

Figures

Figures reproduced from arXiv: 2507.16716 by the authors.

Figure 1
Figure 1. Samples of large-scale RS image-text datasets. SkyScript captions are rule-based, resulting in highly uniform sentence [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the MpGI method. to comprehensively summarize the “multi-view” descriptions, creating multiple semantically complete image captions with different styles via multiple prompts. This process integrates complementary information from different descriptions, filters out grammatical and semantic errors, and creates captions that capture diverse details from various perspectives. To address the uniform style t… view at source ↗
Figure 3
Figure 3. Overview of the caption generation method. A. After being transformed into DET*4 by the M2B algorithm [ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The RS Cross-modal Text-Image Retrieval result of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Pipeline of stage 2. Description 1 refers to the caption [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: A Demo of Original Captions in UCM and Rewriting [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 6
Figure 6. Figure 6: Wordcloud and Caption Length Distribution of HQRS [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: CLIP Similarity Score Distribution of RET3-test, RET3-test-join, and LongRET3-test. To address these shortcomings, LLMs can be employed for caption integration and rewriting [66], effectively resolving redundancy, coherence, and error issues (see [PITH_FULL_IMAGE:figu…
Figure 9
Figure 9. Figure 9: CLIP Similarity Score Distribution Between RS-CoCa [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Sampled Captions generated by RS-CoCa. TABLE IX: Results of RS Captioning task. The best result is in bold. Dataset Method BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE L CIDEr SPICE RSICD BITA 0.774 0.665 0.577 0.504 0.420 0.717 3.045 0.548 RSGPT 0.703 0.542 0.440 0.368 0…
Figure 13
Figure 13. Figure 13: Effectiveness of Different Caption Numbers. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 12
Figure 12. Figure 12: Effectiveness of Different Caption Length. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 14
Figure 14. Figure 14: Effectiveness of Different α Value. 7) Effectiveness of Different α Values: After the two cap￾tion generation stages, we found that mixing two types of LLM-generated captions increases the diversity of the training data, thereby enhancing CLIP’s performance. To identi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Under one zero-shot protocol, general-purpose MLLMs match or outperform remote-sensing-specific MLLMs on several RS benchmarks, while RS-MLLMs keep advantages in visual grounding, RS-VQA, and ultra-high-resolution und...

  2. Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Visual prompts align SAM 3 to remote-sensing geometry while textual prompts inject ground-level bias that degrades detection; a training-free proxy shows strong segmentation Harmonic Mean but persistent small-object a...

Reference graph

Works this paper leans on

69 extracted references · 40 canonical work pages · cited by 2 Pith papers

  1. [1]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR, 2021, pp. 8748–8763

    2021

  2. [2]

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, T. Duerig, Scaling up visual and vision-language representation learning with noisy text supervision, in: International conference on machine learning, PMLR, 2021, pp. 4904–4916

    work page 2021

  3. [3]

    Y . Li, F. Liang, L. Zhao, Y . Cui, W. Ouyang, J. Shao, F. Yu, J. Yan, Supervision exists everywhere: A data effi- cient contrastive language-image pre-training paradigm, arXiv preprint arXiv:2110.05208

    arXiv

  4. [4]

    Q. Sun, Y . Fang, L. Wu, X. Wang, Y . Cao, Eva-clip: Improved training techniques for clip at scale, arXiv preprint arXiv:2303.15389

    arXiv

  5. [5]

    J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhos- seini, Y . Wu, Coca: Contrastive captioners are image-text foundation models, arXiv preprint arXiv:2205.01917

    arXiv

  6. [6]

    J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, X. Wang, Groupvit: Semantic segmentation emerges from text supervision, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18134–18144

    work page 2022

  7. [7]

    Gu, T.-Y

    X. Gu, T.-Y . Lin, W. Kuo, Y . Cui, Open-vocabulary object detection via vision and language knowledge distillation, arXiv preprint arXiv:2104.13921

    arXiv

  8. [8]

    H. Xu, G. Ghosh, P.-Y . Huang, D. Okhonko, A. Agha- janyan, F. Metze, L. Zettlemoyer, C. Feichtenhofer, Videoclip: Contrastive pre-training for zero-shot video- text understanding, arXiv preprint arXiv:2109.14084

    arXiv

Show all 69 references
  1. [9]

    Guzhov, F

    A. Guzhov, F. Raue, J. Hees, A. Dengel, Audioclip: Extending clip to image, text and audio, in: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022, pp. 976–980

    2022

  2. [10]

    Zhang, Z

    R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y . Qiao, P. Gao, H. Li, Pointclip: Point cloud understand- ing by clip, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 8552–8562

    2022

  3. [11]

    H. Liu, C. Li, Q. Wu, Y . J. Lee, Visual instruction tuning, Advances in neural information processing systems 36. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 17

  4. [12]

    Z. Peng, W. Wang, L. Dong, Y . Hao, S. Huang, S. Ma, F. Wei, Kosmos-2: Grounding multimodal large language models to the world, arXiv preprint arXiv:2306.14824

    arXiv

  5. [13]

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, J. Zhou, Qwen-vl: A frontier large vision- language model with versatile abilities, arXiv preprint arXiv:2308.12966

    arXiv

  6. [14]

    Abbas, K

    A. Abbas, K. Tirumala, D. Simig, S. Ganguli, A. S. Morcos, Semdedup: Data-efficient learning at web- scale through semantic deduplication, arXiv preprint arXiv:2303.09540

    arXiv

  7. [15]

    Doveh, A

    S. Doveh, A. Arbelle, S. Harary, R. Herzig, D. Kim, P. Cascante-Bonilla, A. Alfassy, R. Panda, R. Giryes, R. Feris, et al., Dense and aligned captions (dac) pro- mote compositional reasoning in vl models, Advances in Neural Information Processing Systems 36

  8. [16]

    Barham, A

    P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy, et al., Pathways: Asynchronous distributed dataflow for ml, Proceedings of Machine Learning and Systems 4 (2022) 430–449

    2022

  9. [17]

    F. Liu, D. Chen, Z. Guan, X. Zhou, J. Zhu, Q. Ye, L. Fu, J. Zhou, Remoteclip: A vision language foun- dation model for remote sensing, IEEE Transactions on Geoscience and Remote Sensing

  10. [18]

    Z. Wang, R. Prabha, T. Huang, J. Wu, R. Rajagopal, Skyscript: A large and semantically diverse vision- language dataset for remote sensing, in: Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 38, 2024, pp. 5805–5813

    2024

  11. [19]

    Zhang, T

    Z. Zhang, T. Zhao, Y . Guo, J. Yin, Rs5m: A large scale vision-language dataset for remote sens- ing vision-language foundation model, arXiv preprint arXiv:2306.11300

    arXiv

  12. [20]

    X. Li, C. Wen, Y . Hu, N. Zhou, Rs-clip: Zero shot remote sensing scene classification via contrastive vision- language supervision, International Journal of Applied Earth Observation and Geoinformation 124 (2023) 103497

    2023

  13. [21]

    Djoufack Basso, Clip-rs: A cross-modal remote sens- ing image retrieval based on clip, a northern virginia case study, Ph.D

    L. Djoufack Basso, Clip-rs: A cross-modal remote sens- ing image retrieval based on clip, a northern virginia case study, Ph.D. thesis, Virginia Tech (2022)

    2022

  14. [22]

    Y . Bazi, L. Bashmal, M. M. Al Rahhal, R. Ricci, F. Mel- gani, Rs-llava: A large vision-language model for joint captioning and question answering in remote sensing imagery, Remote Sensing 16 (9) (2024) 1477

    2024

  15. [23]

    Y . Hu, J. Yuan, C. Wen, X. Lu, X. Li, Rsgpt: A remote sensing vision language model and benchmark, arXiv preprint arXiv:2307.15266

    arXiv

  16. [24]

    Y . Zhan, Z. Xiong, Y . Yuan, Skyeyegpt: Unifying remote sensing vision-language tasks via instruction tuning with large language model, arXiv preprint arXiv:2401.09712

    arXiv

  17. [25]

    Goyal, P

    S. Goyal, P. Maini, Z. C. Lipton, A. Raghunathan, J. Z. Kolter, Scaling laws for data filtering–data curation cannot be compute agnostic, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22702–22711

    2024

  18. [26]

    G.-S. Xia, J. Hu, F. Hu, B. Shi, X. Bai, Y . Zhong, L. Zhang, X. Lu, Aid: A benchmark data set for per- formance evaluation of aerial scene classification, IEEE Transactions on Geoscience and Remote Sensing 55 (7) (2017) 3965–3981

    2017

  19. [27]

    Urbanek, F

    J. Urbanek, F. Bordes, P. Astolfi, M. Williamson, V . Sharma, A. Romero-Soriano, A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26700–26709

    2024

  20. [28]

    W. Zhou, S. Newsam, C. Li, Z. Shao, Patternnet: A benchmark dataset for performance evaluation of remote sensing image retrieval, ISPRS journal of photogramme- try and remote sensing 145 (2018) 197–209

    2018

  21. [29]

    X. Lu, B. Wang, X. Zheng, X. Li, Exploring models and data for remote sensing image caption generation, IEEE Transactions on Geoscience and Remote Sensing 56 (4) (2017) 2183–2195

    2017

  22. [30]

    Z. Yuan, Z. Xiong, L. Mou, X. X. Zhu, Chatearthnet: A global-scale, high-quality image-text dataset for remote sensing, arXiv preprint arXiv:2402.11325

    arXiv

  23. [31]

    Z. Lai, H. Zhang, B. Zhang, W. Wu, H. Bai, A. Timofeev, X. Du, Z. Gan, J. Shan, C.-N. Chuah, et al., Veclip: Improving clip training via visual-enriched captions, in: European Conference on Computer Vision, Springer, 2025, pp. 111–127

    2025

  24. [32]

    Grubinger, P

    M. Grubinger, P. Clough, H. M ¨uller, T. Deselaers, The iapr tc-12 benchmark: A new evaluation resource for visual information systems, in: International workshop ontoImage, V ol. 2, 2006

    2006

  25. [33]

    Rashtchian, P

    C. Rashtchian, P. Young, M. Hodosh, J. Hockenmaier, Collecting image annotations using amazon’s mechanical turk, in: Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s Mechanical Turk, 2010, pp. 139–147

    2010

  26. [34]

    Hodosh, P

    M. Hodosh, P. Young, J. Hockenmaier, Framing image description as a ranking task: Data, models and evalua- tion metrics, Journal of Artificial Intelligence Research 47 (2013) 853–899

    2013

  27. [35]

    Young, A

    P. Young, A. Lai, M. Hodosh, J. Hockenmaier, From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions, Transactions of the Association for Computational Lin- guistics 2 (2014) 67–78

    2014

  28. [36]

    X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Doll ´ar, C. L. Zitnick, Microsoft coco captions: Data collection and evaluation server, arXiv preprint arXiv:1504.00325

    arXiv

  29. [37]

    Ordonez, G

    V . Ordonez, G. Kulkarni, T. Berg, Im2text: Describing images using 1 million captioned photographs, Advances in neural information processing systems 24

  30. [38]

    Sharma, N

    P. Sharma, N. Ding, S. Goodman, R. Soricut, Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning, in: Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (V olume 1: Long Papers), 2018, pp. 2556–2565

    2018

  31. [39]

    L. Yuan, D. Chen, Y .-L. Chen, N. Codella, X. Dai, J. Gao, IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 18 H. Hu, X. Huang, B. Li, C. Li, et al., Florence: A new foundation model for computer vision, arXiv preprint arXiv:2111.11432

    arXiv

  32. [40]

    B. Zhu, B. Lin, M. Ning, Y . Yan, J. Cui, H. Wang, Y . Pang, W. Jiang, J. Zhang, Z. Li, et al., Lan- guagebind: Extending video-language pretraining to n- modality by language-based semantic alignment, arXiv preprint arXiv:2310.01852

    arXiv

  33. [41]

    L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, D. Lin, Sharegpt4v: Improving large multi- modal models with better captions, arXiv preprint arXiv:2311.12793

    arXiv

  34. [42]

    B. Qu, X. Li, D. Tao, X. Lu, Deep semantic understand- ing of high resolution remote sensing image, in: 2016 International conference on computer, information and telecommunication systems (Cits), IEEE, 2016, pp. 1–5

    2016

  35. [43]

    Z. Yuan, W. Zhang, K. Fu, X. Li, C. Deng, H. Wang, X. Sun, Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval, arXiv preprint arXiv:2204.09868

    arXiv

  36. [44]

    Cheng, H

    Q. Cheng, H. Huang, Y . Xu, Y . Zhou, H. Li, Z. Wang, Nwpu-captions dataset and mlca-net for remote sensing image captioning, IEEE Transactions on Geoscience and Remote Sensing 60 (2022) 1–19

    2022

  37. [45]

    Czerkawski, A

    M. Czerkawski, A. Francis, From laion-5b to laion-eo: Filtering billions of images using anchor datasets for satellite image extraction, arXiv preprint arXiv:2309.15535

    arXiv

  38. [46]

    Vaswani, Attention is all you need, Advances in Neural Information Processing Systems

    A. Vaswani, Attention is all you need, Advances in Neural Information Processing Systems

  39. [47]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosse- lut, E. Brunskill, et al., On the opportunities and risks of foundation models, arXiv preprint arXiv:2108.07258

    arXiv

  40. [48]

    Devlin, Bert: Pre-training of deep bidirectional trans- formers for language understanding, arXiv preprint arXiv:1810.04805

    J. Devlin, Bert: Pre-training of deep bidirectional trans- formers for language understanding, arXiv preprint arXiv:1810.04805

    arXiv

  41. [49]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text trans- former, Journal of machine learning research 21 (140) (2020) 1–67

    2020

  42. [50]

    Lewis, Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension, arXiv preprint arXiv:1910.13461

    M. Lewis, Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension, arXiv preprint arXiv:1910.13461

    1910 arXiv

  43. [51]

    Radford, Improving language understanding by gen- erative pre-training

    A. Radford, Improving language understanding by gen- erative pre-training

  44. [52]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (8) (2019) 9

    2019

  45. [53]

    T. B. Brown, Language models are few-shot learners, arXiv preprint arXiv:2005.14165

    2005 arXiv

  46. [55]

    Z. Wang, Z. Wu, D. Agarwal, J. Sun, Medclip: Con- trastive learning from unpaired medical images and text, arXiv preprint arXiv:2210.10163

    arXiv

  47. [56]

    J. Li, D. Li, S. Savarese, S. Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, in: International conference on machine learning, PMLR, 2023, pp. 19730–19742

    2023

  48. [57]

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, S. Hoi, Instructblip: Towards general- purpose vision-language models with instruction tuning, arXiv preprint arXiv:2305.06500

    arXiv

  49. [58]

    Huang, L

    S. Huang, L. Dong, W. Wang, Y . Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, et al., Language is not all you need: Aligning perception with language models, Advances in Neural Information Processing Systems 36

  50. [59]

    H. Liu, C. Li, Y . Li, Y . J. Lee, Improved baselines with visual instruction tuning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26296–26306

    2024

  51. [60]

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, Y . J. Lee, Llava-next: Improved reasoning, ocr, and world knowledge (2024)

    2024

  52. [61]

    C. Yang, Z. Li, L. Zhang, Bootstrapping interactive image-text alignment for remote sensing image cap- tioning, IEEE Transactions on Geoscience and Remote Sensing

  53. [62]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774

    arXiv

  54. [63]

    W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y . Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al., Cogvlm: Visual expert for pretrained language models, arXiv preprint arXiv:2311.03079

    arXiv

  55. [64]

    Young, B

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al., Yi: Open foundation models by 01. ai, arXiv preprint arXiv:2403.04652

    arXiv

  56. [65]

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, P. Fung, Survey of hallucination in natural language generation, ACM Computing Surveys 55 (12) (2023) 1–38

    2023

  57. [66]

    D. M. Chan, A. Myers, S. Vijayanarasimhan, D. A. Ross, J. Canny, Ic3: Image captioning by committee consensus, arXiv preprint arXiv:2302.01328

    arXiv

  58. [67]

    Teo, How i won singapore’s gpt-4 prompt engineering competition, Towards Data Science, Medium 29

    S. Teo, How i won singapore’s gpt-4 prompt engineering competition, Towards Data Science, Medium 29

  59. [68]

    Loshchilov, F

    I. Loshchilov, F. Hutter, Decoupled weight decay regu- larization, arXiv preprint arXiv:1711.05101

    arXiv

  60. [69]

    A. v. d. Oord, Y . Li, O. Vinyals, Representation learn- ing with contrastive predictive coding, arXiv preprint arXiv:1807.03748

    arXiv

  61. [70]

    C. Qiu, X. Zhang, X. Tong, N. Guan, X. Yi, K. Yang, J. Zhu, A. Yu, Few-shot remote sensing image scene classification: Recent advances, new baselines, and future trends, ISPRS Journal of Photogrammetry and Remote Sensing 209 (2024) 368–382

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.