Pith. sign in

REVIEW 3 major objections 5 minor 53 references

OpenSeg-R: Improving Open-Vocabulary Segmentation via Step-by-Step Visual Reasoning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OpenSeg-R claims that feeding an open-vocabulary segmentor attribute-based text prompts produced by step-by-step visual reasoning outperforms the same segmentor prompted with class names alone, on five semantic segmentation benchmarks and…

desk verdict Useful plug-in idea, but the paper overclaims SOTA and never isolates whether 'reasoning' or just attribute text drives the gains. read the letter →

arxiv 2505.16974 v2 pith:3D7VULUM submitted 2025-05-22 cs.CV

Zongyan Han , Jiale Cao , Shuo Chen , Tong Wang , Jorma Laaksonen , Rao Muhammad Anwer This is my paper · ORCID
classification cs.CV
keywords open-vocabularysegmentationvisualreasoninglargemultimodalmodelsattribute-basedpromptsmaskensemblingsemanticpanopticpromptcomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OpenSeg-R tries to establish that open-vocabulary segmentation improves when a large multimodal model performs explicit step-by-step visual reasoning before masks are produced. Instead of feeding only class names into a frozen segmentor, the framework has the LMM generate a coarse category, a subcategory, and fine-grained perceptual attributes for each object, then composes prompts like "a photo of {c} that has {r}" and averages together masks produced for each attribute. The paper reports consistent gains over the same base segmentors across five semantic segmentation benchmarks and on ADE20K panoptic segmentation. If true, it would mean that a frozen segmentor can be made more accurate purely by enriching text prompts from a pretrained multimodal model, while also making predictions more interpretable.

What carries the argument

The load-bearing mechanism is the reasoning-to-prompt pipeline. For each image the LMM first writes a global description, then selects candidate classes, then for each class outputs a coarse-to-fine reason consisting of a coarse class, a subclass, and at least three observed visual attributes. A class aligner maps the LMM's class names back to the benchmark label set using sentence-embedding cosine similarity, and unobserved classes receive generic coarse-to-fine reasoning. The prompt composer turns each class-reason pair into "a photo of {c} that has {r}", and the segmentor (SED or MAFT+) generates one mask per attribute; average pooling followed by sigmoid and thresholding yields the final binary mask. The reasoning supplies discriminative visual cues, while the per-attribute mask ensembling is how those cues are combined.

What would settle it

Run the full pipeline on one benchmark while replacing each LMM-generated attribute with an equal number of random but plausible visual phrases for the same class; if mIoU stays at the same level as OpenSeg-R, then reasoning content is not the active ingredient and the gain is attributable to ensembling more text variants.

Watch

Extended reading notes

Core claim

The paper's central claim is that the failure of open-vocabulary segmentors on visually or semantically similar classes is not primarily an architecture problem; it is a prompt information problem. An LMM such as Qwen2.5-VL can supply missing discriminative cues through a hierarchical reasoning triplet: broad class, subclass, and visual attributes. Combining image-specific reasoning, which describes what is present in this particular image, with generic class reasoning, which explains how to recognize classes the LMM may have missed, yields prompts the segmentor can use to generate one mask per attribute, reduced to a final mask by averaging. The paper argues this is the first framework to introduce explicit step-by-step visual reasoning into open-vocabulary segmentation and reports state-of-the-art mIoU on A-150, A-847, PC-59, PC-459, and PAS-20, along with consistent PQ, SQ, and RQ gains on panoptic segmentation.

Load-bearing premise

The paper's measured gains could come from the larger number of text prompts per class being averaged together rather than from the step-by-step reasoning content, because the ablations do not hold the number of prompts fixed.

Editorial extensions

If this is right

  • A frozen open-vocabulary segmentor can be improved without retraining or fine-tuning, purely by enriching text prompts with LMM-generated attributes.
  • Predictions become inspectable: each class mask is tied to stated visual reasons such as "white color, cone shape", which supports debugging and human verification.
  • Because generic reasoning covers classes the LMM did not observe, the method does not depend on perfect object recall and should degrade gracefully on long class lists.
  • The same enrichment transfers across two distinct segmentor architectures and to panoptic segmentation, suggesting the gain is prompt-side rather than architecture-specific.
  • Gains are larger with smaller vision-language encoders and can shrink or reverse with larger ones, since larger encoders may suppress attribute-level cues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editors' inference: if the active ingredient is attribute-level prompt diversity rather than reasoning content, the same gains might be obtainable with a cheaper captioning model or even with random visual phrases, which would weaken the interpretability story.
  • Editors' inference: a direct causal test would hold the number of prompts fixed, comparing N LMM-generated attributes against N alternative but valid visual phrases for the same class to see whether the mIoU gap persists.
  • Editors' inference: the coarse-to-fine triplet suggests an interactive extension the paper does not explore, where a user edits one attribute, such as changing "green border" to "red border", and watches the mask respond at test time.
  • Editors' inference: because the method is prompt-side, its ceiling depends on how well the segmentor's text encoder embeds attribute phrases; attribute prompts that CLIP embeds poorly could fail even when the reasoning itself is correct.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes OpenSeg-R, a training-free wrapper around open-vocabulary segmentors. A large multimodal model (Qwen2.5-VL) first performs hierarchical, coarse-to-fine visual reasoning for each image: global description, candidate-class search, and attribute-level reasoning; unobserved classes receive generic reasoning. The resulting attribute descriptions are composed into prompts of the form 'a photo of {c} that has {attributes}', fed into the segmentor (SED or MAFT+), and the per-attribute masks are averaged and thresholded. The method is evaluated on open-vocabulary semantic segmentation (A-847, PC-459, A-150, PC-59, PAS-20) and open-vocabulary panoptic segmentation on ADE20K. The paper claims state-of-the-art results on five semantic segmentation benchmarks and consistent panoptic gains, with code released.

Significance. If the central claim holds, OpenSeg-R provides a plug-in, training-free way to improve frozen open-vocabulary segmentors by enriching prompts with LMM-generated visual attributes, and it adds interpretability to segmentation decisions. The paper's strengths are its simple integration with two established segmentors, evaluation on several standard benchmarks, and public code. However, the evidence does not currently isolate the step-by-step reasoning structure as the active ingredient, and the stated state-of-the-art claims are partially contradicted by the paper's own table.

major comments (3)
  1. [Abstract, Section 4.3, Table 1] The statement 'Our method surpasses all SOTA methods on A-150, A-847, and PC-459' is contradicted by Table 1. CAT-Seg (ViT-L/14) reports 37.9 on A-150 and 23.8 on PC-459, while OpenSeg-R w/ MAFT+ (ViT-L/14) reports 37.1 and 17.1, and OpenSeg-R w/ SED (ViT-L/14) reports 22.0 on PC-459, below both CAT-Seg and the SED baseline's 22.6. On A-847, OpenSeg-R w/ MAFT+ (ViT-L/14) reaches 16.8, above CAT-Seg's 16.0, but OpenSeg-R w/ SED reaches only 14.3. The abstract's claim of significant gains over state-of-the-art across five benchmarks is therefore not supportable as written; the comparison should be restricted to the direct backbones or explicitly qualified.
  2. [Section 3.3, Section 4.4, Appendix B] The central claim that step-by-step reasoning is the cause of the improvement is not isolated. Appendix B states that SED uses 80 prompts and MAFT+ uses 14 prompts per category, while OpenSeg-R generates one mask per attribute and then performs mask ensembling (Eq. 10). Table 3 ablates prompt types (coarse, coarse+attribute, attribute) but does not include a control with equally detailed attribute phrases obtained without the hierarchical chain, nor a control that matches the total number of ensembled prompts. Since attribute-only prompts perform best and coarse-only prompts degrade performance, the observed gains are consistent with an explanation based on attribute-text ensembling and prompt-template change rather than on the coarse-to-fine reasoning structure. Please add an ablation where the LMM is prompted once per class to produce the same number of attribute phrases, and where the number of ensembled prompts is held equal to the number used by the three-step pipeline.
  3. [Section 4.4, Table 4] The conclusion that 'this hybrid approach yields the best overall results' is not uniformly supported by Table 4. On PAS-20, the 'Both' configuration gives 95.1 and 95.8 for ConvNeXt-B and ConvNeXt-L, respectively, while 'Image-specific' gives 96.1 and 96.2. The text later acknowledges that generic reasoning is omitted for PAS-20 in the final system, but the sentence should be qualified to the datasets where the hybrid is actually used, or a dataset-wise breakdown of what 'overall' means should be provided.
minor comments (5)
  1. [Section 4.3] There is a typo in the phrase 'The proposed Open-OpenSeg-R variants'; this should read 'The proposed OpenSeg-R variants'.
  2. [Appendix B, Figure 5] The text 'producing hierarchical, fine-to-grained reasoning' should be 'fine-grained', and the same typo appears in the figure caption.
  3. [Section 4.2] The implementation details state that Qwen2.5-VL-72B-Instruct-AWQ is used, while Appendix B refers to Qwen2.5-VL-72B-Instruct; please harmonize the model designation.
  4. [Section 3.2.2] The similarity threshold sigma and binarization threshold tau are set to 0.5 without sensitivity analysis; a short study or a sentence justifying these values would strengthen the robustness claims.
  5. [Table 2] The panoptic comparison includes only one backbone (MAFT+) and a small baseline set; the claim of 'consistent improvements across all metrics' is supported relative to MAFT+, but the scope of the claim should be stated accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reasoning prompts are external LMM outputs evaluated against independent baselines, not fitted quantities or self-referential definitions.

full rationale

OpenSeg-R is an empirical pipeline in which a pretrained LMM (Qwen2.5-VL) generates image-specific and generic class reasoning, the reasoning is converted into text prompts, and those prompts are fed to a frozen open-vocabulary segmentor. No equation in the paper defines the predicted metric in terms of the input, and no fitted parameter is renamed as a prediction. The segmentor baselines (SED and MAFT+) are independent works with their own published evaluations; even though one co-author is also an author of SED, SED is used here as the segmentor to be improved rather than as the justification for the method's validity, so the self-citation is not load-bearing. The paper's ablations compare prompt types and reasoning types, and the absence of a control that matches the number of prompts per class is a legitimate experimental confound, but it is not circularity by construction: the attributes are generated by an external model, not derived from the segmentation outputs or benchmark metrics. Similarly, the inconsistency between the abstract's SOTA claim and some numbers in Table 1, and any concern that the LMM's pretraining may overlap with evaluation data, are correctness or leakage issues, not circular-reasoning steps. Under the required standard of exhibiting a specific reduction of a claimed result to its own inputs, no such step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. It relies on hand-set thresholds (sigma, tau) and on assumptions about the reliability of a pretrained LMM and the text encoder. The most significant unvalidated assumption is that the LMM's attributes are grounded in the image, since no quality control is applied to the reasoning outputs.

free parameters (2)
  • aligner similarity threshold sigma = 0.5
    Hand-chosen threshold in Eq. 4 for accepting the class alignment. Not fitted to data, but it directly controls which LMM-predicted classes are discarded, affecting the final class set.
  • binarization threshold tau = 0.5
    Hand-chosen threshold in Eq. 10 for converting the averaged mask sigmoid output into a binary mask. Affects precision/recall of final masks.
assumptions (3)
  • domain assumption The LMM-generated attributes are visually grounded and truthful descriptions of the image content.
    The entire method depends on the LMM (Qwen2.5-VL) producing attributes that actually correspond to the image. No human verification or automatic grounding check is performed. Invoked throughout the pipeline, especially in Section 3.2.
  • domain assumption The text encoder and segmentor can effectively use attribute-enriched prompts like 'a photo of c that has r'.
    The method assumes CLIP text embeddings of longer attribute phrases remain aligned with visual features and improve mask prediction. This is not guaranteed and is only tested indirectly via the final segmentation metric.
  • domain assumption Mask ensembling by average pooling over attribute prompts is a valid aggregation for the final mask.
    Eq. 10 averages the masks from all attribute prompts of a class. This assumes the attributes collectively cover the object region and that averaging does not erase valid predictions. No comparison to other aggregation methods is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenSeg-R: Improving Open-Vocabulary Segmentation via Step-by-Step Visual Reasoning." pith.science (2026). https://pith.science/paper/3D7VULUM

@misc{pith2026250516974,
  author       = {Pith},
  title        = {Pith review of: OpenSeg-R: Improving Open-Vocabulary Segmentation via Step-by-Step Visual Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3D7VULUM}},
  note         = {Machine review of arXiv:2505.16974}
}
read the original abstract

Open-Vocabulary Segmentation (OVS) has drawn increasing attention for its capacity to generalize segmentation beyond predefined categories. However, existing methods typically predict segmentation masks with simple forward inference, lacking explicit reasoning and interpretability. This makes it challenging for OVS model to distinguish similar categories in open-world settings due to the lack of contextual understanding and discriminative visual cues. To address this limitation, we propose a step-by-step visual reasoning framework for open-vocabulary segmentation, named OpenSeg-R. The proposed OpenSeg-R leverages Large Multimodal Models (LMMs) to perform hierarchical visual reasoning before segmentation. Specifically, we generate both generic and image-specific reasoning for each image, forming structured triplets that explain the visual reason for objects in a coarse-to-fine manner. Based on these reasoning steps, we can compose detailed description prompts, and feed them to the segmentor to produce more accurate segmentation masks. To the best of our knowledge, OpenSeg-R is the first framework to introduce explicit step-by-step visual reasoning into OVS. Experimental results demonstrate that OpenSeg-R significantly outperforms state-of-the-art methods on open-vocabulary semantic segmentation across five benchmark datasets. Moreover, it achieves consistent gains across all metrics on open-vocabulary panoptic segmentation. Qualitative results further highlight the effectiveness of our reasoning-guided framework in improving both segmentation precision and interpretability. Our code is publicly available at https://github.com/Hanzy1996/OpenSeg-R.

Figures

Figures reproduced from arXiv: 2505.16974 by the authors.

Figure 1
Figure 1. Comparison with open-vocabulary segmentor and our method. (a) Standard open [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The whole framework of our OpenSeg-R. We first use an LMM to generate image-specific reasoning in a step-by-step manner. Predicted classes are aligned to the predefined label set via a class aligner. For unobserved classes, the LMM generates generic coarse-to-fine reasoning. All reasoning results are composed into prompts and fed to the segmentor to produce masks, which are then aggregated into the final prediction.… view at source ↗
Figure 3
Figure 3. Qualitative comparison with MAFT+ [18] on open-vocabulary semantic segmentation. We highlight the visual reasoning behind the class that is missed by MAFT, demonstrating how our method recovers it through step-by-step visual reasoning. attributes alone achieve the best performance in most cases, suggesting that detailed visual cues provide the most discriminative and informative guidance for open-vocabulary segmenta… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison with MAFT+ [18] on open-vocabulary panoptic segmentation. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Prompt design for step-by-step visual reasoning. The left panel shows the three-stage [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 34 canonical work pages

  1. [1]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  2. [2]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Aleš Kubíˇcek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. In AAAI Conference on Artificial Intelligence, 2023

    work page 2023

  3. [3]

    Zero-shot semantic segmentation

    Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. In Advances in Neural Information Processing Systems, volume 32, pages 468–479, 2019

    work page 2019

  4. [4]

    Universeg: Universal medical image segmentation

    Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Universeg: Universal medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21438–21451, 2023

    work page 2023

  5. [5]

    Coco-stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1209–1218, 2018

    work page 2018

  6. [6]

    Open-vocabulary Panoptic Segmentation with Embedding Modulation

    Xi Chen, Shuang Li, Ser-Nam Lim, Antonio Torralba, and Hengshuang Zhao. Open-vocabulary panoptic segmentation with embedding modulation. arXiv preprint arXiv:2303.11324, 2023

    work page Pith review arXiv 2023

  7. [7]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmentation

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4113–4123, 2024

    work page 2024

  8. [8]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016

    2016

Show all 53 references
  1. [9]

    Bert: Pre-training of deep bidirec- tional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  2. [10]

    Decoupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583–11592, 2022

    2022

  3. [11]

    Open-vocabulary panoptic segmentation with maskclip

    Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskclip. arXiv preprint arXiv:2208.08984, 2022

    2022 arXiv

  4. [12]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303–338, 2010

    2010

  5. [13]

    Scaling open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pages 540–557, 2022

    2022

  6. [14]

    Zero-shot semantic segmentation with decoupled one-pass network

    Cong Han, Yujie Zhong, Dengjie Li, Kai Han, and Lin Ma. Zero-shot semantic segmentation with decoupled one-pass network. In IEEE/CVF International Conference on Computer Vision, pages 1086– 1096, 2023

    2023

  7. [15]

    Global knowledge calibration for fast open-vocabulary segmentation

    Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Jiajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, Yao Zhao, et al. Global knowledge calibration for fast open-vocabulary segmentation. In IEEE/CVF International Conference on Computer Vision, pages 797–807, 2023. 10

    2023

  8. [16]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–4916, 2021

    2021

  9. [17]

    Learning mask-aware clip representations for zero-shot segmentation

    Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao, and Humphrey Shi. Learning mask-aware clip representations for zero-shot segmentation. In Advances in Neural Information Processing Systems, 2023

    2023

  10. [18]

    Collaborative vision-text representation optimizing for open-vocabulary segmentation

    Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yunchao Wei, and Humphrey Shi. Collaborative vision-text representation optimizing for open-vocabulary segmentation. In European Conference on Computer Vision, pages 399–416, 2024

    2024

  11. [19]

    Fineclip: Self-distilled region-based clip for better fine-grained understanding

    Dong Jing, Xiaolong He, Yutian Luo, Nanyi Fei, Guoxing Yang, Wei Wei, Huiwen Zhao, and Zhiwu Lu. Fineclip: Self-distilled region-based clip for better fine-grained understanding. Advances in Neural Information Processing Systems, 2024

    2024

  12. [20]

    Language-driven semantic segmentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, pages 1–13, 2022

    2022

  13. [21]

    Visualbert: A simple and performant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. ArXiv, abs/1908.03557, 2019

    1908 arXiv

  14. [22]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061–7070, 2023

    2023

  15. [23]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014

    2014

  16. [24]

    Remoteclip: A vision language foundation model for remote sensing

    Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Remoteclip: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024

    2024

  17. [25]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023

    2023

  18. [26]

    Open-vocabulary segmentation with semantic-assisted calibration

    Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3491–3500, 2024

    2024

  19. [27]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Neural Information Processing Systems, 2019

    2019

  20. [28]

    The role of context for object detection and semantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 891...

    2014

  21. [29]

    Open vocabulary semantic segmentation with patch aligned contrastive learning

    Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19413–19423, 2023

    2023

  22. [30]

    Multi-modal fusion transformer for end-to-end autonomous driving

    Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7077–7087, 2021

    2021

  23. [31]

    Freeseg: Unified, universal and open-vocabulary image segmentation

    Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al. Freeseg: Unified, universal and open-vocabulary image segmentation. arXiv preprint arXiv:2303.17225, 2023

    2023 arXiv

  24. [32]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pa...

    2021

  25. [33]

    Making monolingual sentence embeddings multilingual using knowledge distillation

    Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2020. 11

    2020

  26. [34]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation languag...

    2023 arXiv

  27. [35]

    Hierarchical open-vocabulary universal image segmentation

    Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hierarchical open-vocabulary universal image segmentation. arXiv preprint arXiv:2307.00764, 2023

    2023 arXiv

  28. [36]

    Skyscript: A large and semantically diverse vision-language dataset for remote sensing

    Zhecheng Wang, Rajanie Prabha, Tianyuan Huang, Jiajun Wu, and Ram Rajagopal. Skyscript: A large and semantically diverse vision-language dataset for remote sensing. In AAAI Conference on Artificial Intelligence, 2024

    2024

  29. [37]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022

    2022

  30. [38]

    Towards open vocabulary learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(7):5092–5113, 2024

    Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, et al. Towards open vocabulary learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(7):5092–5113, 2024

    2024

  31. [39]

    Semantic projection network for zero-and few-label semantic segmentation

    Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8256–8265, 2019

    2019

  32. [40]

    Sed: A simple encoder-decoder for open-vocabulary semantic segmentation

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open-vocabulary semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  33. [41]

    Fg-clip: Fine-grained visual and textual alignment

    Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. Fg-clip: Fine-grained visual and textual alignment. arXiv preprint arXiv:2505.05071, 2025

    2025 arXiv

  34. [42]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134–18144, 2022

    2022

  35. [43]

    Open- vocabulary panoptic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955–2966, 2023

    2023

  36. [44]

    Side adapter network for open- vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2945–2954, 2023

    2023

  37. [45]

    A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision, pages 736–753, 2022

    2022

  38. [46]

    Qwen2.5 technical report

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, ...

    2024 arXiv

  39. [47]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, 2023

    2023

  40. [48]

    Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. arXiv preprint arXiv:2308.02487, 2023

    2023 arXiv

  41. [49]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alexander J. Smola. Automatic chain of thought prompting in large language models. In International Conference on Learning Representations, 2023

    2023

  42. [50]

    Open vocabulary scene parsing

    Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In IEEE/CVF International Conference on Computer Vision, pages 2002–2010, 2017. 12

    2002

  43. [51]

    A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities

    Theodore Zhao, Yu Gu, Jianwei Yang, Naoto Usuyama, Ho Hin Lee, Sid Kiblawi, Tristan Naumann, Jianfeng Gao, Angela Crabtree, Jacob Abel, et al. A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nature methods, 22(...

    2025

  44. [52]

    Semantic understanding of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019

    2019

  45. [53]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, page 696–712, 2022. A Qualitative Results of Open-Vocabulary Panoptic Sementation In Fig. 4, we present qualitative comparisons with MAFT+ [ 18] on open-voc...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.