Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that fine-tuning CLIP on ground-truth masks while preserving its original similarity ordering improves open-vocabulary segmentation.

desk verdict Core idea is solid, but headline OVS gains are confounded by a backbone swap and the consistency-alignment principle is thin on evidence. read the letter →

arxiv 2412.11464 v3 pith:3TOIJZDA submitted 2024-12-16 cs.CV

Quan-Sheng Zeng , Yunheng Li , Daquan Zhou , Guanbin Li , Qibin Hou , Ming-Ming Cheng This is my paper · ORCID
classification cs.CV
keywords open-vocabularysegmentationmaskclassificationCLIPfine-tuningconsistencyalignmentground-truthmasksparameterizedsimilaritymodelingzero-shotgenerators
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MaskCLIP++ is a fine-tuning framework that improves open-vocabulary image segmentation by training CLIP's mask classification on ground-truth masks instead of masks produced by a generator. The authors argue that previous methods weaken vision–language alignment because they train on noisy generated masks, and that adding parameters to CLIP only helps if the optimization preserves CLIP's original similarity ordering among region embeddings. After low-cost fine-tuning on COCO Stuff, MaskCLIP++ raises mIoU by +1.7, +2.3, +2.1, +3.1, and +0.3 on A-847, PC-459, A-150, PC-59, and PAS-20 when combined with existing mask generators. The same framework improves panoptic and instance segmentation and works with both open-vocabulary and closed-vocabulary generators, suggesting the bottleneck is mask classification rather than mask generation.

What carries the argument

The load-bearing mechanism is a mask-conditioned modulation of CLIP's visual encoder plus a consistency-preserving similarity projector. The visual encoder is split into an Extractor (layers 1 to K) and a Fuser (layers K+1 to L); ground-truth masks M enter as attention priors through $\phi^{(l)}(M) = \mathrm{softmax}\left(\frac{q(E_m^{(l)})k(F^{(l)})^T}{\sqrt{d}} + \alpha M_{th}\right)$, producing mask-specific tokens $E_m^{(l+1)}$. The parameterized similarity modeling (PSM) then computes the similarity matrix $S = E_m E_t^T$ and projects it through an over-parameterized linear layer along a new dimension orthogonal to both the mask and text dimensions, trained with cross-entropy on the mask-category annotations. The consistency alignment principle requires that similarity orderings before and after this projection agree, which the paper argues is what lets the extra parameters improve alignment without overfitting.

What would settle it

Fine-tune MaskCLIP++ on COCO Stuff with matched generated masks instead of ground-truth masks, keeping the consistency-preserving PSM and all hyperparameters identical; if the generated-mask model still outperforms the inconsistent PSMs by the same margin rather than falling toward the pixel-prior baseline, then the reported gains come from the PSM design rather than from mask quality.

Watch

Extended reading notes

Core claim

The central claim is that generated masks are the weak link in mask-based open-vocabulary segmentation: mask generators already propose many valid masks for unseen categories, but classification errors keep those masks from being used. MaskCLIP++ therefore removes the generator from training and fine-tunes CLIP directly on ground-truth masks, obtaining region embeddings by feeding masks into the later “Fuser” layers of CLIP's visual encoder through mask-conditioned attention. To keep this fine-tuning from overfitting the closed vocabulary of the training set, the authors introduce the consistency alignment principle: additional parameters should preserve CLIP's original similarity ordering among mask embeddings and text embeddings, formalized in a parameterized similarity modeling (PSM) block that projects the similarity matrix along an over-parameterized orthogonal dimension. They report that this design improves mask classification accuracy on multi-domain datasets and, combined with existing mask generators, improves open-vocabulary semantic, panoptic, and instance segmentation.

Load-bearing premise

The load-bearing premise is that preserving CLIP's original similarity ordering among region embeddings is both necessary and sufficient for additional parameters to improve alignment without overfitting; if this principle fails to transfer to other mask generators or datasets, the reported gains could be an artifact of the chosen architecture.

Editorial extensions

If this is right

  • Mask classification, not mask generation, is the current bottleneck: closed-vocabulary generators like Mask2Former gain substantially when paired with MaskCLIP++.
  • Training can be decoupled from the mask generator, so the same fine-tuned CLIP works with semantic, panoptic, and instance-level generators at inference time.
  • High-quality priors matter more than their form: ground-truth masks outperform pixels, boxes, and matched generated masks for fine-tuning.
  • Fine-tuning is data-efficient: 1% of COCO-Stuff (about 1K images) already gives competitive mask accuracy.
  • Consistency-preserving PSM raises both seen and unseen category mIoU, whereas inconsistent PSMs can raise seen mIoU while dropping unseen mIoU below the no-fine-tuning baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The consistency alignment principle is stated generally and could be tested on other CLIP adaptation tasks, such as open-vocabulary detection or referring segmentation, where preserving the pre-trained similarity order is also desirable.
  • The paper's logic implies that investing in better mask classifiers may be a cheaper path to open-vocabulary segmentation than training larger open-vocabulary mask generators; this is an inference, not a claim the authors make.
  • The authors report results only after fine-tuning on COCO; an obvious extension is fine-tuning on multiple segmentation datasets at once, which the consistency principle predicts should continue to avoid categorical bias.
  • MaskCLIP++ also improves mask-free, unsupervised methods such as SCLIP, ClearCLIP, and ProxyCLIP when its weights are loaded, suggesting the fine-tuned CLIP's improved local representations transfer beyond mask-based pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MaskCLIP++, a fine-tuning framework for CLIP that replaces generated masks with ground-truth masks during training, adds a mask-conditioned fusion module in the later CLIP layers, and introduces a parameterized similarity modeling (PSM) module guided by a 'consistency alignment' principle. The method is evaluated on mask classification (MESS benchmark) and on open-vocabulary semantic, panoptic, and instance segmentation, where MaskCLIP++ is combined with the mask generators of FC-CLIP and MAFT+. The authors report consistent mask-accuracy gains over pretrained CLIP and over CLIPSelf and CAT-Seg under a fixed ViT-L/14 backbone, and report mIoU improvements over FC-CLIP and MAFT+ when integrated with their mask generators.

Significance. If the headline results hold, MaskCLIP++ would be a simple, low-cost way to improve mask classification for open-vocabulary segmentation without retraining mask generators, and the 'consistency alignment' idea could be useful beyond this specific task. The paper's strengths include a controlled mask-accuracy comparison in Tab. 1 (same ViT-L/14 architecture and input resolution for all methods), a clear empirical motivation that ground-truth masks are better training priors than generated masks, and publicly released code. The main reservation is that the abstract's integrated OVS gains are not established by the experiments as reported, because the comparisons in Tab. 2 mix a backbone change with the proposed fine-tuning.

major comments (3)
  1. [§4.2, Table 2] The advertised improvements over FC-CLIP and MAFT+ are confounded: the FC-CLIP and MAFT+ rows use a ConvNeXt-L backbone, while both MaskCLIP++ rows use EVA02 ViT-L/14. The training data also differ for the FC-CLIP row (COCO Panoptic vs COCO Stuff for MaskCLIP++). Table 13 shows that the backbone choice alone changes mask accuracy by about 5 points on A-150 (61.0 vs 66.0 between ConvNeXt-L and ViT-L/14), and Tab. 1 shows the EVA02 ViT-L/14 already outperforms OpenAI CLIP in several domains before any fine-tuning. Therefore the +1.7, +2.3, +2.1, +3.1, +0.3 deltas in the abstract cannot be attributed solely to MaskCLIP++. The authors should provide a same-backbone controlled comparison, e.g., MaskCLIP++ with a ConvNeXt-L CLIP or FC-CLIP/MAFT+ with ViT-L/14, to support the headline claim.
  2. [§3.2, Appendix A.2, Table 4] The 'consistency alignment' principle is the load-bearing mechanism claimed to prevent overfitting, but its support is a two-step toy example (Appendix A.2) and an ablation (Tab. 4) run on ViT-B/16 with COCO Panoptic, while the main results use ViT-L/14 with COCO Stuff. The principle is stated only verbally; the paper does not specify the class of PSM parameterizations that satisfy it, nor does it test whether the ordering-preservation property actually holds for the chosen Linear<Em, Et> PSM during real training. Because the PSM design was selected using the ADE20K validation split, and ADE20K is also the main evaluation benchmark, the generalization of the principle to other architectures and datasets remains unestablished.
  3. [§4.3, Table 5] The claim that 'using generated masks during training hinders CLIP fine-tuning' is supported only by one comparison (Gen. masks vs GT masks) in Tab. 5. The comparison uses matched generated masks but the matching procedure is not described in detail, and the result is reported on a single validation split (ADE20K). Given that the entire method's motivation rests on this observation, the paper should report the variance across at least one other dataset or provide a more precise description of how generated masks are matched to ground-truth masks, so that the reader can assess whether the comparison is biased by the matching criterion.
minor comments (5)
  1. [Abstract] Typo: 'avaliable' should be 'available'.
  2. [§4.2, text vs Table 2] The text reports '+3.2' for PC-59, but Table 2 and the abstract report '+3.1'. Please make the numbers consistent.
  3. [§4.3, 'Impact of different mask generators'] The sentence 'the original performance of ... mask generators do do not significantly outperform' contains a duplicated 'do'. Please fix.
  4. [Table 13 and §4.2] The strong sensitivity of MaskCLIP++ to the CLIP backbone (e.g., A-150 66.0 with ViT-L/14 vs 61.0 with ConvNeXt-L in Tab. 13) is only reported in the supplementary material. Since this bears directly on the interpretation of Tab. 2, it should be discussed in the main text.
  5. [§3.2] The PSM is described informally as 'projected through an over-parameterized linear projection along a newly created orthogonal dimension'. Please give the exact mathematical form (dimensions, whether the projection is on the similarity matrix or on embeddings, and how the orthogonal dimension is constructed) in the main text or a clearly referenced equation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the claims are supported by controlled mask-classification experiments and held-out benchmarks, with only minor model-selection and comparison confounds that are not circularity.

full rationale

The paper's derivation chain is empirical rather than definitional: it observes that generated masks degrade CLIP's region alignment, proposes fine-tuning with ground-truth masks, introduces the consistency alignment principle to avoid overfitting, and then integrates the resulting mask classifier with frozen off-the-shelf mask generators. Each step is evaluated on datasets (MESS, ADE20K, Pascal Context, PASCAL VOC) that are not used to fit the mask-classification parameters; training is on COCO Stuff/Panoptic. The consistency alignment principle is supported by a toy example and ablations, not by an equation that reduces to its own assumption. The choice of the Linear<Em,Et> PSM was made using validation-set performance, but this is a model-selection concern rather than a fitted constant being renamed a prediction. The Tab. 2 headline gains compare MaskCLIP++ with ViT-L/14 against FC-CLIP and MAFT+ baselines using ConvNeXt-L, so those numbers do not isolate the proposed fine-tuning; however, this is a comparison-validity confound, not a circular reduction, and the controlled mask-classification evidence in Tab. 1 uses the same ViT-L/14 architecture for all methods. No load-bearing self-citation or imported uniqueness theorem appears; the cited works (SAN, FC-CLIP, MAFT+) are used as external baselines or design inspirations, not as the sole justification for the central claim. Therefore the paper's derivation is self-contained against external benchmarks and exhibits no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The method's performance depends on several hand-picked hyperparameters and two ad hoc assumptions: the Extractor/Fuser split and the consistency alignment principle. The latter is not derived from theory; it is supported by one toy example and by the PSM ablation. No new physical entities are invoked, only an architectural 'orthogonal dimension' in the similarity projection.

free parameters (4)
  • Fuser depth L-K = 2
    Number of transformer layers used for mask-conditioned fusion, chosen by ablation in Tab. 14a, affects the architecture and results.
  • alpha initial value = e^-5
    Controls mask strictness in Eq. (2), initialized by hand and tuned adaptively; ablation in Tab. 14b shows it matters.
  • PSM projection dimension = 768
    Over-parameterized dimension of the linear projection in PSM, set as a hyperparameter and central to the consistency alignment mechanism.
  • Ensemble weight gamma = 0.1
    Mixing weight for mask generator class scores in Eq. (3), chosen and reported in Sec. A.3; used in all main results.
assumptions (4)
  • domain assumption CLIP vision and text embeddings are pre-aligned in a shared embedding space
    Used throughout Sec. 3.1 to justify preserving similarity ordering as a meaningful constraint.
  • ad hoc to paper CLIP-V layers can be split into an Extractor and a Fuser, with spatial fusion concentrated in the later layers
    Assumed in Sec. 3.2 to justify inserting mask conditioning only in layers K+1..L; necessary for the mask embedding extraction.
  • ad hoc to paper Ground-truth masks provide the best training prior for region features
    Core observation tested in Tab. 5, but the framework depends on this assumption for its design choice.
  • ad hoc to paper Preserving CLIP's original similarity ordering prevents overfitting
    The consistency alignment principle, formalized in Sec. 3.2 and demonstrated only through the toy example and ablation in Tab. 4.
invented entities (1)
  • Orthogonal projection dimension in PSM
    purpose: Over-parameterized linear layer that projects the mask-text similarity matrix while aiming to preserve ordering
    A new architectural element proposed for consistency alignment; its benefit is shown only through validation-set ablations, not through an external falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation." pith.science (2026). https://pith.science/paper/3TOIJZDA

@misc{pith2026241211464,
  author       = {Pith},
  title        = {Pith review of: High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3TOIJZDA}},
  note         = {Machine review of arXiv:2412.11464}
}
read the original abstract

Open-vocabulary image segmentation has been advanced through the synergy between mask generators and vision-language models like Contrastive Language-Image Pre-training (CLIP). Previous approaches focus on generating masks while aligning mask features with text embeddings during training. In this paper, we observe that relying on generated low-quality masks can weaken the alignment of vision and language in regional representations. This motivates us to present a new fine-tuning framework, named MaskCLIP++, which uses ground-truth masks instead of generated masks to enhance the mask classification capability of CLIP. Due to the limited diversity of image segmentation datasets with mask annotations, we propose incorporating a consistency alignment principle during fine-tuning, which alleviates categorical bias toward the fine-tuning dataset. After low-cost fine-tuning, MaskCLIP++ significantly improves the mask classification performance on multi-domain datasets. Combining with the mask generator in previous state-of-the-art mask-based open vocabulary segmentation methods, we achieve performance improvements of +1.7, +2.3, +2.1, +3.1, and +0.3 mIoU on the A-847, PC-459, A-150, PC-59, and PAS-20 datasets, respectively. Code is avaliable at https://github.com/HVision-NKU/MaskCLIPpp .

Figures

Figures reproduced from arXiv: 2412.11464 by the authors.

Figure 1
Figure 1. Observations: (a) demonstrates the potential negative impact of low-quality generated masks on CLIP’s mask classifi￾cation learning, while (b) showcases the untapped generalization potential of existing mask generators. Mask generators are trained on the COCO [36], with results reported on the ADE20K [80]. high. Large-scale image-text pre-training models, such as CLIP [47] and ALIGN [25], have demonstrated strong ze… view at source ↗
Figure 2
Figure 2. Comparison of training pipeline between previous mask-based OVS methods. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Detailed framework of MaskCLIP++ for OVS tasks. The PSM represents the parameterized similarity modeling, which is [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization of the weight function. The regions of [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Visualizations of open-vocabulary semantic segmentation on ADE20K [ [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: A toy example: Because the inconsistency alignment between [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: More visualizations of semantic- and instance-level of open-vocabulary segmentation. We use mask generator from FC-CLIP [ [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OGGSplat: Open Gaussian Growing for Generalizable Reconstruction with Expanded Field-of-View

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A method that grows open-vocabulary 3D Gaussians beyond the input view cone by bidirectionally consistent RGB and semantic diffusion inpainting.

  2. SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM

    cs.CV 2025-11 conditional novelty 4.0 of 10

    SAM-MI improves open-vocabulary segmentation by injecting aggregated SAM masks as low- and high-frequency guidance into CLIP cost maps, with sparse text-guided point prompts for speed.

Reference graph

Works this paper leans on

83 extracted references · 66 canonical work pages · cited by 2 Pith papers

  1. [1]

    Fossil: Free open-vocabulary semantic seg- mentation through synthetic references retrieval

    Luca Barsellotti, Roberto Amoroso, Lorenzo Baraldi, and Rita Cucchiara. Fossil: Free open-vocabulary semantic seg- mentation through synthetic references retrieval. In WACV, pages 1464–1473, 2024. 2

    2024

  2. [2]

    Zerowaste dataset: To- wards deformable object segmentation in cluttered scenes

    Dina Bashkirova, Mohamed Abdelfattah, Ziliang Zhu, James Akl, Fadi Alladkani, Ping Hu, Vitaly Ablavsky, Berk Calli, Sarah Adel Bargal, and Kate Saenko. Zerowaste dataset: To- wards deformable object segmentation in cluttered scenes. In CVPR, pages 21147–21157, 2022. 12

    2022

  3. [3]

    Corrosion condition state semantic segmentation dataset

    Eric Bianchi and Matthew Hebdon. Corrosion condition state semantic segmentation dataset. University Libraries, Virginia Tech: Blacksburg, VA, USA, 3:2, 2021. 12

    2021

  4. [4]

    What a MESS: Multi-Domain Evaluation of Zero-shot Semantic Segmentation

    Benedikt Blumenstiel, Johannes Jakubik, Hilde K ¨uhne, and Michael V¨ossing. What a MESS: Multi-Domain Evaluation of Zero-shot Semantic Segmentation. NeurIPS, 2023. 5, 6, 13, 14

    work page 2023

  5. [5]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In CVPR, pages 1209–1218, 2018. 5, 6, 12

    work page 2018

  6. [6]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In CVPR, pages 11165–11174, 2023. 2

    work page 2023

  7. [7]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In CVPR, pages 3558–3568, 2021. 2

    work page 2021

  8. [8]

    Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only

    Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Sean Chang Cu- latana, and Mohamed Elhoseiny. Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only. In ICCV, pages 699–710, 2023. 2

    work page 2023

Show all 83 references
  1. [9]

    Per- pixel classification is not all you need for semantic segmen- tation

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per- pixel classification is not all you need for semantic segmen- tation. NeurIPS, 34:17864–17875, 2021. 1, 2

    2021

  2. [10]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, pages 1290–1299, 2022. 1, 2, 7

    2022

  3. [11]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmenta- tion

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmenta- tion. In CVPR, pages 4113–4123, 2024. 2, 5, 6, 8

    2024

  4. [12]

    Semantic segmentation in art paintings

    Nadav Cohen, Yael Newman, and Ariel Shamir. Semantic segmentation in art paintings. In Computer graphics forum, pages 261–275. Wiley Online Library, 2022. 12

    2022

  5. [13]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213–3223, 2016. 12

    2016

  6. [14]

    De- coupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. In CVPR, pages 11583–11592, 2022. 2

    2022

  7. [15]

    Open- vocabulary universal image segmentation with maskclip

    Zheng Ding, Jieke Wang, and Zhuowen Tu. Open- vocabulary universal image segmentation with maskclip. In ICML. JMLR.org, 2023. 1, 2, 3, 6

    2023

  8. [16]

    Atlantis: A benchmark for semantic segmentation of waterbody images

    Seyed Mohammad Hassan Erfani, Zhenyao Wu, Xinyi Wu, Song Wang, and Erfan Goharian. Atlantis: A benchmark for semantic segmentation of waterbody images. Environmental Modelling & Software, 149:105333, 2022. 12

    2022

  9. [17]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88:303–338, 2010. 5, 12

    2010

  10. [18]

    An ensemble classification-based approach applied to retinal blood vessel segmentation

    Muhammad Moazam Fraz, Paolo Remagnino, Andreas Hoppe, Bunyarit Uyyanonvara, Alicja R Rudnicka, Christo- pher G Owen, and Sarah A Barman. An ensemble classification-based approach applied to retinal blood vessel segmentation. IEEE Transactions on Biomedical Engineer- ing, 59(9)...

    2012

  11. [19]

    Scal- ing open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scal- ing open-vocabulary image segmentation with image-level labels. In ECCV, pages 540–557. Springer, 2022. 2

    2022

  12. [20]

    Open-vocabulary semantic segmentation with decou- pled one-pass network

    Cong Han, Yujie Zhong, Dengjie Li, Kai Han, and Lin Ma. Open-vocabulary semantic segmentation with decou- pled one-pass network. In ICCV, pages 1086–1096, 2023. 2

    2023

  13. [21]

    A crop/weed field im- age dataset for the evaluation of computer vision based pre- cision agriculture tasks

    Sebastian Haug and J ¨orn Ostermann. A crop/weed field im- age dataset for the evaluation of computer vision based pre- cision agriculture tasks. In Computer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6-7 and 12, 2014, Proceedings, Part IV 13 , pages 105–116. Springer,

    2014

  14. [22]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In ICCV, pages 2961–2969, 2017. 2

    2017

  15. [23]

    Semantic segmentation of underwater im- agery: Dataset and benchmark

    Md Jahidul Islam, Chelsey Edge, Yuyang Xiao, Peigen Luo, Muntaqim Mehtaz, Christopher Morse, Sadman Sakib Enan, and Junaed Sattar. Semantic segmentation of underwater im- agery: Dataset and benchmark. In 2020 IEEE/RSJ Interna- tional Conference on Intelligent Robots and System...

    2020

  16. [24]

    Floods of the world: quantitative analysis of natural characteristics and parame- ters of social-economic damages

    MN Istomina and SG Dobrovoski. Floods of the world: quantitative analysis of natural characteristics and parame- ters of social-economic damages. Water resources, 43:459– 471, 2016. 12

    2016

  17. [25]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904–

  18. [26]

    Learning mask-aware clip representations for zero-shot segmentation

    Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao, and Humphrey Shi. Learning mask-aware clip representations for zero-shot segmentation. NeurIPS, 36:35631–35653,

  19. [27]

    Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation

    Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yun- chao Wei, and Humphrey Shi. Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation. In ECCV, pages 399–416. Springer, 2024. 1, 2, 3, 5, 6, 8

    2024

  20. [28]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, pages 4015–4026, 2023. 2 9

    2023

  21. [29]

    Clearclip: Decom- posing clip representations for dense vision-language infer- ence

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decom- posing clip representations for dense vision-language infer- ence. In ECCV, pages 143–160. Springer, 2024. 2, 8

    2024

  22. [30]

    Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation. In ECCV, pages 70–88. Springer, 2024. 2, 8

    2024

  23. [31]

    Language-driven semantic seg- mentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic seg- mentation. In ICLR, 2022. 2

    2022

  24. [32]

    Multiple- human parsing in the wild.arXiv preprint arXiv:1705.07206,

    Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, Terence Sim, Shuicheng Yan, and Jiashi Feng. Multiple- human parsing in the wild.arXiv preprint arXiv:1705.07206,

    arXiv

  25. [33]

    Clip surgery for better explainability with enhancement in open- vocabulary tasks

    Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open- vocabulary tasks. arXiv preprint arXiv:2304.05653 , 2023. 2

    2023 arXiv

  26. [34]

    Cascade-clip: Cascaded vision-language embeddings alignment for zero-shot semantic segmentation

    Yunheng Li, ZhongYu Li, Quansheng Zeng, Qibin Hou, and Ming-Ming Cheng. Cascade-clip: Cascaded vision-language embeddings alignment for zero-shot semantic segmentation. In ICML, 2024. 2

    2024

  27. [35]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, pages 7061–7070, 2023. 2, 6

    2023

  28. [36]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 1, 2, 6, 7, 12, 13, 14, 15

    2014

  29. [37]

    Deepcrack: A deep hierarchical feature learning architec- ture for crack segmentation

    Yahui Liu, Jian Yao, Xiaohu Lu, Renping Xie, and Li Li. Deepcrack: A deep hierarchical feature learning architec- ture for crack segmentation. Neurocomputing, 338:139–153,

  30. [38]

    Open-vocabulary segmentation with semantic-assisted calibration

    Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. In CVPR, pages 3491–3500, 2024. 2, 6

    2024

  31. [39]

    Multi- grained cross-modal alignment for learning open-vocabulary semantic segmentation from text supervision

    Yajie Liu, Pu Ge, Qingjie Liu, and Di Huang. Multi- grained cross-modal alignment for learning open-vocabulary semantic segmentation from text supervision. arXiv preprint arXiv:2403.03707, 2024. 2

    2024 arXiv

  32. [40]

    Segclip: Patch aggregation with learn- able centers for open-vocabulary semantic segmentation

    Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segclip: Patch aggregation with learn- able centers for open-vocabulary semantic segmentation. In ICML, pages 23033–23044. PMLR, 2023. 2

    2023

  33. [41]

    Uavid: A semantic segmentation dataset for uav imagery

    Ye Lyu, George V osselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery. ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020. 12

    2020

  34. [42]

    Cryonuseg: A dataset for nuclei instance segmenta- tion of cryosectioned h&e-stained histological images.Com- puters in biology and medicine, 132:104349, 2021

    Amirreza Mahbod, Gerald Schaefer, Benjamin Bancher, Christine L ¨ow, Georg Dorffner, Rupert Ecker, and Isabella Ellinger. Cryonuseg: A dataset for nuclei instance segmenta- tion of cryosectioned h&e-stained histological images.Com- puters in biology and medicine, 132:104349, 2021. 12

    2021

  35. [43]

    The role of context for object detection and se- mantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and se- mantic segmentation in the wild. In CVPR, pages 891–898,

  36. [44]

    Open vocabulary semantic segmentation with patch aligned con- trastive learning

    Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned con- trastive learning. In CVPR, pages 19413–19423, 2023. 2

    2023

  37. [45]

    Kvasir: A multi- class image dataset for computer aided gastrointestinal dis- ease detection

    Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Gri- wodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, et al. Kvasir: A multi- class image dataset for computer aided gastrointestinal...

    2017

  38. [46]

    Freeseg: Unified, universal and open-vocabulary im- age segmentation

    Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xue- feng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al. Freeseg: Unified, universal and open-vocabulary im- age segmentation. In CVPR, pages 19446–19455, 2023. 2

    2023

  39. [47]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 1, 6

    2021

  40. [48]

    Floodnet: A high resolution aerial imagery dataset for post flood scene understanding

    Maryam Rahnemoonfar, Tashnim Chowdhury, Argho Sarkar, Debvrat Varshney, Masoud Yari, and Robin Rober- son Murphy. Floodnet: A high resolution aerial imagery dataset for post flood scene understanding. IEEE Access , 9:89644–89654, 2021. 12

    2021

  41. [49]

    Guided curriculum model adaptation and uncertainty-aware evalua- tion for semantic nighttime image segmentation

    Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided curriculum model adaptation and uncertainty-aware evalua- tion for semantic nighttime image segmentation. In CVPR, pages 7374–7383, 2019. 12

    2019

  42. [50]

    De- tailed annotations of chest x-rays via ct projection for report understanding

    Constantin Seibold, Simon Reiß, Saquib Sarfraz, Matthias A Fink, Victoria Mayer, Jan Sellner, Moon Sung Kim, Klaus H Maier-Hein, Jens Kleesiek, and Rainer Stiefelhagen. De- tailed annotations of chest x-rays via ct projection for report understanding. arXiv preprint arXiv:2210...

    arXiv

  43. [51]

    Aligning and prompting everything all at once for univer- sal visual perception

    Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for univer- sal visual perception. In CVPR, pages 13193–13203, 2024. 2

    2024

  44. [52]

    Reco: Re- trieve and co-segment for zero-shot transfer

    Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Re- trieve and co-segment for zero-shot transfer. NeurIPS, 35: 33754–33767, 2022. 2

    2022

  45. [53]

    Pst900: Rgb- thermal calibration, dataset and segmentation network

    Shreyas S Shivakumar, Neil Rodrigues, Alex Zhou, Ian D Miller, Vijay Kumar, and Camillo J Taylor. Pst900: Rgb- thermal calibration, dataset and segmentation network. In 2020 IEEE international conference on robotics and au- tomation (ICRA), pages 9441–9447. IEEE, 2020. 12

    2020

  46. [54]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 5, 6

    2023 arXiv

  47. [55]

    Clip as rnn: Segment countless visual concepts without training endeavor

    Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip as rnn: Segment countless visual concepts without training endeavor. In CVPR, pages 13171–13182, 2024. 2 10

    2024

  48. [56]

    Isprs potsdam dataset within the isprs test project on urban classification, 3d building reconstruction and semantic labeling, 2012

    BSF Swissphoto. Isprs potsdam dataset within the isprs test project on urban classification, 3d building reconstruction and semantic labeling, 2012. 12

    2012

  49. [57]

    Yfcc100m: The new data in multimedia research

    Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. 2

    2016

  50. [58]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Per- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 12, 13

    2011

  51. [59]

    Sclip: Rethink- ing self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethink- ing self-attention for dense vision-language inference. In European Conference on Computer Vision, pages 315–332. Springer, 2024. 2, 8

    2024

  52. [60]

    Max-deeplab: End-to-end panoptic seg- mentation with mask transformers

    Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic seg- mentation with mask transformers. In CVPR, pages 5463– 5474, 2021. 2

    2021

  53. [61]

    Sam-clip: Merging vision foundation models to- wards semantic and spatial understanding

    Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models to- wards semantic and spatial understanding. In CVPR, pages 3635...

    2024

  54. [62]

    Solo: Segmenting objects by locations

    Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In ECCV, pages 649–665. Springer, 2020. 2

    2020

  55. [63]

    Open-vocabulary segmentation with unpaired mask-text supervision

    Zhaoqing Wang, Xiaobo Xia, Ziye Chen, Xiao He, Yandong Guo, Mingming Gong, and Tongliang Liu. Open-vocabulary segmentation with unpaired mask-text supervision. arXiv preprint arXiv:2402.08960, 2024. 2

    2024 arXiv

  56. [64]

    isaid: A large- scale dataset for instance segmentation in aerial images

    Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. isaid: A large- scale dataset for instance segmentation in aerial images. In CVPRW, pages 28–37, 2019. 12

    2019

  57. [65]

    CLIPSelf: Vision trans- former distills itself for open-vocabulary dense prediction

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. CLIPSelf: Vision trans- former distills itself for open-vocabulary dense prediction. In ICLR, 2024. 2, 5, 6

    2024

  58. [66]

    A large-scale benchmark for food im- age segmentation

    Xiongwei Wu, Xin Fu, Ying Liu, Ee-Peng Lim, Steven CH Hoi, and Qianru Sun. A large-scale benchmark for food im- age segmentation. In Proceedings of the 29th ACM interna- tional conference on multimedia, pages 506–515, 2021. 12

    2021

  59. [67]

    Detectron2

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github. com/facebookresearch/detectron2, 2019. 5

    2019

  60. [68]

    Sed: A simple encoder-decoder for open- vocabulary semantic segmentation

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open- vocabulary semantic segmentation. In CVPR, pages 3426– 3436, 2024. 2, 6

    2024

  61. [69]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, pages 18134–18144, 2022. 2

    2022

  62. [70]

    Learning open-vocabulary semantic segmentation models from natural language supervision

    Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocabulary semantic segmentation models from natural language supervision. In CVPR, pages 2935–2944, 2023. 2

    2023

  63. [71]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In CVPR, pages 2955–2966, 2023. 2, 5, 6, 12

    2023

  64. [72]

    A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model. In ECCV, pages 736–753. Springer, 2022. 2, 3, 12

    2022

  65. [73]

    San: Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. San: Side adapter network for open-vocabulary semantic segmentation. IEEE TPAMI, 45(12):15546–15561,

  66. [74]

    Masq- clip for open-vocabulary universal image segmentation

    Xin Xu, Tianyi Xiong, Zheng Ding, and Zhuowen Tu. Masq- clip for open-vocabulary universal image segmentation. In ICCV, pages 887–898, 2023. 2, 6

    2023

  67. [75]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Dar- rell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, pages 2636–2645, 2020. 12

    2020

  68. [76]

    Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. NeurIPS, 36, 2024. 1, 2, 3, 5, 6, 7, 12, 15

    2024

  69. [77]

    Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively

    Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively. arXiv preprint arXiv:2401.02955, 2024. 2

    2024 arXiv

  70. [78]

    Uncovering prototypical knowledge for weakly open- vocabulary semantic segmentation

    Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Uncovering prototypical knowledge for weakly open- vocabulary semantic segmentation. NeurIPS, 36:73652– 73665, 2023. 2

    2023

  71. [79]

    K-net: Towards unified image seg- mentation

    Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image seg- mentation. NeurIPS, 34:10326–10338, 2021. 2

    2021

  72. [80]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633–641, 2017. 1, 2, 5, 7, 8, 12, 15

    2017

  73. [81]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, pages 696–712. Springer,

  74. [82]

    Zegclip: Towards adapting clip for zero-shot se- mantic segmentation

    Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot se- mantic segmentation. In CVPR, pages 11175–11185, 2023. 2

    2023

  75. [83]

    High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In CVPR, pages 15116–15127, 2023. 2 11 Supplementary Material for “High-Quality Mask Tuning Matt...

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.