Pith. sign in

REVIEW 3 cited by

ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.04883 v1 pith:JXILNYDP submitted 2024-08-09 cs.CV

Mengcheng Lan , Chaofeng Chen , Yiping Ke , Xinjiang Wang , Litong Feng , Wayne Zhang This is my paper · ORCID
classification cs.CV
keywords vfmsclipopen-vocabularysemanticproxyclipsegmentationattentionmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Open-vocabulary semantic segmentation requires models to effectively integrate visual representations with open-vocabulary semantic labels. While Contrastive Language-Image Pre-training (CLIP) models shine in recognizing visual concepts from text, they often struggle with segment coherence due to their limited localization ability. In contrast, Vision Foundation Models (VFMs) excel at acquiring spatially consistent local visual representations, yet they fall short in semantic understanding. This paper introduces ProxyCLIP, an innovative framework designed to harmonize the strengths of both CLIP and VFMs, facilitating enhanced open-vocabulary semantic segmentation. ProxyCLIP leverages the spatial feature correspondence from VFMs as a form of proxy attention to augment CLIP, thereby inheriting the VFMs' robust local consistency and maintaining CLIP's exceptional zero-shot transfer capacity. We propose an adaptive normalization and masking strategy to get the proxy attention from VFMs, allowing for adaptation across different VFMs. Remarkably, as a training-free approach, ProxyCLIP significantly improves the average mean Intersection over Union (mIoU) across eight benchmarks from 40.3 to 44.4, showcasing its exceptional efficacy in bridging the gap between spatial precision and semantic richness for the open-vocabulary segmentation task.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A feedback self-adaptive attention module uses CLIP's own output predictions as a spatial coherence prior to reweight intermediate attention, improving training-free open-vocabulary segmentation across 8 benchmarks.

  2. Bridge Feature Matching and Cross-Modal Alignment with Mutual-filtering for Zero-shot Anomaly Detection

    cs.CV 2025-07 conditional novelty 6.0 of 10

    FiSeCLIP achieves state-of-the-art zero-shot anomaly detection by using a batch of test images as mutual references and filtering noisy features with text-guided masks, without any training.

  3. FIX-CLIP: Dual-Branch Hierarchical Contrastive Learning via Synthetic Captions for Better Understanding of Long Text

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A dual-branch CLIP training pipeline with regional prompts and hierarchical feature alignment reaches state-of-the-art on long- and short-text retrieval.

Pith tools