Pith. sign in

REVIEW 3 major objections 4 minor 63 references

ResCLIP: Residual Attention for Training-free Dense Vision-language Inference

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ResCLIP claims that the cross-correlation (query-key) attention maps in CLIP's intermediate layers already carry class-specific localization, and that averaging them into the final layer's attention plus a segmentation-map feedback loop…

desk verdict Useful training-free attention fix, but the headline gains are tuned on the test sets and the priority claim overreaches. read the letter →

arxiv 2411.15851 v2 pith:KJYAICR3 submitted 2024-11-24 cs.CV

Yuhang Yang , Jinhong Deng , Wen Li , Lixin Duan This is my paper · ORCID
classification cs.CV
keywords open-vocabularysemanticsegmentationtraining-freeinferenceCLIPattentioncross-correlationself-attentionself-correlationresidualfeedbackrefinementdensevision-language
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ResCLIP is built on a single observation: inside CLIP's vision transformer, the standard query-key (cross-correlation) attention maps in the non-final layers are not spatially uniform — they concentrate on regions of the same class, e.g. a 'cat' patch attending to other cat patches. The paper treats the final layer's diffuse, spatial-invariant attention as a defect that can be repaired by borrowing those earlier maps. Concretely, it averages the intermediate cross-correlation attention matrices and adds them as a residual term to the last block's attention, then runs a refinement loop that uses the resulting segmentation map to suppress attention to patches of other predicted classes while preserving local consistency. Reported mIoU improves consistently when the module is dropped into SCLIP, ClearCLIP, and NACLIP over eight open-vocabulary segmentation benchmarks, with the largest boosts on the ViT-L/14 encoder. The stakes are that CLIP's poor dense prediction is not a missing capability but a discard problem: the localization information is already present in its own intermediate layers and can be recovered without any training.

What carries the argument

Residual Cross-correlation Self-attention (RCS) plus Semantic Feedback Refinement (SFR). RCS averages the query-key attention maps from non-final layers (Eq. 5) and mixes them with the self-correlation attention used by previous training-free methods (Eq. 6, weight $\lambda_{\text{rcs}}$). SFR takes the segmentation map produced by that mixed attention, builds a binary mask $S_{m,n}^{i}=\mathbb{I}(M_{i',j'}=M_{m,n})$ that keeps only same-class patches, applies a connectivity-aware Chebyshev-distance decay $h(V,D)$ and a 1-D Gaussian smoother to obtain $\hat{S}$, and blends it with the self-correlation score under weight $\lambda_{\text{sfr}}$ (Eq. 12); Eq. 13 then combines the refined attention with the intermediate cross-correlation average. The mechanism works by re-introducing spatial correspondence that the last block has smeared out, and by using the model's own coarse prediction as a consistency prior.

What would settle it

Run ResCLIP on a fixed test set with the semantic feedback mask $\mathbf{M}$ in Eq. (7) replaced by a randomly permuted version of itself, keeping the RCS term and all hyperparameters unchanged. If mIoU stays near the full ResCLIP level, the SFR gain is not caused by semantic feedback; if it drops to or below the RCS-only baseline on the same images, the feedback loop is load-bearing as claimed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the cross-correlation (query-key) self-attention of CLIP's intermediate layers exhibits localization properties, and that feeding this information forward as a residual correction heals the last layer's spatial-invariant attention. The final attention is assembled as $A_{\text{ResCLIP}}=(1-\lambda_{\text{rcs}})A_{\text{sfr}}+\lambda_{\text{rcs}}A_c$, where $A_c$ is the average of query-key attention matrices from a chosen range of intermediate layers and $A_{\text{sfr}}$ is the softmax of a score $S_r$ that blends self-correlation attention with a semantically refined score $\hat{S}$ derived from an initial segmentation map (Eqs. 5-13). The authors report that this training-free module, applied on top of SCLIP, ClearCLIP, and NACLIP, yields consistent mIoU gains across eight benchmarks and substantially mitigates the ViT-L/14 degradation of prior methods, attributing the gains to the reorganized spatial correspondence and to same-class, local-consistent attention enforced by the feedback loop.

Load-bearing premise

The feedback loop assumes the initial segmentation map produced by the RCS-treated attention is accurate enough that zeroing out attention to patches with a different predicted class suppresses noise rather than suppressing correct correspondences; if that first map is systematically wrong, the error is baked into the refinement.

Editorial extensions

If this is right

  • Existing training-free open-vocabulary segmenters can be upgraded by a plug-in module with no retraining, so future self-correlation attention (SCSA) methods should report results with and without this residual correction.
  • The ViT-L/14 performance cliff in prior training-free methods is substantially reduced: ResCLIP reports an average +13.1 mIoU over SCLIP on the five no-background datasets, indicating the last-layer attention deficiency is the main culprit.
  • The localization property transfers across CLIP checkpoints and backbones (ViT-B/16, ViT-B/32, ViT-L/14) and to OpenCLIP, so the residual-correction recipe is not architecture-specific within the ViT family.
  • RCS and SFR are complementary: on the NACLIP baseline, RCS alone adds +5.8 mIoU and SFR alone +1.8 mIoU on VOC20, while the combination adds +6.3, implying the two modules target different failure modes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same recipe of recruiting earlier layers to heal a saturated final layer could be tested in other transformer stacks (text, audio, or other vision models) wherever the deepest attention map becomes spatially or temporally diffuse; the paper only demonstrates it for CLIP.
  • Editorial inference: SFR is a self-confirmation loop — it strengthens attention to patches whose initial class already agrees with the query patch. For small, thin, or ambiguous objects where the initial map is wrong, the process cannot recover those correspondences, and the reported aggregate gains may hide a concentration on large, confident regions.
  • Editorial inference: the scalar blending weights (about 0.5 for RCS and 0.7 for SFR) suggest an ensemble view of the two attention streams; a learnable per-pixel weighting could outperform the fixed convex combination, though it would no longer be training-free.
  • Editorial inference: because the method only needs attention maps and a segmentation mask, it could be combined with the heavier post-processing steps the paper deliberately avoids for fair comparison; whether the gains are additive or redundant is untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. ResCLIP proposes a training-free plug-in module for open-vocabulary semantic segmentation with CLIP. It consists of Residual Cross-correlation Self-attention (RCS), which averages query-key (cross-correlation) attention from intermediate CLIP layers and fuses it with the last-layer self-correlation attention used by SCLIP/ClearCLIP/NACLIP, and Semantic Feedback Refinement (SFR), which reweights attention using the initial segmentation map so that patches attend mainly to same-class patches with a distance-based decay. Experiments on eight benchmarks report consistent mIoU gains when RCS and SFR are added to SCLIP, ClearCLIP, and NACLIP, with the largest absolute gains on ViT-L/14.

Significance. The observation that intermediate-layer cross-correlation attention retains localization properties and can be used as a residual signal to heal the spatial-invariant last-layer attention is plausible and, if confirmed, useful: ResCLIP is drop-in, training-free, and the code is released. The two modules are clearly separated, and the ablations in Table 3 and Table A1 suggest both contribute. However, the empirical claim of consistent plug-and-play gains is currently weakened by hyperparameter selection on the same test benchmarks, the self-referential nature of the SFR loop, and the absence of error bars or a held-out validation protocol. These issues are fixable and do not invalidate the core idea, but they must be addressed before the central claim is fully established.

major comments (3)
  1. [Sec. 4.3, Fig. 6, Table 4] The four tunable choices of ResCLIP, namely lambda_rcs, lambda_sfr, the layer range (s,e) in Eq. (5), and the sliding-window choice in Table 4, are all selected by maximizing mIoU on the same benchmarks (VOC20, Object) that are then reported in Tables 1 and 2. Because the method is training-free, these selections effectively serve as the learning signal, and reported gains may be inflated by test-set fitting. Please provide a fixed-hyperparameter evaluation protocol, e.g., tune on one dataset or backbone and evaluate on all others, and report the variance (or seed-to-seed spread) of the mIoU numbers. Without this, the central claim of consistent out-of-the-box gains is not yet quantitatively supported.
  2. [Sec. 3.2.2, Eq. (7)] The SFR module is self-referential: the indicator matrix in Eq. (7) is built from the segmentation map M obtained via Eq. (4) from the RCS output, and Eq. (13) then recomputes the final prediction from the refined attention. Any systematic error in M is inherited by the refinement and can be amplified by the feedback loop. The paper does not analyze this failure mode or quantify how sensitive the final mIoU is to errors in M. A concrete test would be to perturb M (e.g., randomly flip a fraction of the predicted labels) and measure the effect on the final prediction; adding such an analysis would substantially strengthen the SFR claim in Table 3.
  3. [Tables 1, 2, A2] The reported gains are not accompanied by error bars or by a statement of how many evaluation runs are averaged, and the magnitude of the gain varies widely across configurations (e.g., +13.1 mIoU for SCLIP ViT-L/14 on VOC20 in Table 1 versus +0.5 mIoU for ClearCLIP on OpenCLIP in Table A2). Given that the hyperparameters are selected on the same test benchmarks, these numbers alone do not yet support the conclusion that ResCLIP yields consistent improvements across all settings. Please specify the exact hyperparameter configuration used for each row of Tables 1–2 and provide per-dataset confidence intervals or at least multiple independent runs, especially for the ViT-L/14 results.
minor comments (4)
  1. [Sec. 4.1] The first paragraph lists 'datasets with background category' twice; the second group should read 'datasets without a background class,' and the sentence 'We perform slide inference' should be 'We perform sliding-window inference.'
  2. [Fig. 6] The figure axes are not labeled, so the reader cannot tell which curve corresponds to lambda_rcs and which to lambda_sfr, or which dataset each panel uses.
  3. [Eq. (7)] The notation for the attention row S_i is ambiguous: i indexes the query patch, but the definition i' = floor(i/w), j' = (i mod w) assumes a row-major patch index without the cls token, which is inconsistent with the earlier use of attention matrices of dimension (1+hw) x (1+hw). Clarify how the cls token is handled in the SFR mask.
  4. [Fig. A1 caption] The caption contains a typo: 'SCLP' should be 'SCLIP'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: RCS is a forward attention-fusion operation and SFR is an explicit self-feedback loop, not a by-construction reduction of the output to the input.

full rationale

The claimed derivation chain is RCS (Eqs. 5-6), SFR (Eqs. 7-12), and the final combination (Eq. 13). RCS averages cross-correlation attention from intermediate layers and mixes it with the baseline SCSA attention; this is a forward computation with no parameter fitted to the target metric and no load-bearing self-citation. SFR builds a mask from the model's own initial segmentation map M and uses it to reweight attention; although this is self-referential in the sense of feeding the model's own prediction back into the attention computation, it is not circular by construction: the final prediction is recomputed from the refined attention and is not defined to equal M, and the paper explicitly presents this as a refinement loop rather than as an independent first-principles derivation. The hyperparameters lambda_rcs, lambda_sfr, and the layer range are chosen empirically, and selecting them on the test benchmarks is an evaluation-validity concern, not a logical reduction of the method to its inputs. No self-citation is load-bearing; the cited baselines are external works. Therefore, no step in the derivation reduces by definition or by fit to the claim being tested.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or architectural entities, but it rests on two empirical observations (intermediate-layer localization and trustworthy initial segmentation) plus four tunable choices. Two continuous hyperparameters and the layer range are selected on target benchmarks, and the smoothing kernel is unspecified, so the reported gains are not fully parameter-free.

free parameters (4)
  • lambda_rcs = 0.5
    Trade-off weight between SCSA attention and averaged intermediate C2SA in Eq. (6)/(13); selected by sensitivity analysis on the same benchmarks used for reporting (Fig. 6).
  • lambda_sfr = 0.7
    Trade-off weight between SCSA attention and semantic-feedback-refined attention in Eq. (12); selected on the same benchmarks, with stable range 0.6 to 0.8.
  • layer aggregation range (s,e) = s=6, e=9
    Sliding-window aggregation was chosen from Table 4 based on best VOC20 and Object mIoU, with no separate validation split described.
  • smoothing kernel phi
    A 1-dimensional Gaussian kernel in Eq. (11) is mentioned but its bandwidth and kernel size are not specified, an undocumented tunable.
assumptions (4)
  • domain assumption CLIP's final-layer query-key attention is spatially invariant and harms dense prediction.
    Invoked in Sections 1 and 3.2 and attributed to prior works; the whole method is built on repairing this defect.
  • domain assumption Self-correlation attention (query-query and key-key variants) provides spatial-covariant features.
    Used as the SCSA component As in Eqs. (6) and (13), following SCLIP, ClearCLIP, and NACLIP.
  • ad hoc to paper Intermediate-layer cross-correlation attention has class-specific localization properties.
    Asserted from Fig. 1 visualizations and used in Eq. (5); not quantified or formally verified, and GEM already touches on localization properties.
  • ad hoc to paper The initial RCS-produced segmentation map M is reliable enough to serve as semantic feedback.
    SFR Eq. (7) assumes M's labels are trustworthy for refining attention; if M is wrong, the loop amplifies the error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ResCLIP: Residual Attention for Training-free Dense Vision-language Inference." pith.science (2026). https://pith.science/paper/KJYAICR3

@misc{pith2026241115851,
  author       = {Pith},
  title        = {Pith review of: ResCLIP: Residual Attention for Training-free Dense Vision-language Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJYAICR3}},
  note         = {Machine review of arXiv:2411.15851}
}
read the original abstract

While vision-language models like CLIP have shown remarkable success in open-vocabulary tasks, their application is currently confined to image-level tasks, and they still struggle with dense predictions. Recent works often attribute such deficiency in dense predictions to the self-attention layers in the final block, and have achieved commendable results by modifying the original query-key attention to self-correlation attention, (e.g., query-query and key-key attention). However, these methods overlook the cross-correlation attention (query-key) properties, which capture the rich spatial correspondence. In this paper, we reveal that the cross-correlation of the self-attention in CLIP's non-final layers also exhibits localization properties. Therefore, we propose the Residual Cross-correlation Self-attention (RCS) module, which leverages the cross-correlation self-attention from intermediate layers to remold the attention in the final block. The RCS module effectively reorganizes spatial information, unleashing the localization potential within CLIP for dense vision-language inference. Furthermore, to enhance the focus on regions of the same categories and local consistency, we propose the Semantic Feedback Refinement (SFR) module, which utilizes semantic segmentation maps to further adjust the attention scores. By integrating these two strategies, our method, termed ResCLIP, can be easily incorporated into existing approaches as a plug-and-play module, significantly boosting their performance in dense vision-language inference. Extensive experiments across multiple standard benchmarks demonstrate that our method surpasses state-of-the-art training-free methods, validating the effectiveness of the proposed approach. Code is available at https://github.com/yvhangyang/ResCLIP.

Figures

Figures reproduced from arXiv: 2411.15851 by the authors.

Figure 1
Figure 1. The attention visualization from different layers of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Cross-correlation self-attention (C2 SA). The query and key are mapped by different project matrices. The attention is obtained by matrix multiplication between query and key. (b) Self-correlation self-attention (SCSA). The attention is calculated by the self-correlation such as key-key or query-query. (c) Residual Cross-correlation Self-attention (RCS) and Semantic Feedback Refinement (SFR). (d) The performance… view at source ↗
Figure 3
Figure 3. Comparison of attention maps across different versions [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Overview of our ResCLIP consisting of Residual Cross-correlation Self-attention (RCS) and Semantic Feedback Refinement [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Analysis of hyper-parameters λrcs and λsfr [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 40 canonical work pages

  1. [1]

    Self- supervised multimodal versatile networks

    Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovi´c, Jason Ramapuram, Jeffrey De Fauw, Lu- cas Smaira, Sander Dieleman, and Andrew Zisserman. Self- supervised multimodal versatile networks. NeurIPS, 33:25– 37, 2020. 3

    work page 2020

  2. [3]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InICCV, pages 2425–2433,

    work page

  3. [4]

    Single-stage semantic segmentation from image labels

    Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In CVPR, pages 4253– 4262, 2020. 7

    work page 2020

  4. [5]

    Fossil: Free open-vocabulary semantic seg- mentation through synthetic references retrieval

    Luca Barsellotti, Roberto Amoroso, Lorenzo Baraldi, and Rita Cucchiara. Fossil: Free open-vocabulary semantic seg- mentation through synthetic references retrieval. In WACV, pages 1464–1473, 2024. 6

    work page 2024

  5. [6]

    Grounding everything: Emerging localiza- tion properties in vision-language transformers

    Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding everything: Emerging localiza- tion properties in vision-language transformers. In CVPR, pages 3828–3837, 2024. 3, 6, 7

    work page 2024

  6. [7]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 1

    arXiv 2005

  7. [8]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In CVPR, pages 1209–1218, 2018. 6, 14, 16

    work page 2018

  8. [9]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, pages 9650–9660, 2021. 1

    work page 2021

Show all 63 references
  1. [10]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In CVPR, pages 11165–11174, 2023. 1, 3, 6, 7

    2023

  2. [11]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In CVPR, pages 2818–2829, 2023. 3, 12, 14

    2023

  3. [12]

    Mmsegmentation: Open- mmlab semantic segmentation toolbox and benchmark,

    MMSegmentation Contributors. Mmsegmentation: Open- mmlab semantic segmentation toolbox and benchmark,

  4. [13]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213–3223, 2016. 6

    2016

  5. [14]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Pi- otr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023. 1

    2023 arXiv

  6. [15]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 1

    2018 arXiv

  7. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 7

    2010 arXiv

  8. [17]

    The pascal visual object classes challenge: A retrospective.IJCV, 111:98–136, 2015

    Mark Everingham, SM Ali Eslami, Luc Van Gool, Christo- pher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective.IJCV, 111:98–136, 2015. 1, 6, 14, 17

    2015

  9. [18]

    Improved base- lines for vision-language pre-training

    Enrico Fini, Pietro Astolfi, Adriana Romero-Soriano, Jakob Verbeek, and Michal Drozdzal. Improved base- lines for vision-language pre-training. arXiv preprint arXiv:2305.08675, 2023. 3

    2023 arXiv

  10. [19]

    Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation

    Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation. In WACV, 2025. 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14

    2025

  11. [20]

    Open-vocabulary semantic segmentation with decou- pled one-pass network

    Cong Han, Yujie Zhong, Dengjie Li, Kai Han, and Lin Ma. Open-vocabulary semantic segmentation with decou- pled one-pass network. In ICCV, pages 1086–1096, 2023. 3

    2023

  12. [21]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904–

  13. [22]

    Learning mask-aware clip representations for zero-shot segmentation

    Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao, and Humphrey Shi. Learning mask-aware clip representations for zero-shot segmentation. NeurIPS, 36:35631–35653,

  14. [23]

    In defense of lazy vi- sual grounding for open-vocabulary semantic segmentation

    Dahyun Kang and Minsu Cho. In defense of lazy vi- sual grounding for open-vocabulary semantic segmentation. arXiv preprint arXiv:2408.04961, 2024. 3

    2024 arXiv

  15. [24]

    Weakly supervised ground- ing for vqa in vision-language transformers

    Aisha Urooj Khan, Hilde Kuehne, Chuang Gan, Niels Da Vi- toria Lobo, and Mubarak Shah. Weakly supervised ground- ing for vqa in vision-language transformers. In ECCV, pages 652–670. Springer, 2022. 1, 3

    2022

  16. [25]

    Vilt: Vision- and-language transformer without convolution or region su- pervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In ICML, pages 5583–5594. PMLR, 2021. 3

    2021

  17. [26]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, pages 4015–4026, 2023. 1, 3

    2023

  18. [27]

    Efficient infer- ence in fully connected crfs with gaussian edge potentials

    Philipp Kr ¨ahenb¨uhl and Vladlen Koltun. Efficient infer- ence in fully connected crfs with gaussian edge potentials. NeurIPS, 24, 2011. 7

    2011

  19. [28]

    Clearclip: Decom- posing clip representations for dense vision-language infer- ence

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decom- posing clip representations for dense vision-language infer- ence. In ECCV, 2024. 1, 3, 4, 5, 6, 7, 8, 12, 13, 14

    2024

  20. [29]

    Proxyclip: Proxy 9 attention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy 9 attention improves clip for open-vocabulary segmentation. arXiv preprint arXiv:2408.04883, 2024. 1, 3

    2024 arXiv

  21. [30]

    Align before fuse: Vision and language representation learn- ing with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. NeurIPS, 34:9694–9705,

  22. [31]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888–12900. PMLR, 2022. 3

    2022

  23. [32]

    Clip surgery for better explainability with enhancement in open- vocabulary tasks

    Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open- vocabulary tasks. arXiv preprint arXiv:2304.05653, 2023. 3, 6, 7

    2023 arXiv

  24. [33]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, pages 7061–7070, 2023. 3

    2023

  25. [34]

    Segclip: Patch aggregation with learn- able centers for open-vocabulary semantic segmentation

    Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segclip: Patch aggregation with learn- able centers for open-vocabulary semantic segmentation. In ICML, pages 23033–23044. PMLR, 2023. 1, 3, 6, 7

    2023

  26. [35]

    End-to-end learning of visual representations from uncurated instruc- tional videos

    Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instruc- tional videos. In CVPR, pages 9879–9889, 2020. 3

    2020

  27. [36]

    The role of context for object detection and se- mantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and se- mantic segmentation in the wild. In CVPR, pages 891–898,

  28. [37]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1, 5

    2023 arXiv

  29. [38]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 1, 3, 6, 7, 13

    2021

  30. [39]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1–67, 2020. 1

    2020

  31. [40]

    Denseclip: Language-guided dense prediction with context- aware prompting

    Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context- aware prompting. In CVPR, pages 18082–18091, 2022. 1

    2022

  32. [41]

    Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency

    Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guan- grun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. arXiv preprint arXiv:2302.10307, 2023. 1, 3, 7

    2023 arXiv

  33. [42]

    Ex- plore the potential of clip for training-free open vocabulary semantic segmentation

    Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Ex- plore the potential of clip for training-free open vocabulary semantic segmentation. In ECCV, pages 139–156. Springer,

  34. [43]

    Reco: Re- trieve and co-segment for zero-shot transfer

    Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Re- trieve and co-segment for zero-shot transfer. NeurIPS, 35: 33754–33767, 2022. 3, 6, 7

    2022

  35. [44]

    Clip as rnn: Segment countless visual concepts without training endeavor

    Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip as rnn: Segment countless visual concepts without training endeavor. In CVPR, pages 13171–13182, 2024. 3

    2024

  36. [45]

    Learning to decompose vi- sual features with latent textual prompts

    Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alexan- der G Schwing, and Heng Ji. Learning to decompose vi- sual features with latent textual prompts. arXiv preprint arXiv:2210.04287, 2022. 3

    2022 arXiv

  37. [46]

    Sclip: Rethinking self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. In ECCV, pages 315–332. Springer, 2025. 1, 3, 4, 5, 6, 7, 8, 12, 13, 14

    2025

  38. [47]

    Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. In ICCV, pages 21372–21383, 2023. 3

    2023

  39. [48]

    Rewrite caption semantics: Bridging se- mantic gaps for language-supervised semantic segmentation

    Yun Xing, Jian Kang, Aoran Xiao, Jiahao Nie, Ling Shao, and Shijian Lu. Rewrite caption semantics: Bridging se- mantic gaps for language-supervised semantic segmentation. NeurIPS, 36, 2024. 1, 3, 6, 7

    2024

  40. [49]

    Demystify- ing clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystify- ing clip data. arXiv preprint arXiv:2309.16671, 2023. 3

    2023 arXiv

  41. [50]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, pages 18134–18144, 2022. 1, 3, 6, 7

    2022

  42. [51]

    Learning open-vocabulary semantic segmentation models from natural language supervision

    Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocabulary semantic segmentation models from natural language supervision. In CVPR, pages 2935–2944, 2023. 1, 3, 6, 7

    2023

  43. [52]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR, pages 2945–2954, 2023. 1

    2023

  44. [53]

    Unified contrastive learning in image-text-label space

    Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR, pages 19163–19173,

  45. [54]

    Tuning-free universally- supervised semantic segmentation

    Xiaobo Yang and Xiaojin Gong. Tuning-free universally- supervised semantic segmentation. arXiv preprint arXiv:2405.14294, 2024. 3

    2024 arXiv

  46. [55]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021. 3

    2021 arXiv

  47. [56]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 1

    2022 arXiv

  48. [57]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, 10 Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. 3

    2021 arXiv

  49. [58]

    Uncovering prototypical knowledge for weakly open- vocabulary semantic segmentation

    Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tianjiao Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Uncovering prototypical knowledge for weakly open- vocabulary semantic segmentation. NeurIPS, 36:73652– 73665, 2023. 1, 3, 6, 7

    2023

  50. [59]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. IJCV, 127: 302–321, 2019. 6, 14, 15

    2019

  51. [60]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, pages 696–712. Springer,

  52. [61]

    Image seg- mentation in foundation model era: A survey

    Tianfei Zhou, Fei Zhang, Boyu Chang, Wenguan Wang, Ye Yuan, Ender Konukoglu, and Daniel Cremers. Image seg- mentation in foundation model era: A survey. arXiv preprint arXiv:2408.12957, 2024. 3

    2024 arXiv

  53. [62]

    Zegclip: Towards adapting clip for zero-shot se- mantic segmentation

    Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot se- mantic segmentation. In CVPR, pages 11175–11185, 2023. 1

    2023

  54. [63]

    Segment everything everywhere all at once

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. NeurIPS, 36, 2024. 3 11 ResCLIP: Residual Attention for Training-free Dense Vision-language Inference Supplementary M...

    2024

  55. [64]

    For example, in the COCO Ob- ject dataset (see Fig

    ResCLIP presents superior performance in handling mul- tiple object instances, demonstrating its enhanced spatial- semantic understanding. For example, in the COCO Ob- ject dataset (see Fig. A3), our method accurately segments groups of animals while maintaining clear boundari...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.