Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

SAM-MI claims open-vocabulary segmentation improves when SAM's masks are injected into CLIP's pixel-text cost maps at separate low and high frequencies, rather than being hard-combined with labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:21 UTC pith:MCR452OU

load-bearing objection Useful SAM-CLIP integration with a real speedup, but the headline gains shrink once you notice the baselines are borrowed and the MESS comparison is trained-vs-zero-shot. the 3 major comments →

arxiv 2511.20027 v2 pith:MCR452OU submitted 2025-11-25 cs.CV

SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM

Lin Chen , Yingjian Zhu , Qi Yang , Xin Niu , Kun Ding , Shiming Xiang This is my paper
classification cs.CV
keywords open-vocabulary semantic segmentationSegment Anything Modelmask injectioncost mapssparse point promptsCLIPDecoupled Mask InjectionShallow Mask Aggregation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish a new way to couple the Segment Anything Model (SAM) with CLIP-based open-vocabulary segmentation: instead of using SAM's masks as fixed region proposals that are then labeled (mask-driven) or prompting SAM with per-class boxes (prompt-driven), SAM-MI uses SAM's masks as soft guidance injected into the pixel-text cost maps that CLIP already produces. The injection is decoupled into low-frequency (global, mask-interior context) and high-frequency (edge and detail) components, which the authors argue fixes two failure modes: SAM's over-segmentation and the rigid, error-prone combination of fixed masks with labels. The support is a set of mIoU gains — 4.2%, 4.2%, and 3.5% relative over the same SAM-less backbone on ADE20K-847, PC-459, and ADE20K-150 with a ViT-B/16 CLIP, and 16.7% relative over Grounded-SAM on the cross-domain MESS benchmark — plus a 1.6x inference speedup from a text-guided sparse point prompter that reduces SAM's prompt count by 96%. If true, this gives practitioners a cheap, general recipe for adding SAM's universal segmentation ability to any cost-map-based OVSS model.

Core claim

The paper's central discovery is that decoupled mask injection — adding SAM's masks as low-frequency context and high-frequency detail to CLIP's cost maps — is a better way to equip open-vocabulary segmentation with SAM than labeling fixed masks or prompting SAM with boxes. This turns SAM's universal masks from competitive region proposals into corrective guidance, which is robust to SAM's over-segmentation and low-quality masks. The ablations confirm each component matters: removing the high-frequency injection costs about 0.3 mIoU, removing the low-frequency injection costs more, and removing the aggregation step costs 2.2 mIoU on ADE20K-150. The sparse prompter matches dense grid sampling

What carries the argument

Three modules carry the argument. Decoupled Mask Injection (DMI) splits the guidance: low-frequency injection uses mask-pooling and cross-attention to model intra-mask and inter-mask context in the coarse cost aggregation stage; high-frequency injection concatenates mask embeddings with shallow CLIP features and passes them through a depthwise convolution plus MLP to sharpen edges in the upsampling decoder. Shallow Mask Aggregation (SMAgg) merges SAM's over-segmented proposals using text-guided masks (from the TSPP head) and a matching-score threshold (0.5), reducing mask redundancy. Text-guided Sparse Point Prompter (TSPP) learns text-specific sampling probabilities from cost maps, supervis

Load-bearing premise

The whole framework depends on TSPP's hand-crafted probability targets — that useful SAM prompts are points near ground-truth mask skeletons with point budgets proportional to mask size; if unseen classes and domains violate this prior, the sparse prompts will miss objects and the injected masks will be too few or wrong, and the MESS comparison is partly confounded by training data.

What would settle it

A controlled experiment on a novel-category benchmark (e.g., classes not in COCO-Stuff, with no mask-skeleton supervision for TSPP) comparing SAM-MI against its own SAM-less backbone: if the gap disappears or reverses when TSPP is replaced by uniform grid sampling at the same point budget, the claim that sparse text-guided prompting preserves accuracy fails. Also, retraining Grounded-SAM on COCO-Stuff and re-running MESS would test whether the 16.7% lead survives matched training.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the mask-injected recipe generalizes, any future CLIP-based OVSS model can plug in a frozen SAM with the three modules and obtain consistent mIoU gains without retraining the VLM from scratch.
  • Because TSPP cuts SAM's prompt count by 96% and the inference latency by about 1.6x versus Grounded-SAM, SAM-powered OVSS becomes practical on a single GPU for deployment.
  • The gains are larger when the CLIP backbone is mid-size (ViT-B/16): 4-5% relative improvements, whereas a larger ViT-L/14 sees only 1.8-2.5% gains, suggesting mask injection compensates for weaker cost maps.
  • The ablation removing SMAgg (a 2.2 mIoU drop on A-150) shows that over-segmentation is the main liability of SAM-based OVSS, so aggregation is as important as injection.
  • Cross-domain results on MESS (earth monitoring: +45% relative over Grounded-SAM) indicate the benefit is strongest on specialized domains where CLIP's cost maps are noisy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The decoupling principle likely transfers beyond semantic segmentation: the same low/high-frequency injection could improve open-vocabulary panoptic or instance segmentation, where the boundary-detail channel matters even more.
  • TSPP's reliance on skeleton-derived targets trained on COCO-Stuff is a weakness for truly open classes; a purely self-supervised point selector (e.g., based on cost-map uncertainty) would make the framework annotation-free.
  • The MESS comparison with Grounded-SAM is not perfectly controlled: SAM-MI is trained on COCO-Stuff while Grounded-SAM is not, so part of the 16.7% gain may be training-data, not the mask-injected architecture; re-running with matched training would clarify.
  • If one replaced the frozen SAM-H with MobileSAM, the accuracy drop is small (about 0.4 mIoU on A-150) while latency drops by roughly 300 ms, pointing to a deployable variant that keeps most of the benefit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SAM-MI is a framework for open-vocabulary semantic segmentation that combines CLIP-based cost maps with frozen SAM masks. Three components are proposed: TSPP learns sparse text-guided point prompts for SAM, reducing the number of prompts by ~96% compared to a dense 32x32 grid; SMAgg aggregates over-segmented SAM masks using coarse text-mask overlap; and DMI injects the aggregated masks into low-frequency and high-frequency stages of a CAT-Seg-style cost aggregation network. The method is trained on COCO-Stuff and evaluated on A-847, PC-459, A-150, PC-59, PAS-20, and the MESS benchmark. The paper reports gains over CAT-Seg on the five standard benchmarks (e.g., +1.1 mIoU on A-150) and a 16.7% relative improvement over Grounded-SAM on MESS, with a 1.6x speedup.

Significance. If the empirical claims are reliable, the mask-injected paradigm is a useful and general alternative to mask-driven and prompt-driven ways of coupling SAM with CLIP-based OVSS: using SAM masks as soft guidance inside cost-map aggregation is conceptually clean and could transfer to other cost-map architectures. TSPP's sparse prompting also gives a very large latency reduction with little accuracy loss, which is practically valuable. The paper is clearly written and the ablations are organized in a way that, in principle, isolates each module. However, the current evaluation does not establish the central claim. The gains over the same backbone are small (0.5-1.1 mIoU absolute) with no error bars; the w/o DMI ablation row exactly reproduces published CAT-Seg numbers, suggesting the baseline was not retrained under the same conditions; and the headline MESS comparison is confounded because SAM-MI is COCO-Stuff-trained while Grounded-SAM is used zero-shot. A controlled rerun plus uncertainty quantification could turn this into a solid contribution, but as it stands the evidence is not yet convincing.

major comments (3)
  1. [§4.3, Tables 3 and 4] The 'w/o DMI' row reports exactly the same five numbers (12.0/19.0/31.8/57.5/94.6) as the CAT-Seg ViT-B/16 row in Table 2. Removing DMI should leave a CAT-Seg-like model, so the numbers should be close, but an exact match across all five benchmarks strongly suggests that the authors copied the published CAT-Seg results rather than running this ablation in their own pipeline. This matters because the paper's central claim — that mask injection improves OVSS — rests on the difference between SAM-MI and this row. Please rerun the w/o DMI configuration under the same training code, data, schedule, and hardware, and report mean ± std over multiple seeds. If the run was already performed, please provide logs or other evidence.
  2. [§4.2, Table 2] The improvements over CAT-Seg are small: +0.5 mIoU on A-847, +0.8 on PC-459, +1.1 on A-150, +0.9 on PC-59, and +0.6 on PAS-20. No variance, confidence intervals, or multiple-seed results are reported. Given that the w/o DMI ablation appears to reproduce published CAT-Seg numbers exactly, it is uncertain whether the differences are due to the proposed modules or to run-to-run variation. The paper should include at least three seeds for SAM-MI and the controlled baseline, and ideally a paired significance test. The Limitations section itself notes that the benchmarks contain incomplete/incorrect annotations, which further underscores the need for statistical robustness when claiming small gains.
  3. [Abstract and §4.2, Table 1] The headline MESS claim — a 16.7% relative mIoU improvement over Grounded-SAM — is not an apples-to-apples comparison. SAM-MI is trained on COCO-Stuff, whereas Grounded-SAM is an off-the-shelf zero-shot pipeline (Grounding DINO + SAM) with no COCO-Stuff training. The training-data advantage alone could explain much of the difference. To support the claimed cross-domain benefit of the mask-injected framework, the MESS table should include a COCO-Stuff-trained baseline using the same CLIP backbone (e.g., CAT-Seg or the SAM-MI model without DMI/SMAgg). The abstract and conclusion should be reworded so that the comparison is not presented as an architectural improvement.
minor comments (5)
  1. [General] Typos: 'Renmin Univeristy' in the affiliations; 'even even more significant' in §4.3; 'xFurthermore' before Table 10; 'denotes as M' in Eq. (8). Please proofread.
  2. [§4.2, MESS discussion] The text states that in the earth-monitoring domain Grounded-SAM reaches 25.97 mIoU and SAM-MI reaches 37.66, a 45.0% relative improvement. These averages do not match the values in Table 1 for either the five or six datasets in that domain. Please recompute or correct the text.
  3. [Tables 3 and 5] The latency for the uniform-grid configuration is inconsistent: Table 3 reports 5458 ms for w/o TSPP, while Table 5 reports 5967 ms for the same 32x32 uniform grid on A-150. Clarify the exact dataset, image size, hardware, and measurement setup for each latency number.
  4. [Eq. (3)] The probability-target formula is typeset ambiguously: it is not clear whether P_k is in the numerator or denominator. Please write the expression explicitly and define all symbols (e.g., d is the distance transform, M_gt^k is the ground-truth mask).
  5. [§4.2 and Tables] Please state whether reported numbers are the best over runs or the mean, and report standard deviations where feasible. Also, since no code is provided, a reproducibility statement or plan to release code would strengthen the paper.

Circularity Check

0 steps flagged

No significant circularity: central claims are held-out empirical evaluations; self-citation is peripheral and non-load-bearing.

full rationale

Walking the derivation chain, the main empirical claims are benchmark results on held-out data, not quantities forced by construction. TSPP's sampling probabilities are trained against ground-truth-derived targets (Eqs. 3-5), but at inference the ground-truth masks are not reused; sampling is driven by learned text-guided probabilities applied to unseen images. SMAgg (Eqs. 6-7) uses TSPP's predicted text masks and SAM proposals, and DMI injects the resulting masks into CLIP cost maps; no equation defines the output in terms of evaluation labels. The reported mIoU gains over CAT-Seg and Grounded-SAM are external evaluations, and the method is trained only on COCO-Stuff. The sole self-citation ([44], by the same first author) appears only as related-work context ('CAT-Seg [42], SED [43] and ERR-Seg [44] model OVSS tasks following the FCN [45] methodology') and is not load-bearing. The exact match of the 'w/o DMI' row to CAT-Seg's published numbers and the MESS comparison against zero-shot Grounded-SAM are evaluation-control concerns, not circular reductions, and therefore do not raise the circularity score.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The method is entirely empirical. It introduces one learned module (TSPP) plus five hand-set hyperparameters and relies on frozen SAM/CLIP and on CAT-Seg's cost-aggregation design. No new physical or conceptual entities are posited.

free parameters (5)
  • alpha_sma = 0.50
    Aggregation threshold in Eq. (7); tuned on A-847/PC-459/A-150 validation sets; Fig. 8 shows sensitivity outside [0.4,0.6].
  • g_p = 5
    Density divisor in Eq. (5); controls expected number of points per mask; selected via Table 7 ablation.
  • m_p = 10
    Upper limit on expected points per mask in Eq. (5); selected via Table 7 ablation.
  • lambda_mse = 0.5
    Weight on the MSE loss in TSPP training; selected via Table 8 ablation.
  • lambda_tspp = 0.1
    Weight on TSPP loss during stage-2 training; set in Sec. 3.5 without a dedicated ablation.
axioms (5)
  • domain assumption Frozen SAM-H, trained on 1.1B masks, supplies high-quality universal masks across domains.
    Secs. 3.1 and 4.1 treat SAM masks as trustworthy priors; failure cases in Sec. 4.3 show small-object limitations.
  • domain assumption CLIP pixel-text cost maps encode enough spatial and class information to drive sparse point sampling and mask aggregation.
    Secs. 3.2 and 3.3 use S as the sole signal for TSPP probabilities and SMAgg matching.
  • domain assumption CAT-Seg's cost aggregation framework is a valid backbone, and its low/high-frequency decomposition matches the DMI design.
    Sec. 3.4 defines DMI on CAT-Seg's F_v^l, F_v^h1, F_v^h2; no independent justification of the decomposition is given.
  • ad hoc to paper Skeleton-distance Gaussian target probabilities (Eqs. 3-5) are a sound inductive bias for sampling SAM prompts.
    Introduced in Sec. 3.2 solely to make TSPP trainable; no prior evidence is cited for this specific prior.
  • ad hoc to paper Overlap ratio O(i,j) in Eq. (6) with threshold alpha_sma reliably identifies SAM masks belonging to the same semantic class.
    Sec. 3.3 assumes coarse TSPP text masks are reliable enough for merging despite being imperfect themselves.

pith-pipeline@v1.3.0-alltime-deepseek · 19160 in / 14740 out tokens · 157304 ms · 2026-08-03T20:21:21.444002+00:00 · methodology

0 comments
read the original abstract

Open-vocabulary semantic segmentation (OVSS) aims to segment and recognize objects universally. Trained on extensive high-quality segmentation data, the segment anything model (SAM) has demonstrated remarkable universal segmentation capabilities, offering valuable support for OVSS. Although previous methods have made progress in leveraging SAM for OVSS, there are still some challenges: (1) SAM's tendency to over-segment and (2) hard combinations between fixed masks and labels. This paper introduces a novel mask-injected framework, SAM-MI, which effectively integrates SAM with OVSS models to address these challenges. Initially, SAM-MI employs a Text-guided Sparse Point Prompter to sample sparse prompts for SAM instead of previous dense grid-like prompts, thus significantly accelerating the mask generation process. The framework then introduces Shallow Mask Aggregation (SMAgg) to merge partial masks to mitigate the SAM's over-segmentation issue. Finally, Decoupled Mask Injection (DMI) incorporates SAM-generated masks for guidance at low-frequency and high-frequency separately, rather than directly combining them with labels. Extensive experiments on multiple benchmarks validate the superiority of SAM-MI. Notably, the proposed method achieves a 16.7% relative improvement in mIoU over Grounded-SAM on the MESS benchmark, along with a 1.6$\times$ speedup. We hope SAM-MI can serve as an alternative methodology to effectively equip the OVSS model with SAM.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Towards Realistic Open-Vocabulary Remote Sensing Segmentation: Benchmark and Baseline

    cs.CV 2026-04 unverdicted novelty 7.0

    OVRSISBenchV2 is a realistic benchmark expanding scene and category coverage for open-vocabulary remote sensing segmentation, with Pi-Seg baseline showing strong transfer via positive-incentive noise perturbations.

  2. WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

    cs.CV 2026-04 unverdicted novelty 6.0

    WikiSeeker boosts KB-VQA performance by using VLMs to rewrite image-informed queries for better retrieval and to decide when to route to external LLM or rely on internal VLM knowledge.

  3. Towards Realistic Open-Vocabulary Remote Sensing Segmentation: Benchmark and Baseline

    cs.CV 2026-04 unverdicted novelty 5.5

    OVRSISBenchV2 expands open-vocabulary remote-sensing segmentation evaluation to 170K images and 128 categories, and Pi-Seg uses positive-incentive noise to improve transfer on that harder benchmark.

Reference graph

Works this paper leans on

68 extracted references · 21 linked inside Pith · cited by 2 Pith papers

  1. [1]

    A sim- ple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model,

    M. Xu, Z. Zhang, F. Wei, Y. Lin, Y. Cao, H. Hu, and X. Bai, “A sim- ple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model,” inProceedings of the European Conference on Computer Vision, 2022, pp. 736–753

    2022

  2. [2]

    Decoupling zero-shot semantic segmentation,

    J. Ding, N. Xue, G.-S. Xia, and D. Dai, “Decoupling zero-shot semantic segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11 583–11 592

    2022

  3. [3]

    Open-vocabulary semantic segmentation with mask- adapted clip,

    F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu, “Open-vocabulary semantic segmentation with mask- adapted clip,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7061–7070

    2023

  4. [4]

    Maskclip: Masked self-distillation advances contrastive language-image pretraining,

    X. Dong, J. Bao, Y. Zheng, T. Zhang, D. Chen, H. Yang, M. Zeng, W. Zhang, L. Yuan, D. Chenet al., “Maskclip: Masked self-distillation advances contrastive language-image pretraining,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10 995–11 005

    2023

  5. [5]

    Open- vocabulary panoptic segmentation with text-to-image diffusion models,

    J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello, “Open- vocabulary panoptic segmentation with text-to-image diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2955–2966

    2023

  6. [6]

    Side adapter net- work for open-vocabulary semantic segmentation,

    M. Xu, Z. Zhang, F. Wei, H. Hu, and X. Bai, “Side adapter net- work for open-vocabulary semantic segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2945–2954

    2023

  7. [7]

    Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,

    Q. Yu, J. He, X. Deng, X. Shen, and L.-C. Chen, “Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  8. [8]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Loet al., “Segment anything,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015–4026

    2023

  9. [9]

    Open-vocabulary semantic segmentation with image embedding balancing,

    X. Shan, D. Wu, G. Zhu, Y. Shao, N. Sang, and C. Gao, “Open-vocabulary semantic segmentation with image embedding balancing,” inProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 28 412–28 421

    2024

  10. [10]

    Frozenseg: Harmoniz- ing frozen foundation models for open-vocabulary segmentation,

    X. Chen, H. Yang, S. Jin, X. Zhu, and H. Yao, “Frozenseg: Harmoniz- ing frozen foundation models for open-vocabulary segmentation,”arXiv preprint arXiv:2409.03525, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Semantic segment anything,

    J. Chen, Z. Yang, and L. Zhang, “Semantic segment anything,” https: //github.com/fudan-zvg/Semantic-Segment-Anything, 2023

    2023

  12. [12]

    Sam-cp: Marrying sam with composable prompts for versatile segmentation,

    P. Chen, L. Xie, X. Huo, X. Yu, X. Zhang, Y. Sun, Z. Han, and Q. Tian, “Sam-cp: Marrying sam with composable prompts for versatile segmentation,”arXiv preprint arXiv:2407.16682, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Sam-clip: Merging vision foundation models towards semantic and spatial understanding,

    H. Wang, P. K. A. Vasu, F. Faghri, R. Vemulapalli, M. Farajtabar, S. Mehta, M. Rastegari, O. Tuzel, and H. Pouransari, “Sam-clip: Merging vision foundation models towards semantic and spatial understanding,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3635–3647

    2024

  14. [14]

    Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively,

    H. Yuan, X. Li, C. Zhou, Y. Li, K. Chen, and C. C. Loy, “Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively,”arXiv preprint arXiv:2401.02955, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yanet al., “Grounded sam: Assembling open-world models for diverse visual tasks,”arXiv preprint arXiv:2401.14159, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Learning to prompt segment anything models,

    J. Huang, K. Jiang, J. Zhang, H. Qiu, L. Lu, S. Lu, and E. Xing, “Learning to prompt segment anything models,”arXiv preprint arXiv:2401.04651, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Boosting segment anything model towards open-vocabulary learning,

    X. Han, L. Wei, X. Yu, Z. Dou, X. He, K. Wang, Z. Han, and Q. Tian, “Boosting segment anything model towards open-vocabulary learning,” arXiv preprint arXiv:2312.03628, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Open- vocabulary segmentation with unpaired mask-text supervision,

    Z. Wang, X. Xia, Z. Chen, X. He, Y. Guo, M. Gong, and T. Liu, “Open- vocabulary segmentation with unpaired mask-text supervision,”arXiv preprint arXiv:2402.08960, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Semantic understanding of scenes through the ade20k dataset,

    B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Tor- ralba, “Semantic understanding of scenes through the ade20k dataset,” International Journal of Computer Vision, vol. 127, pp. 302–321, 2019

    2019

  20. [20]

    The role of context for object detection and semantic segmentation in the wild,

    R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urta- sun, and A. Yuille, “The role of context for object detection and semantic segmentation in the wild,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2014, pp. 891–898

    2014

  21. [21]

    What a MESS: Multi-Domain Evaluation of Zero-shot Semantic Segmentation,

    B. Blumenstiel, J. Jakubik, H. K¨ uhne, and M. V¨ ossing, “What a MESS: Multi-Domain Evaluation of Zero-shot Semantic Segmentation,” Advances in Neural Information Processing Systems, 2023

    2023

  22. [22]

    Semantic-sam: Segment and recognize anything at any granularity,

    F. Li, H. Zhang, P. Sun, X. Zou, S. Liu, J. Yang, C. Li, L. Zhang, and J. Gao, “Semantic-sam: Segment and recognize anything at any granularity,”arXiv preprint arXiv:2307.04767, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Segment everything everywhere all at once,

    X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J. Gao, and Y. J. Lee, “Segment everything everywhere all at once,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  24. [24]

    Segment anything in high quality,

    L. Ke, M. Ye, M. Danelljan, Y.-W. Tai, C.-K. Tang, F. Yuet al., “Segment anything in high quality,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  25. [25]

    Fast segment anything,

    X. Zhao, W. Ding, Y. An, Y. Du, T. Yu, M. Li, M. Tang, and J. Wang, “Fast segment anything,”arXiv preprint arXiv:2306.12156, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Faster segment anything: Towards lightweight sam for mobile applications,

    C. Zhang, D. Han, Y. Qiao, J. U. Kim, S.-H. Bae, S. Lee, and C. S. Hong, “Faster segment anything: Towards lightweight sam for mobile applications,”arXiv preprint arXiv:2306.14289, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Efficientsam: Leveraged masked image pretraining for efficient segment anything,

    Y. Xiong, B. Varadarajan, L. Wu, X. Xiang, F. Xiao, C. Zhu, X. Dai, D. Wang, F. Sun, F. Iandolaet al., “Efficientsam: Leveraged masked image pretraining for efficient segment anything,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 16 111–16 121

    2024

  28. [28]

    Segment anything is not always perfect: An investigation of sam on different real-world applications,

    W. Ji, J. Li, Q. Bi, T. Liu, W. Li, and L. Cheng, “Segment anything is not always perfect: An investigation of sam on different real-world applications,” 2024

    2024

  29. [29]

    Balanced representation learning for long-tailed skeleton-based action recognition,

    H. Liu, Y. Wang, M. Ren, J. Hu, Z. Luo, G. Hou, and Z. Sun, “Balanced representation learning for long-tailed skeleton-based action recognition,” Machine Intelligence Research, pp. 1–18, 2025

    2025

  30. [30]

    Caption anything: Interactive image description with diverse multimodal controls,

    T. Wang, J. Zhang, J. Fei, H. Zheng, Y. Tang, Z. Li, M. Gao, and S. Zhao, “Caption anything: Interactive image description with diverse multimodal controls,”arXiv preprint arXiv:2305.02677, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Editanything: Empowering unparalleled flexibility in image editing and generation,

    S. Gao, Z. Lin, X. Xie, P. Zhou, M.-M. Cheng, and S. Yan, “Editanything: Empowering unparalleled flexibility in image editing and generation,” in Proceedings of the 31st ACM International Conference on Multimedia, Demo track, 2023

    2023

  32. [32]

    Clipsam: Clip and sam collaboration for zero-shot anomaly segmentation,

    S. Li, J. Cao, P. Ye, Y. Ding, C. Tu, and T. Chen, “Clipsam: Clip and sam collaboration for zero-shot anomaly segmentation,”arXiv preprint arXiv:2401.12665, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Segment any anomaly without training via hybrid prompt regularization,

    Y. Cao, X. Xu, C. Sun, Y. Cheng, Z. Du, L. Gao, and W. Shen, “Segment any anomaly without training via hybrid prompt regularization,”arXiv preprint arXiv:2305.10724, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Deep industrial image anomaly detection: A survey,

    J. Liu, G. Xie, J. Wang, S. Li, C. Wang, F. Zheng, and Y. Jin, “Deep industrial image anomaly detection: A survey,”Machine Intelligence Research, vol. 21, no. 1, pp. 104–135, 2024

    2024

  35. [35]

    Medical sam 2: Segment medical images as video via segment anything model 2,

    J. Zhu, Y. Qi, and J. Wu, “Medical sam 2: Segment medical images as video via segment anything model 2,”arXiv preprint arXiv:2408.00874, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Segment anything in medical images,

    J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang, “Segment anything in medical images,”Nature Communications, vol. 15, no. 1, p. 654, 2024

    2024

  37. [37]

    Learning transfer- able visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transfer- able visual models from natural language supervision,” inInternational Conference on Machine Learning, 2021, pp. 8748–8763

    2021

  38. [38]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.- H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” inInternational Conference on Machine Learning, 2021, pp. 4904–4916

    2021

  39. [39]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Conference on Machine Learning, 2022, pp. 12 888–12 900

    2022

  40. [40]

    Regionclip: Region-based language-image pretrain- ing,

    Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Liet al., “Regionclip: Region-based language-image pretrain- ing,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 793–16 803

    2022

  41. [41]

    Mrovseg: Break- ing the resolution curse of vision-language models in open-vocabulary semantic segmentation,

    Y. Zhu, B. Zhu, Z. Chen, H. Xu, M. Tang, and J. Wang, “Mrovseg: Break- ing the resolution curse of vision-language models in open-vocabulary semantic segmentation,”arXiv preprint arXiv:2408.14776, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmentation,

    S. Cho, H. Shin, S. Hong, A. Arnab, P. H. Seo, and S. Kim, “Cat-seg: Cost aggregation for open-vocabulary semantic segmentation,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 4113–4123

    2024

  43. [43]

    Sed: A simple encoder- decoder for open-vocabulary semantic segmentation,

    B. Xie, J. Cao, J. Xie, F. S. Khan, and Y. Pang, “Sed: A simple encoder- decoder for open-vocabulary semantic segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3426–3436

    2024

  44. [44]

    Efficient redundancy reduction for open-vocabulary semantic segmentation.arXiv preprint arXiv:2501.17642, 2025

    Lin Chen, Qi Yang, Kun Ding, Zhihao Li, Gang Shen, Fei Li, Qiyuan Cao, and Shiming Xiang. Efficient redundancy reduction for open-vocabulary semantic segmentation.arXiv preprint arXiv:2501.17642, 2025

    arXiv 2025

  45. [45]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440

    2015

  46. [46]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhuet al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,”arXiv preprint arXiv:2303.05499, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    Personalize segment anything model with one shot,

    R. Zhang, Z. Jiang, Z. Guo, S. Yan, J. Pan, X. Ma, H. Dong, P. Gao, and H. Li, “Personalize segment anything model with one shot,”arXiv preprint arXiv:2305.03048, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    Automating medsam by learning prompts with weak few-shot supervision,

    M. Gaillochet, C. Desrosiers, and H. Lombaert, “Automating medsam by learning prompts with weak few-shot supervision,” inInternational Workshop on Foundation Models for General Medical AI. Springer, 2024, pp. 61–70

    2024

  49. [49]

    Denseclip: Language-guided dense prediction with context-aware prompting,

    Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu, “Denseclip: Language-guided dense prediction with context-aware prompting,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 082–18 091

    2022

  50. [50]

    Coco-stuff: Thing and stuff classes in context,

    H. Caesar, J. Uijlings, and V. Ferrari, “Coco-stuff: Thing and stuff classes in context,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 1209–1218

    2018

  51. [51]

    The pascal visual object classes (voc) challenge,

    M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,”International Journal of Computer Vision, vol. 88, pp. 303–338, 2010

    2010

  52. [52]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning,

    F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “Bdd100k: A diverse driving dataset for heterogeneous multitask learning,”arXiv preprint arXiv:1805.04687, 2020

    Pith/arXiv arXiv 2020

  53. [53]

    isaid: A large-scale dataset for instance segmentation in aerial images,

    S. Waqas Zamir, A. Arora, A. Gupta, S. Khan, G. Sun, F. Shahbaz Khan, F. Zhu, L. Shao, G.-S. Xia, and X. Bai, “isaid: A large-scale dataset for instance segmentation in aerial images,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 28–37

    2019

  54. [54]

    Cryonuseg: A dataset for nuclei instance segmentation of cryosectioned h&e-stained histological images,

    A. Mahbod, G. Schaefer, B. Bancher, C. L¨ ow, G. Dorffner, R. Ecker, and I. Ellinger, “Cryonuseg: A dataset for nuclei instance segmentation of cryosectioned h&e-stained histological images,”Computers in Biology and Medicine, vol. 132, p. 104349, 2021

    2021

  55. [55]

    Deepcrack: Learning hierarchical convolutional features for crack detection,

    Q. Zou, Z. Zhang, Q. Li, X. Qi, Q. Wang, and S. Wang, “Deepcrack: Learning hierarchical convolutional features for crack detection,”IEEE Transactions on Image Processing, vol. 28, no. 3, pp. 1498–1512, 2018

    2018

  56. [56]

    A crop/weed field image dataset for the eval- uation of computer vision based precision agriculture tasks,

    S. Haug and J. Ostermann, “A crop/weed field image dataset for the eval- uation of computer vision based precision agriculture tasks,” inComputer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6-7 and 12, 2014, Proceedings, Part IV 13. Springer, 2015, pp. 105–116

    2014

  57. [57]

    Reproducible scal- ing laws for contrastive language-image learning,

    M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scal- ing laws for contrastive language-image learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2818–2829

    2023

  58. [58]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszko- reit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,”ICLR, 2021

    2021

  59. [59]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  60. [60]

    Generalized decoding for pixel, image, and language,

    X. Zou, Z.-Y. Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang, L. Yuanet al., “Generalized decoding for pixel, image, and language,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 116–15 127

    2023

  61. [61]

    A simple framework for open-vocabulary segmentation and detection,

    H. Zhang, F. Li, X. Zou, S. Liu, C. Li, J. Yang, and L. Zhang, “A simple framework for open-vocabulary segmentation and detection,” inProceed- ings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 1020–1031

    2023

  62. [62]

    Open-vocabulary seman- tic segmentation with decoupled one-pass network,

    C. Han, Y. Zhong, D. Li, K. Han, and L. Ma, “Open-vocabulary seman- tic segmentation with decoupled one-pass network,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 1086–1096

    2023

  63. [63]

    Open-vocabulary segmenta- tion with semantic-assisted calibration,

    Y. Liu, S. Bai, G. Li, Y. Wang, and Y. Tang, “Open-vocabulary segmenta- tion with semantic-assisted calibration,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3491–3500

    2024

  64. [64]

    Learning mask-aware clip representations for zero-shot segmentation,

    S. Jiao, Y. Wei, Y. Wang, Y. Zhao, and H. Shi, “Learning mask-aware clip representations for zero-shot segmentation,”Advances in Neural Information Processing Systems, vol. 36, pp. 35 631–35 653, 2023

    2023

  65. [65]

    Collaborative vision-text representation optimizing for open-vocabulary segmentation,

    S. Jiao, H. Zhu, J. Huang, Y. Zhao, Y. Wei, and H. Shi, “Collaborative vision-text representation optimizing for open-vocabulary segmentation,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 399– 416

    2024

  66. [66]

    Eov-seg: Efficient open-vocabulary panoptic segmentation,

    H. Niu, J. Hu, J. Lin, G. Jiang, and S. Zhang, “Eov-seg: Efficient open-vocabulary panoptic segmentation,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 6, 2025, pp. 6254–6262

    2025

  67. [67]

    High- quality mask tuning matters for open-vocabulary segmentation,

    Q.-S. Zeng, Y. Li, D. Zhou, G. Li, Q. Hou, and M.-M. Cheng, “High- quality mask tuning matters for open-vocabulary segmentation,”arXiv preprint arXiv:2412.11464, 2024

    Pith/arXiv arXiv 2024

  68. [68]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R¨ adle, C. Rolland, L. Gustafsonet al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024

    Pith/arXiv arXiv 2024