Pith. sign in

REVIEW 8 cited by

EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.20076 v5 pith:NCNPVANI submitted 2024-06-28 cs.CV

Yuxuan Zhang , Tianheng Cheng , Lianghui Zhu , Rui Hu , Lei Liu , Heng Liu , Longjin Ran , Xiaoxin Chen
show 2 more authors
Wenyu Liu Xinggang Wang
This is my paper · ORCID
classification cs.CV
keywords segmentationevf-sampromptsreferringvision-languageearlymodelfusion
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Segment Anything Model (SAM) has attracted widespread attention for its superior interactive segmentation capabilities with visual prompts while lacking further exploration of text prompts. In this paper, we empirically investigate what text prompt encoders (e.g., CLIP or LLM) are good for adapting SAM for referring expression segmentation and introduce the Early Vision-language Fusion-based SAM (EVF-SAM). EVF-SAM is a simple yet effective referring segmentation method which exploits multimodal prompts (i.e., image and text) and comprises a pre-trained vision-language model to generate referring prompts and a SAM model for segmentation. Surprisingly, we observe that: (1) multimodal prompts and (2) vision-language models with early fusion (e.g., BEIT-3) are beneficial for prompting SAM for accurate referring segmentation. Our experiments show that the proposed EVF-SAM based on BEIT-3 can obtain state-of-the-art performance on RefCOCO/+/g for referring expression segmentation and demonstrate the superiority of prompting SAM with early vision-language fusion. In addition, the proposed EVF-SAM with 1.32B parameters achieves remarkably higher performance while reducing nearly 82% of parameters compared to previous SAM methods based on large multimodal models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPO

    cs.CV 2026-07 accept novelty 7.0 of 10

    A single MLLM jointly improves region captioning and localization by rewarding captions that let it reconstruct the original mask, needing only region inputs.

  2. CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CROSS improves remote sensing referring segmentation by combining cascaded SAM distillation with contrastive learning, reporting state-of-the-art cIoU on RefSegRS and RRSIS-D.

  3. Attribute Retrieving for Open-Vocabulary Endoscopic Compositional Referring Segmentation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ReferEndoscopy plus attribute-retrieval and frequency-aware fusion yields open-vocabulary compositional referring segmentation that outperforms natural-image RIS baselines on endoscopic data and generalizes to an unse...

  4. SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A SAM2-based framework that uses a fused text-audio-visual token to prompt video segmentation achieves 58.5 J&F on Ref-AVS, outperforming the previous state of the art by 8.5 points.

  5. SynRES: Towards Referring Expression Segmentation in the Wild via Synthetic Data

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SynRES, a synthetic data pipeline with grouped pseudo-mask consensus and domain-aware augmentations, improves referring expression segmentation on the new WildRES benchmark by 2.0 to 6.2 gIoU.

  6. CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentation

    eess.IV 2025-06 conditional novelty 5.0 of 10

    A text-guided SAM2 variant with cross-modal attention, semantic prompt generation, and a similarity-sorted memory bank achieves top Dice and surface scores on seven public multi-organ CT datasets.

  7. MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping

    cs.RO 2025-06 conditional novelty 5.0 of 10

    A two-stage language-driven grasping system that pools visual features inside a predicted object mask improves grasp accuracy and training efficiency versus CLIP baselines, supported by a new 219M-grasp dataset.

  8. Detection of Breast Cancer Lumpectomy Margin with SAM-incorporated Forward-Forward Contrastive Learning

    cs.CV 2025-06 reject novelty 4.0 of 10

    FFCL-SAM, a patch-level classifier plus SAM-based refinement, reports AUC 0.8455 and improved margin segmentation on intraoperative breast radiographs, but the test set excludes negative patients.

Pith tools