Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SAMWISE makes a frozen SAM2 follow natural-language video queries by adding less than 5M trainable parameters and beats offline state of the art on three benchmarks.

desk verdict A genuinely useful RVOS adaptation of SAM2 with strong results and a clean adapter design; the CME correction is under-specified but fixable, and the SOTA claim deserves a serious referee. read the letter →

arxiv 2411.17646 v2 pith:MX7TIZY3 submitted 2024-11-26 cs.CV

Claudia Cuttano , Gabriele Trivigno , Gabriele Rosi , Carlo Masone , Giuseppe Averta This is my paper · ORCID
classification cs.CV
keywords referringvideoobjectsegmentationSAM2streamingcross-modaladaptertemporalmodelingtrackingbiasparameter-efficientadaptationfrozenfoundationmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a frozen SAM2 video-segmentation model can be made to follow natural-language referring expressions without retuning its weights, by inserting small learnable modules into its feature-extraction path. The method, SAMWISE, adds fewer than five million trainable parameters and still surpasses prior state-of-the-art methods, including offline methods that see the whole video at once, on MeViS, Ref-YouTube-VOS, and Ref-DAVIS. The paper also identifies a failure mode it calls tracking bias, where SAM2 commits to an initially plausible object and ignores a better text match that appears later, and introduces a learnable correction that lets the model switch. If the results hold, streaming text-guided video segmentation becomes practical as a lightweight add-on to an existing foundation model.

What carries the argument

The central object is the Cross-Modal Temporal Adapter (CMT), a small residual block placed inside each frozen encoder's intermediate layers. It combines Hierarchical Selective Attention (HSA), which divides the T-frame feature volume into non-overlapping 3-D spatio-temporal patches of size T×P×P, runs self-attention inside each patch, and grows P with feature resolution to capture multi-scale motion, with Visual-to-Text and Text-to-Visual attention that inject the other modality into each feature stream. From the adapted text features the model derives two prompts, a Contextual Prompt from the [CLS] token and a Motion Prompt from verb embeddings, projected through a three-layer MLP (Eq. 1) into SAM2's mask decoder. The companion mechanism is the Conditional Memory Encoder (CME), a self-attention plus linear classifier that compares the decoder's mask token with a memory-less token computed from the unbiased, text-aligned features, and on detecting a discrepancy fuses the memory-less prediction into the memory bank through a soft mask combination (Eq. 10), so SAM2 can shift its tracked object.

What would settle it

Run SAMWISE on the MeViS validation set with CME on and off, and on every frame where the CME fires, compare its proposed mask against the ground-truth object. If the CME's proposals are no more often correct than a randomly chosen distractor, or if forcing detections at every frame does not hurt J&F relative to never firing, then the correction is not the actual source of the reported gain and the tracking-bias story fails.

Watch

Extended reading notes

Core claim

SAMWISE establishes that SAM2's frozen feature extraction can be made text-aware and temporally aware while remaining frozen: a Cross-Modal Temporal Adapter (CMT) inserted at each intermediate layer performs Hierarchical Selective Attention over spatio-temporal patches to encode motion, along with visual-to-text and text-to-visual attention so that each modality conditions the other. The adapter outputs a context prompt from the [CLS] token and a motion prompt from verb embeddings, both projected through a learnable MLP and fed to SAM2's mask decoder at every frame. The paper further demonstrates that SAM2's memory features exhibit tracking bias, and that the memory-less features, being unbiased and text-aligned, can drive a Conditional Memory Encoder (CME) that detects when a newly visible object is more aligned with the caption and encodes its mask into the memory bank so the tracker can refocus. With 4.9M trainable parameters, no SAM2 fine-tuning, and no external vision-language model, SAMWISE reports J&F scores of 49.5 on MeViS, 69.2 on Ref-YouTube-VOS, and 70.6 on Ref-DAVIS, surpassing the previous offline state of the art DsHmp by +3.1, +2.1, and +5.7 points respectively.

Load-bearing premise

The load-bearing premise is that SAM2's memory-less features, once aligned with the text by the adapter, are unbiased enough that the token they produce reliably indicates when the currently tracked object is wrong and a newly visible object matches the caption; if that alignment is unreliable, the correction module would inject false candidates into the memory bank and the reported +1.3 J&F gain from the CME would shrink or disappear.

Editorial extensions

If this is right

  • Streaming referring-video segmentation becomes competitive with offline methods: SAMWISE beats the offline state of the art DsHmp on MeViS, Ref-YouTube-VOS, and Ref-DAVIS while consuming frames as they arrive.
  • A frozen video foundation model can gain a new modality without fine-tuning: under 5M trainable parameters preserve SAM2's original behavior.
  • SAM2's tracking bias can be corrected without a human in the loop by detecting text-aligned candidates in memory-less features and softly encoding them into the memory bank.
  • The comparison with GroundingDINO+SAM2 (37.7 vs 48.3 on MeViS) shows that early cross-modal and temporal adaptation, rather than box prompting from a detector, is what carries the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The CMT adapter is not tied to video: the supplementary reports competitive accuracy on image-level referring segmentation (RefCOCO/RefCOCO+/RefCOCOg), suggesting a single text-promptable adapter could serve both tasks.
  • A stress test would swap the frozen Hiera backbone for a stronger vision encoder without retraining the adapters: transfer of the gains would indicate CMT captures text-temporal alignment rather than benchmark-specific overfitting.
  • The CME's detection threshold is set at 0.5 and never swept; measuring how the +1.3 J&F gain varies with that threshold would reveal whether the correction is robust or a knife-edge.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents SAMWISE, a method for referring video object segmentation (RVOS) that augments a frozen SAM2 with two lightweight trainable components: a Cross-Modal Temporal Adapter (CMT) inserted into the image and text encoders, and a Conditional Memory Encoder (CME) that detects when SAM2's tracking bias causes it to follow the wrong object and then injects a corrective mask into the memory bank. The method processes videos in a streaming fashion with clips of T frames, avoids fine-tuning SAM2 and does not rely on external large VLMs. The authors report state-of-the-art results on MeViS, Ref-YouTube-VOS, and Ref-DAVIS17 with fewer than 5M trainable parameters, alongside ablations on MeViS and additional experiments in the supplementary material.

Significance. If the reported results hold, SAMWISE would be a meaningful advance: it shows that a frozen SAM2 can be made text-aware and temporally aware with a small adapter, outperforming offline methods that process the entire video and large-VLM-based approaches, while operating in a streaming setting. The work also identifies a concrete failure mode of SAM2 (tracking bias) and proposes a mechanism to mitigate it. Strengths include evaluation on three public benchmarks with official evaluation servers, a clear parameter-efficiency claim, and a component-level ablation. The main risk is the CME module: its self-supervised training signal is derived from the same decoder it is meant to correct, and the paper does not provide sufficient evidence that the module's detections are precise enough to justify the reported +1.3 J&F gain. Since the CME is a stated contribution and part of the headline results, this issue needs to be resolved before the central claims can be fully accepted.

major comments (4)
  1. [Sec. 3.5, Eq. (13), Table 2] The CME training label is self-referential and does not enforce text-aligned proposals. In Eq. (13), y_t = 1 if Y_l[t] ∩ Y_m[t] = ∅, where both masks are produced by the same frozen SAM2 decoder from different feature inputs (memory features vs. memory-less features). This label fires whenever the two masks are disjoint, including when both are wrong but cover different non-target regions, and it does not measure whether τ_l is actually aligned with the referring expression. The paper's own Fig. 8 shows that incorrect CME proposals occur frequently, yet no precision/recall statistics for the detection module are reported. The +1.3 J&F gain in Table 2 is the only quantitative evidence for CME, and the inference-time fusion in Eq. (10) depends on an undisclosed hyperparameter λ. Please report CME detection precision/recall, specify the value of λ and a sensitivity analysis, and clarify whether the CME's outputs are used during training to update the memory bank (to rule out train/inference distribution shift).
  2. [Tables 1-3] No variance or significance testing is reported. All results appear to be from single runs, and several comparisons are close: SAMWISE with CLIP-B achieves 67.2 J&F on Ref-YouTube-VOS versus 67.5 for MUTR, and the CME ablation is +1.3 J&F. Without error bars or multiple seeds, it is not possible to tell whether these gaps are meaningful, especially for the +1.3 CME gain and the +1.1 hierarchical patch-size gain in Table 3. Please provide mean and standard deviation over at least three seeds, or a paired significance test, for the main tables and the key ablations.
  3. [Sec. 4.2, Table 2] The ablation table is not self-explanatory. The text says 'the first line reports the result using the naive solution' and '+5.1% with both adapters', but the checkmark columns (MLP-only, Text-to-Visual, Visual-to-Text, HSA, CME) are not mapped row-by-row in the text, so the reader cannot determine which row corresponds to MLP plus one adapter versus MLP plus both adapters. Please add explicit row labels (e.g., 'MLP', '+T2V', '+V2T', '+HSA', '+CME') to make the incremental contributions unambiguous.
  4. [Sec. 3.5] The assumption that memory-less features F are 'unbiased' and 'aligned with the textual prompt via CMT' is stated as an observation but is not quantitatively supported. The paper provides a PCA visualization in Fig. 5 but no metric measuring text-visual alignment of F before and after CMT. Since the correctness of the CME's candidate proposals rests on this assumption, please provide a quantitative alignment measure (e.g., retrieval accuracy of τ_l against the caption, or the IoU of the argmax proposal with the ground-truth object) to justify the 'unbiased' claim.
minor comments (5)
  1. [Eq. (10)] The definition of M(h,w) as '1(h,w)[h,w: P_l > 0]' is confusing; it should simply be the binary mask M = 1_{P_l > 0}. Also, λ is never given a value in the paper or the supplementary material.
  2. [Abstract and Table 1] The abstract states the method 'achieves state-of-the-art across various benchmarks' with 'less than 5M parameters,' but Table 1 shows that the CLIP-B variant (67.2 J&F on Ref-YouTube-VOS) is below MUTR (67.5). The claim should specify that the state-of-the-art results are obtained with the RoBERTa-based variant, or the CLIP result should be contextualized.
  3. [Supplementary, Tab. 5] The 'Adapter layers' ablation table is hard to read: the row 'Layer 1 Layer 2 Layer 3 Params J&F' followed by checkmarks does not clearly indicate which Hiera layer corresponds to which column. Please make the layer indices explicit.
  4. [Sec. 3.3] The acronyms TVA and VTA are introduced with inconsistent spacing ('TV A' in one place) and the two cross-attention directions are easy to confuse; consider using a single consistent notation such as 'T2V' and 'V2T'.
  5. [Fig. 8] The figure demonstrates incorrect CME proposals but provides no quantitative estimate of how often this happens; adding a success/failure rate or a confusion matrix in the supplementary would strengthen the discussion of CME's robustness.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: SAMWISE's SOTA claim rests on external benchmarks, and the CME self-supervision is a training-signal design choice, not an analytic shortcut.

full rationale

The paper's central claim is empirical: a frozen SAM2 plus ~5M trainable parameters (CMT adapters, prompt MLP, and CME) achieves state-of-the-art J&F on MeViS, Ref-YouTube-VOS, and Ref-DAVIS. There is no derivation of benchmark numbers from fitted constants, no load-bearing self-citation chain, and no imported uniqueness theorem. The only self-referential element is the CME training signal (Supp. Eq. 13), where the label y_t = 1[Y_l ∩ Y_m = ∅] is computed from two masks produced by the same frozen SAM2 decoder. This is a self-supervised proxy for mask disagreement, not a circular derivation of the reported +1.3 J&F gain: the gain is measured on an external benchmark, and the paper's own 'Always' baseline (50.7 vs. 54.2 without CME) shows the correction is not beneficial by construction. The paper also explicitly acknowledges that CME proposals can be incorrect (Supp. Fig. 8), which is a robustness concern about the unverified assumption that memory-less features are text-aligned, not a circularity in the argument. All benchmark comparisons are against external methods and external evaluation servers, so the SOTA claim is independently checkable. The derivation chain from architecture to experimental result is non-circular; the CME's self-supervision merely raises a correctness-risk question, not a circularity finding.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The central claim rests mostly on empirical comparisons; the design includes several tuned hyperparameters and domain assumptions about SAM2's features. No new physical entities are introduced; CMT and CME are learned modules with benchmark and ablation evidence.

free parameters (5)
  • Fusion weight lambda in Eq. 10 = not reported
    Balances memory-based and memory-less masks when CME detects a new object; no value is given in the main text or supplement.
  • HSA patch size hierarchy = 8/4/2 for Hiera-B (16/8/4 variant tested)
    Table 3 shows hierarchical patch sizes beat fixed sizes by +1.1 J&F; selected on MeViS valid-u.
  • CMT hidden dimensionality = 256
    Table 5-bottom: 256 gives best J&F 54.2 among tested values; 384 doubles parameters and drops performance.
  • Number of CMT adapter layers = 3
    Table 5-top: three adapters give best J&F 54.2 vs 52.1 for two layers.
  • Clip length T = 8
    Table 6: T=8 gives 54.2, T=12 gives 54.3, so T=8 is chosen for the online setting.
assumptions (4)
  • domain assumption HSA assumes object motion across adjacent frames is spatially localized, so self-attention within T x P x P patches captures motion context without global attention.
    Introduced in Sec. 3.3 and Fig. 4; the gain from HSA in Table 2 validates it empirically on MeViS, but it is a modeling assumption about video motion.
  • domain assumption Memory-less SAM2 features are unbiased relative to the tracked object and are aligned with the text prompt by CMT, so a cross-attention token can propose the correct object.
    Sec. 3.5 states this explicitly; if false, CME would inject wrong candidates into the memory bank.
  • domain assumption Zero mask intersection between memory-based and memory-less predictions indicates two different objects (Supp. Eq. 13), giving a valid self-supervised label for CME.
    The label ignores partial overlaps and uses the model's own predictions as ground truth.
  • domain assumption The frozen SAM2 Mask Decoder can be prompted with a learned projection of text embeddings without degrading its original segmentation capability.
    Sec. 3.2 and 3.4; the paper preserves SAM2 weights, so this assumption is load-bearing for the no-fine-tuning claim.
invented entities (2)
  • Cross-Modal Temporal Adapter (CMT) independent evidence
    purpose: Injects cross-modal and temporal cues into frozen SAM2 and text encoder features.
    Described in Sec. 3.3; ablations show CMT components improve J&F from 45.2 to 54.2, and code is released.
  • Conditional Memory Encoder (CME) independent evidence
    purpose: Detects when memory-less features indicate a new text-aligned object and softly encodes its mask into the memory bank.
    Described in Sec. 3.5; ablation shows +1.3 J&F on MeViS, qualitative examples in Figs. 7-8, and code is released.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation." pith.science (2026). https://pith.science/paper/MX7TIZY3

@misc{pith2026241117646,
  author       = {Pith},
  title        = {Pith review of: SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MX7TIZY3}},
  note         = {Machine review of arXiv:2411.17646}
}
read the original abstract

Referring Video Object Segmentation (RVOS) relies on natural language expressions to segment an object in a video clip. Existing methods restrict reasoning either to independent short clips, losing global context, or process the entire video offline, impairing their application in a streaming fashion. In this work, we aim to surpass these limitations and design an RVOS method capable of effectively operating in streaming-like scenarios while retaining contextual information from past frames. We build upon the Segment-Anything 2 (SAM2) model, that provides robust segmentation and tracking capabilities and is naturally suited for streaming processing. We make SAM2 wiser, by empowering it with natural language understanding and explicit temporal modeling at the feature extraction stage, without fine-tuning its weights, and without outsourcing modality interaction to external models. To this end, we introduce a novel adapter module that injects temporal information and multi-modal cues in the feature extraction process. We further reveal the phenomenon of tracking bias in SAM2 and propose a learnable module to adjust its tracking focus when the current frame features suggest a new object more aligned with the caption. Our proposed method, SAMWISE, achieves state-of-the-art across various benchmarks, by adding a negligible overhead of less than 5 M parameters. Code is available at https://github.com/ClaudiaCuttano/SAMWISE .

Figures

Figures reproduced from arXiv: 2411.17646 by the authors.

Figure 1
Figure 1. SAMWISE. Our approach infuses knowledge about natural language in the Segment-Anything 2 model, adding explicit temporal cues in the feature extraction for the task of streaming-based Referring Video Segmentation (RVOS). We use a learnable mechanism to mitigate the so-called tracking bias, i.e. SAM2 tendency to overlook a correct object once it becomes identifiable, due to its ongoing tracking of a different object.… view at source ↗
Figure 2
Figure 2. Overview of SAMWISE. We build on a frozen SAM2 and a frozen Text Encoder to segment images in video given a textual description. We incorporate the Cross-Modal Temporal Adapter (CMT) into the text and visual encoders at every intermediate layer k to model temporal dynamics within visual features while contaminating each modality with the other. Then, we extract the [CLS] and verb embeddings, namely Contextual and Mo… view at source ↗
Figure 3
Figure 3. Architecture of our Cross Modal Temporal (CMT) Adapter, made up of Hierarchical Selective Attention (HSA) to model temporal cues, a Visual-to-Text Attention (VTA) and Text￾to-Visual Attention (VTA) modules. Overview. We first provide a brief discussion of the SAM2 model (Sec. 3.1). We then outline the pipeline of our pro￾posed SAMWISE, starting from the prompting strategy in Sec. 3.2. In Sec. 3.3, we detail our nove… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Cross Modal Temporal Adapter: we show via PCA that our CMT provides contextualized visual features based on the given textual prompt, compared to SAM2 original ones. cues (e.g., actions), facilitating early alignment with the prompt, as visible in [PITH_FULL_IMAGE:fig…
Figure 6
Figure 6. Figure 6: Effect of our Conditional Memory Encoder. The cap￾tion above requires disambiguating multiple instances of the same class (e.g., “cat”) by identifying a specific action (e.g., “climb￾ing”). Since none of the instances perform this action initially, the model begins tra…
Figure 7
Figure 7. Figure 7: Correct CME detections. The plot shows examples where our CME correctly identifies (green masks) the referred object when the action starts unfolding. SAMWISE recognizes that the newly proposed object is more aligned with the query and thus switches its tracking focus …
Figure 8
Figure 8. Figure 8: Incorrect CME detections.The plot shows examples where our CME provides wrong object proposals (red masks) due to lack of contextual information. In these examples, SAMWISE determines that, when tacking into account past video context, the previously object is more ali…
Figure 9
Figure 9. Figure 9: Effect of Tracking bias. The figure shows how memory features (PCA) reinforce the initial choice, leading to tracking bias and preventing focus to more semantically aligned objects. In the first row, the model fails to shift attention when the correct object begins the…
Figure 10
Figure 10. Figure 10: Qualitative examples from MeViS. The figure highlights SAMWISE ability to handle challenging RVOS scenarios, including occlusions, multiple instances, and distinguishing between similar objects based on actions and descriptive attributes. Method Text Encoder Referring…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    The paper introduces a 3,254-video benchmark with pixel-level artifact masks for AI-generated video, and reports that fine-tuning on it improves artifact localization.

  2. ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ReSurgSAM2 reports large gains in surgical referring video segmentation by adding a cross-modal Mamba detection stage, credible initial frame selection, and a diversity-driven long-term memory to SAM2, at 61.2 FPS.

Reference graph

Works this paper leans on

58 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    One token to seg them all: Language in- structed reasoning segmentation in videos

    Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Lei Liu, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Language in- structed reasoning segmentation in videos. arXiv preprint arXiv:2409.19603, 2024. 3, 7, 8

    arXiv 2024

  2. [2]

    A closer look at referring expressions for video object segmentation

    Miriam Bellver, Carles Ventura, Carina Silberer, Ioannis Kazakos, Jordi Torres, and Xavier Giro-i Nieto. A closer look at referring expressions for video object segmentation. Multimedia Tools and Applications, 82(3):4419–4438, 2023. 2, 6, 5

    work page 2023

  3. [3]

    End-to-end referring video object segmentation with multi- modal transformers

    Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multi- modal transformers. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 4985–4995, 2022. 1, 2, 7

    work page 2022

  4. [4]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 2

    2020

  5. [5]

    Adaptformer: Adapting vision transformers for scalable visual recogni- tion

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recogni- tion. Advances in Neural Information Processing Systems , 35:16664–16678, 2022. 8

    work page 2022

  6. [6]

    Mask grounding for referring image seg- mentation

    Yong Xien Chng, Henry Zheng, Yizeng Han, Xuchong Qiu, and Gao Huang. Mask grounding for referring image seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26573– 26583, 2024. 4

    work page 2024

  7. [7]

    Vlt: Vision-language transformer and query generation for referring segmentation

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vlt: Vision-language transformer and query generation for referring segmentation. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(6):7900–7916, 2022. 5

    work page 2022

  8. [8]

    MeViS: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. InICCV, 2023. 1, 2, 3, 6, 7, 8, 5

    work page 2023

Show all 58 references
  1. [9]

    Progressive multimodal interaction network for referring video object segmentation

    Zihan Ding, Tianrui Hui, Shaofei Huang, Si Liu, Xuan Luo, Junshi Huang, and Xiaoming Wei. Progressive multimodal interaction network for referring video object segmentation. The 3rd Large-scale Video Object Segmentation Challenge , 8(10), 2021. 2

    2021

  2. [10]

    Actor and action video segmentation from a sentence

    Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. Actor and action video segmentation from a sentence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5958–5966, 2018. 1, 2

    2018

  3. [11]

    Html: Hybrid temporal-scale mul- timodal learning framework for referring video object seg- mentation

    Mingfei Han, Yali Wang, Zhihui Li, Lina Yao, Xiaojun Chang, and Yu Qiao. Html: Hybrid temporal-scale mul- timodal learning framework for referring video object seg- mentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13414–13423, 2023. 1, 2, 7

    2023

  4. [12]

    Decoupling static and hier- archical motion perception for referring video segmentation

    Shuting He and Henghui Ding. Decoupling static and hier- archical motion perception for referring video segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13332–13341, 2024. 2, 3, 7

    2024

  5. [13]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019. 2, 3, 4

    2019

  6. [14]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 8

    2022

  7. [15]

    Temporal context enhanced referring video object seg- mentation

    Xiao Hu, Basavaraj Hampiholi, Heiko Neumann, and Jochen Lang. Temporal context enhanced referring video object seg- mentation. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision (WACV), pages 5574–5583, 2024. 7, 3

    2024

  8. [16]

    Cross-modal adapter for text-video retrieval

    Haojun Jiang, Jianke Zhang, Rui Huang, Chunjiang Ge, Zan- lin Ni, Jiwen Lu, Jie Zhou, Shiji Song, and Gao Huang. Cross-modal adapter for text-video retrieval. arXiv preprint arXiv:2211.09623, 2022. 3

    2022 arXiv

  9. [17]

    Mv-adapter: Multimodal video transfer learning for video text retrieval

    Xiaojie Jin, Bowen Zhang, Weibo Gong, Kai Xu, Xueqing Deng, Peng Wang, Zhao Zhang, Xiaohui Shen, and Jiashi Feng. Mv-adapter: Multimodal video transfer learning for video text retrieval. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pa...

    2024

  10. [18]

    Video object segmentation with language referring expressions

    Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. In Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Re- vised Selected Papers, Part IV 14, pages 123–141. Springer,

    2018

  11. [19]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 4

    2023

  12. [20]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9579–9589, 2024. 2, 3, 7, 8

    2024

  13. [21]

    Refsam: Efficiently adapting segmenting anything model for referring video object segmentation, 2024

    Yonglin Li, Jing Zhang, Xiao Teng, Long Lan, and Xin- wang Liu. Refsam: Efficiently adapting segmenting anything model for referring video object segmentation, 2024. 3, 7

    2024

  14. [22]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  15. [23]

    Revisiting temporal modeling for clip-based image-to-video knowledge transferring

    Ruyang Liu, Jingjia Huang, Ge Li, Jiashi Feng, Xinglong Wu, and Thomas H Li. Revisiting temporal modeling for clip-based image-to-video knowledge transferring. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6555–6564, 2023. 4

    2023

  16. [24]

    Li, Ying Shan, and Ge Li

    Ruyang Liu, Chen Li, Yixiao Ge, Thomas H. Li, Ying Shan, and Ge Li. Bt-adapter: Video conversation is fea- sible without video instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13658–13667, 2024. 4

    2024

  17. [25]

    Cross-modal progressive comprehension for referring segmentation

    Si Liu, Tianrui Hui, Shaofei Huang, Yunchao Wei, Bo Li, and Guanbin Li. Cross-modal progressive comprehension for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4761–4775, 2021. 1, 2

    2021

  18. [26]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3, 7, 4

    2023 arXiv

  19. [27]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692,

    1907 arXiv

  20. [28]

    Uniadapter: Uni- fied parameter-efficient transfer learning for cross-modal modeling

    Haoyu Lu, Yuqi Huo, Guoxing Yang, Zhiwu Lu, Wei Zhan, Masayoshi Tomizuka, and Mingyu Ding. Uniadapter: Uni- fied parameter-efficient transfer learning for cross-modal modeling. arXiv preprint arXiv:2302.06605, 2023. 3

    2023 arXiv

  21. [29]

    Soc: Semantic-assisted object cluster for referring video object segmentation

    Zhuoyan Luo, Yicheng Xiao, Yong Liu, Shuyan Li, Yi- tong Wang, Yansong Tang, Xiu Li, and Yujiu Yang. Soc: Semantic-assisted object cluster for referring video object segmentation. Advances in Neural Information Processing Systems, 36, 2024. 1, 3, 7

    2024

  22. [30]

    Spectrum-guided multi-granularity referring video object segmentation

    Bo Miao, Mohammed Bennamoun, Yongsheng Gao, and Ajmal Mian. Spectrum-guided multi-granularity referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 920– 930, 2023. 2

    2023

  23. [31]

    Mod- eling context between objects for referring expression under- standing

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Mod- eling context between objects for referring expression under- standing. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 792–807. Springer,

    2016

  24. [32]

    Video object segmentation using space-time memory networks

    Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9226–9235, 2019. 2

    2019

  25. [33]

    Keeping your eye on the ball: Tra- jectory attention in video transformers

    Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on the ball: Tra- jectory attention in video transformers. Advances in neural information processing systems, 34:12493–12506...

    2021

  26. [34]

    To tune or not to tune? adapting pretrained representations to diverse tasks

    Matthew E Peters, Sebastian Ruder, and Noah A Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.05987, 2019. 3

    1903 arXiv

  27. [35]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  28. [36]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 2, 4

    2024 arXiv

  29. [37]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 ,

    arXiv

  30. [38]

    Hi- era: A hierarchical vision transformer without the bells-and- whistles

    Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hi- era: A hierarchical vision transformer without the bells-and- whistles. In International Conference on Machine Learn...

    2023

  31. [39]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In European Conference on Com- puter Vision, 2020. 1, 2, 7

    2020

  32. [40]

    Temporal collection and distribution for referring video object segmentation

    Jiajin Tang, Ge Zheng, and Sibei Yang. Temporal collection and distribution for referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15466–15476, 2023. 1

    2023

  33. [41]

    A multimodal, multi-task adapting frame- work for video action recognition

    Mengmeng Wang, Jiazheng Xing, Boyuan Jiang, Jun Chen, Jianbiao Mei, Xingxing Zuo, Guang Dai, Jingdong Wang, and Yong Liu. A multimodal, multi-task adapting frame- work for video action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5517– ...

    2024

  34. [42]

    Image as a foreign language: Beit pretraining for all vision and vision- language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision- language tasks. arXiv preprint arXiv:2208.10442, 2022. 2

    2022 arXiv

  35. [43]

    OnlineRefer: A simple online baseline for referring video object segmentation

    Dongming Wu, Tiancai Wang, Yuang Zhang, Xiangyu Zhang, and Jianbing Shen. OnlineRefer: A simple online baseline for referring video object segmentation. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 2761–2770, 2023. 2, 3, 7, 8

    2023

  36. [44]

    Language as queries for referring video object seg- mentation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4974– 4984, 2022. 1, 2, 7, 3

    2022

  37. [45]

    Bridging vision and language en- coders: Parameter-efficient tuning for referring image seg- mentation

    Zunnan Xu, Zhihong Chen, Yong Zhang, Yibing Song, Xi- ang Wan, and Guanbin Li. Bridging vision and language en- coders: Parameter-efficient tuning for referring image seg- mentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17503–17512, 2023. 3

    2023

  38. [46]

    Visa: Reasoning video object segmentation via large language models

    Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. arXiv preprint arXiv:2407.11325, 2024. 3, 7, 8

    2024 arXiv

  39. [47]

    Referred by multi-modality: A unified tem- poral transformer for video object segmentation

    Shilin Yan, Renrui Zhang, Ziyu Guo, Wenchao Chen, Wei Zhang, Hongyang Li, Yu Qiao, Hao Dong, Zhongjiang He, and Peng Gao. Referred by multi-modality: A unified tem- poral transformer for video object segmentation. InProceed- ings of the AAAI Conference on Artificial Intelligen...

    2024

  40. [48]

    Mma: Multi-modal adapter for vision-language models

    Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, and Xiao- hua Xie. Mma: Multi-modal adapter for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23826– 23837, 2024. 3

    2024

  41. [49]

    Cross-modal self-attention network for referring image seg- mentation

    Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image seg- mentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10502– 10511, 2019. 1, 2

    2019

  42. [50]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016. 7

    2016

  43. [51]

    Evf- sam: Early vision-language fusion for text-prompted seg- ment anything model

    Yuxuan Zhang, Tianheng Cheng, Rui Hu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, Xinggang Wang, et al. Evf- sam: Early vision-language fusion for text-prompted seg- ment anything model. arXiv preprint arXiv:2406.20076 ,

    arXiv

  44. [52]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2, 8 SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation Supple...

    2023 arXiv

  45. [53]

    We train our Conditional Mem- ory Encoder (CME) via self-supervision

    Training protocol Following [44], we train our model with a combination of DICE loss and binary mask focal loss. We train our Conditional Mem- ory Encoder (CME) via self-supervision. For each video clip, given the prompt ρ we compute the predicted masks using SAM2 Mask Decoder...

  46. [54]

    CME: Qualitative impact In this section, we analyze the impact of the Conditional Mem- ory Encoder (CME) within SAMWISE. In Fig. 7 and Fig. 8, the model is tasked to segment the correct object in the video based on the provided referring expression. We use yellow masks to rep-...

  47. [55]

    9, where we plot the memory features

    Tracking Bias We provide additional qualitative examples to further exemplify the effect of tracking bias, as visualized in Fig. 9, where we plot the memory features. Tracking bias occurs when the model mis- takenly focuses on an incorrect object, failing to transition its at-...

  48. [56]

    Additional Ablations Number of CMT adapters. In Tab. 5-top we assess how the number of adapters influences performance. Without any adapter (i.e. relying only on a learnable MLP to project text prompts), the model achieves a modestJ &F of 45.2%. Adding a single adapter at the ...

  49. [57]

    SAMWISE vs naive baselines with SAM2 In Tab. 9, we compare SAMWISE with two baselines utilizing SAM2: • GroundingDINO + SAM2 1st frame: This approach employs GroundingDINO [26] to identify the referred object in the first frame based on the textual query. The resulting boundin...

  50. [58]

    10, we present qualitative examples from the MeViS dataset that highlight the effectiveness of SAMWISE

    Qualitative results In Fig. 10, we present qualitative examples from the MeViS dataset that highlight the effectiveness of SAMWISE. These ex- amples cover a range of challenges typical in RVOS. SAMWISE shows strong robustness in dealing with occlusions (case e.), ac- curately ...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.