Pith. sign in

REVIEW 4 major objections 5 minor 61 references

MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that SAM 2 can be adapted to referring video object segmentation by injecting BEiT-3-derived pseudo masks as dense prompts and adding a hierarchical global-historical aggregator, achieving best-reported J&F of 73.9…

desk verdict Solid modules and clean ablations, but the SOTA headline is riding on a bigger backbone than the baselines; the paper needs a matched-scale comparison or a softer claim. read the letter →

arxiv 2501.13667 v5 pith:DDDQXKYH submitted 2025-01-23 cs.CV

Fu Rong , Meng Lan , Qian Zhang , Lefei Zhang This is my paper · ORCID
classification cs.CV
keywords referringvideoobjectsegmentationSAM2maskpriorsdensepromptsglobalcontexttemporalconsistencymultimodalfusionmask-textsimilarityloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Referring video object segmentation (RVOS) asks a model to cut out the object named in a sentence from every frame of a video. This paper tries to establish that SAM 2, a general-purpose promptable video segmenter, can be adapted to that task without retraining its core, by feeding it two kinds of text-derived guidance: per-frame pseudo masks from a joint BEiT-3 encoder as dense prompts, and a global video context fused with SAM 2's memory. The paper reports best-reported J&F scores of 73.9 on Ref-YouTube-VOS, 53.7 on MeViS, and 72.4 on Ref-DAVIS17, ahead of the previous best by 6.8, 7.3, and 2.0 points, and its ablations tie the gains to the proposed mask-prior generator, the hierarchical global-historical aggregator, and a mask-text similarity loss. If the claim holds, referring video segmentation becomes a prompting problem for a frozen video segmenter rather than a retraining problem.

What carries the argument

The load-bearing machinery is a mask prior generator paired with a hierarchical global-historical aggregator. The mask prior generator takes BEiT-3's joint video-text embeddings and [CLS] tokens, lets the video embeddings interact across the whole spatiotemporal sequence with self-attention, lets the class tokens cross-attend to those video embeddings, and element-wise multiplies class tokens back into the video embeddings; an MLP then turns that product into per-frame pseudo masks. Those masks, upsampled and added into SAM 2's prompt encoder, act as dense prompts, while the [CLS] tokens pass through a token MLP as sparse prompts. The aggregator adds a global view to SAM 2's online memory: a pixel-level module runs memory attention and then global attention against a compressed global video feature, and an object-level module cross-attends current mask tokens to the global video feature and to historical mask tokens. A mask-text similarity loss, comparing cosine similarity of sentence embeddings to predicted and ground-truth masks, closes the loop by tying mask predictions back to language.

What would settle it

Replace the generated pseudo masks with random masks (or with masks from a deliberately misaligned encoder) and run the same training recipe on Ref-YouTube-VOS: if J&F does not fall from 73.9 toward the 71.9 of the model without the mask prior generator, then the dense-prompt mechanism is not the actual driver. A second check is to match MPG-SAM 2's parameter count to that of DsHmp or ReferFormer; if the 6.8-point gap largely disappears, the result is mostly scale, not architecture.

Watch

Extended reading notes

Core claim

The central discovery is that SAM 2's weakness on offline referring video object segmentation is not its segmentation machinery but the quality of the prompts and the absence of a global view. By running a BEiT-3 encoder jointly over video frames and the referring sentence, and then generating pseudo masks through spatiotemporal self- and cross-attention on the class and patch tokens, the model produces frame-specific dense prompts that tell SAM 2 where the referent is. A hierarchical global-historical aggregator then upgrades the online memory mechanism: pixel-level fusion makes the current frame feature attend to memory and to a global unified feature, and object-level fusion makes the mask tokens attend to the global video feature and historical mask tokens, so the decoder sees both history and the whole video. With these additions, the authors obtain J&F 73.9 on Ref-YouTube-VOS, 53.7 on MeViS, and 72.4 on Ref-DAVIS17, and step-by-step ablations attribute almost all of the gain from the 69.4 baseline to the three added components.

Load-bearing premise

The load-bearing premise is that BEiT-3's joint video-text embeddings and SAM 2's image features are compatible enough that lightweight attention and MLP fusion align them; if they are not, the pseudo masks become misleading dense prompts and the reported gains would be attributable mostly to the model's larger parameter count rather than to the proposed mechanism.

Editorial extensions

If this is right

  • If the central claim is right, SAM 2's pretrained decoder transfers to referring video segmentation unchanged; the RVOS task becomes one of prompt construction and context aggregation, not re-training the segmenter.
  • A joint multimodal encoder can serve as a general prompt generator for promptable segmenters: its class tokens give sparse prompts and its mask priors give dense prompts, a recipe that should extend to other referring tasks.
  • Giving an online video segmenter a global view through pixel- and object-level fusion of global video features and memory should improve temporal consistency whenever the full video is available at inference time.
  • The mask-text similarity loss is a lightweight training objective that the paper's supplementary experiments show also raises the accuracy of earlier RVOS models such as ReferFormer and SgMg, so it can be applied beyond this architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test, not run in the paper, is swapping BEiT-3 for a different multimodal encoder while keeping SAM 2 frozen; if the recipe is truly modular, the gains should persist roughly unchanged.
  • The global video feature is computed over the whole offline video, so the method as presented does not extend to true streaming settings; a causal variant of the aggregator would be needed for online RVOS.
  • The full model (0.92B parameters) is substantially larger than most baselines (about 0.24B), so the reported gap over prior work is not a clean measure of architecture alone; the paper's smaller 0.46B variant still reaches 69.9 J&F, suggesting scale helps but is not the whole story.
  • Because the dense prompts are generated frame-by-frame from the same encoder, the approach could in principle be applied to referring image segmentation as well; the paper does not test that transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MPG-SAM 2, an adaptation of SAM 2 for referring video object segmentation (RVOS). A BEiT-3 multimodal encoder jointly encodes video frames and text; a mask prior generator (MPG) produces pseudo masks from video embeddings and multimodal [CLS] tokens, used as dense prompts for SAM 2; a hierarchical global-historical aggregator (HGA) injects global and historical context at pixel and object levels; a mask-text similarity loss is also introduced. Experiments on Ref-YouTube-VOS, MeViS, and Ref-DAVIS17 report J&F scores of 73.9, 53.7, and 72.4 respectively, stated as state-of-the-art. Ablations on Ref-YouTube-VOS show each component contributes 0.9 to 2.3 J&F over a same-backbone baseline.

Significance. If the results hold, the paper demonstrates a practical recipe for adapting SAM 2 to language-conditioned video segmentation: dense mask priors generated from aligned multimodal features plus global-historical context aggregation. The strengths are the systematic same-backbone ablations (Tables 3, 4, 7), official-server evaluation on two datasets, and release of code. The significance is however tempered by a scale confound: the 69.4 J&F baseline (Table 3) already exceeds every prior method listed in Table 1, and the full model uses a 0.92B-parameter combination of SAM 2-Hiera-Large and BEiT-3-Large. Thus the contribution of the proposed components is credible, but the claimed superiority over prior RVOS methods is not cleanly attributable to the architecture.

major comments (4)
  1. [Section 4.3, Table 6] The state-of-the-art claim is confounded by model scale. The same-backbone baseline in Table 3 reaches 69.4 J&F on Ref-YouTube-VOS, which already surpasses every prior method in Table 1 (best prior is MUTR at 68.4). The full model is 0.92B parameters, while most compared methods (ReferFormer, SgMg, DsHmp) are around 0.24B. The supplementary 'Tiny' variant (0.46B) still uses the SAM 2-Hiera-Large image encoder, so it does not provide a matched-scale control. The reported SOTA gaps (+6.8 on Ref-YouTube-VOS, +7.3 on MeViS) cannot be attributed to the proposed modules rather than to the stronger pretrained backbone. Please add an experiment with a smaller SAM 2 variant (e.g., Hiera-B or Hiera-S) or at least explicitly state that the gains over prior methods are partly due to the larger backbone.
  2. [Section 4.4, Tables 3-4] All results are single runs without error bars or significance tests. Component contributions range from 0.4 to 2.3 J&F, and the training uses batch size 1 for fine-tuning, which can be noisy. To support the claim that each module helps, please report the mean and standard deviation over at least three seeds, or provide a significance test. Without this, the 0.9-point improvement from L_sim and the 0.4-0.6-point differences in Table 4 may lie within run-to-run variance.
  3. [Section 4.4, Table 7] The ablation setup for removing MPG is ambiguous. The text says 'when the MPG module is omitted, only the mask prior is not generated, while the global video feature production remains.' However, the global video feature V_g is an output of the mask prior generator described in Section 3.3 and Eq. (3). It is unclear what produces V_g in the ablation without MPG. This ambiguity directly affects the interpretability of the HGA-only row (72.3 J&F). Please clarify the exact configuration used for the MPG-omitted ablation.
  4. [Supplementary, Table 6] The parameter-efficiency discussion is misleading because 'MPG-SAM 2-Tiny' replaces only BEiT-3-Large with BEiT-3-Base while retaining the SAM 2-Hiera-Large encoder. The sentence 'Compared to previous methods with relatively small parameter sizes... our low-configuration model exhibits a slightly larger parameter count' understates that 0.46B is still roughly double the 0.24B of ReferFormer and SgMg. A proper comparison at comparable scale requires a smaller SAM 2 image encoder as well.
minor comments (5)
  1. [Section 4.3] Please state explicitly how the Ref-DAVIS17 results are obtained: unlike Ref-YouTube-VOS and MeViS, this dataset is evaluated locally, not on an official server. This information is relevant for comparability.
  2. [Section 3.3, Eq. (2)] The notation for the residual connection in V' = R(V) + MHSA(R(V)) is ambiguous because R(V) appears both as the input to MHSA and as the residual term. Please define R clearly and consider writing the equation with separate symbols for the reshaped input.
  3. [Section 3.6, Eq. (5)] The sentence embeddings T_s are described as 'dimensionally compressed to a singular scalar through MLP layers, subsequently expanded to match the dimensions of the mask.' This is an unusual way to obtain a dense similarity map; please clarify whether the MLP outputs a 2D map directly or a scalar that is then broadcast.
  4. [Section 4.2] The fine-tuning uses batch size 1 with 5 frames per video; please report whether gradient accumulation is used, since batch size 1 can lead to high variance and is worth clarifying for reproducibility.
  5. [Table 5] The generalizability experiments show a 0.8 J&F gain from L_sim on both ReferFormer and SgMg. As with the main ablations, these are single runs; the conclusion 'confirming the effectiveness' would be stronger with variance estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: benchmark comparisons and ablations are self-contained; parameter-scale confounding is an external-validity concern, not a circular derivation.

full rationale

The paper's central results are empirical benchmark scores against held-out validation sets with fixed ground truth, submitted to official servers (Section 4.1). The proposed modules (mask prior generator, hierarchical global-historical aggregator, mask-text similarity loss) are trained with standard supervised losses, including DICE, focal, and the mask-text similarity loss, the latter explicitly computed against ground-truth masks during training (Eqs. 4-6). No fitted constant is renamed as a prediction, and no result is defined in terms of its own output. The ablation study (Tables 3, 4, 7) compares architectural variants under the same training setup, providing independent evidence for the module contributions. Self-citations appear only in related work (Refs. 23, 39) and are not load-bearing for the derivation. The concern that the 0.92B model outperforms smaller baselines due to parameter scale is a fairness/confound issue about SOTA attribution, not a circularity in the derivation chain. Therefore no circular step can be exhibited from the paper's equations or cited premises.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical performance claim, so the ledger records the hand-chosen hyperparameters and background assumptions the benchmark numbers rely on. There are no physically invented entities; the new modules are compositions of standard attention/MLP blocks.

free parameters (5)
  • Loss weights (lambda_dice=5, lambda_focal=2, lambda_sim=2) = 5, 2, 2
    Set by hand in Section 4.2; the central loss in Eq. 4 and the ablation gains depend on these weights.
  • HGA global-feature patch size p_g = 2
    Chosen by validation sweep over 1, 2, 4 in Table 4; affects pixel-level fusion resolution.
  • HGA fusion layer counts N_p, N_o = N_p=1, N_o=1
    Chosen by validation sweep over 1-3 in Table 8; higher counts reduced J&F.
  • Memory bank capacities = 7 historical mask features, 16 mask tokens
    Copied from SAM 2 configuration (Section 4.2); affects how much history the pixel and object fusion can use.
  • Training frame counts and learning rates = 5 or 8 frames; lr 1e-5, 2e-6, 5e-5
    Chosen per dataset in Section 4.2; part of the training recipe the benchmark results depend on.
assumptions (5)
  • domain assumption Pretrained BEiT-3-Large and SAM 2-Hiera-Large weights transfer to the video frames and text of RVOS benchmarks.
    Section 4.2 initializes all modules from these weights; if transfer fails, the reported results would not follow.
  • domain assumption The official evaluation servers for Ref-YouTube-VOS and MeViS return correct, comparable metrics.
    Section 4.1 relies on server submissions because public validation ground truth is unavailable.
  • domain assumption Global video context, including future frames, can be used in offline RVOS and is beneficial.
    Section 3.4 injects a global video feature into every frame's decoding; the paper assumes this does not harm temporal consistency.
  • domain assumption The learned fusion between BEiT-3 embeddings and SAM 2 features is sufficient to semantically align the two spaces.
    Sections 3.3-3.5 concatenate and attend across the two encoders' features without an explicit alignment objective or analysis.
  • ad hoc to paper The mask-text similarity loss form (cosine between MLP-projected sentence embedding and masks) is a valid auxiliary objective.
    Eq. 5-6 define a new loss designed for this paper; its benefit is shown in ablation but the form is not derived from a standard objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation." pith.science (2026). https://pith.science/paper/DDDQXKYH

@misc{pith2026250113667,
  author       = {Pith},
  title        = {Pith review of: MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDDQXKYH}},
  note         = {Machine review of arXiv:2501.13667}
}
read the original abstract

Referring video object segmentation (RVOS) aims to segment objects in a video according to textual descriptions, which requires the integration of multimodal information and temporal dynamics perception. The Segment Anything Model 2 (SAM 2) has shown great effectiveness across various video segmentation tasks. However, its application to offline RVOS is challenged by the translation of the text into effective prompts and a lack of global context awareness. In this paper, we propose a novel RVOS framework, termed MPG-SAM 2, to address these challenges. Specifically, MPG-SAM 2 employs a unified multimodal encoder to jointly encode video and textual features, generating semantically aligned video and text embeddings, along with multimodal class tokens. A mask prior generator utilizes the video embeddings and class tokens to create pseudo masks of target objects and global context. These masks are fed into the prompt encoder as dense prompts along with multimodal class tokens as sparse prompts to generate accurate prompts for SAM 2. To provide the online SAM 2 with a global view, we introduce a hierarchical global-historical aggregator, which allows SAM 2 to aggregate global and historical information of target objects at both pixel and object levels, enhancing the target representation and temporal consistency. Extensive experiments on several RVOS benchmarks demonstrate the superiority of MPG-SAM 2 and the effectiveness of our proposed modules. The code is available at https://github.com/rongfu-dsb/MPG-SAM2.

Figures

Figures reproduced from arXiv: 2501.13667 by the authors.

Figure 1
Figure 1. Comparison of two SAM 2 adaptations for RVOS. (a) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of the proposed MPG-SAM 2. It mainly consists of four parts: the multimodal encoder, the mask prior generator, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The structure of the hierarchical global-historical aggre [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization result on Ref-YouTube-VOS. (a) SgMg [ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Additional visualization results on several datasets. (a), (b) MeViS, (c), (d) Ref-YouTube-VOS. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 40 canonical work pages

  1. [1]

    Xmem++: Production-level video segmentation from few annotated frames

    Maksym Bekuzarov, Ariana Bermudez, Joon-Young Lee, and Hao Li. Xmem++: Production-level video segmentation from few annotated frames. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 635– 644, 2023. 1

    work page 2023

  2. [2]

    Refvos: A closer look at referring expressions for video object segmen- tation.arXiv preprint arXiv:2010.00263, 2020

    Miriam Bellver, Carles Ventura, Carina Silberer, Ioannis Kazakos, Jordi Torres, and Xavier Giro-i Nieto. Refvos: A closer look at referring expressions for video object segmen- tation.arXiv preprint arXiv:2010.00263, 2020. 2

    arXiv 2010

  3. [3]

    End-to-end referring video object segmentation with multi- modal transformers

    Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multi- modal transformers. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4985–4995, 2022. 2

    work page 2022

  4. [4]

    End-to-end referring video object segmentation with multi- modal transformers

    Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multi- modal transformers. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 4985–4995, 2022. 7

    work page 2022

  5. [5]

    End- to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- to-end object detection with transformers. InProceedings of the European Conference on Computer Vision (ECCV), pages 213–229, 2020. 2

    work page 2020

  6. [6]

    Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model

    Ho Kei Cheng and Alexander G Schwing. Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model. InEuropean Conference on Computer Vi- sion, pages 640–658. Springer, 2022. 1

    work page 2022

  7. [7]

    Putting the object back into video object segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3151–3161, 2024. 5

    work page 2024

  8. [8]

    Segment and track anything.arXiv preprint arXiv:2305.06558, 2023

    Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything.arXiv preprint arXiv:2305.06558, 2023. 3

    arXiv 2023

Show all 61 references
  1. [9]

    Unsupervised cross-lingual representation learning at scale.arXiv preprint arXiv:1911.02116, 2019

    A Conneau. Unsupervised cross-lingual representation learning at scale.arXiv preprint arXiv:1911.02116, 2019. 4

    1911 arXiv

  2. [10]

    Vision-language transformer and query generation for refer- ring segmentation

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for refer- ring segmentation. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 16321–16330,

  3. [11]

    Mevis: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 2694–2703, 2023. 2, 6, 7, 10

    2023

  4. [12]

    Language-bridged spatial-temporal interaction for referring video object segmentation

    Zihan Ding, Tianrui Hui, Junshi Huang, Xiaoming Wei, Jizhong Han, and Si Liu. Language-bridged spatial-temporal interaction for referring video object segmentation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4964–4973, 2022. 2, 7

    2022

  5. [13]

    Unified embedding alignment for open-vocabulary video instance segmentation

    Hao Fang, Peng Wu, Yawei Li, Xinxin Zhang, and Xiankai Lu. Unified embedding alignment for open-vocabulary video instance segmentation. InEuropean Conference on Com- puter Vision, pages 225–241. Springer, 2024. 2

    2024

  6. [14]

    Html: Hybrid temporal-scale mul- timodal learning framework for referring video object seg- mentation

    Mingfei Han, Yali Wang, Zhihui Li, Lina Yao, Xiaojun Chang, and Yu Qiao. Html: Hybrid temporal-scale mul- timodal learning framework for referring video object seg- mentation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 13414–13423, 2023. 7

    2023

  7. [15]

    Decoupling static and hier- archical motion perception for referring video segmentation

    Shuting He and Henghui Ding. Decoupling static and hier- archical motion perception for referring video segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13332–13341, 2024. 2, 6, 7

    2024

  8. [16]

    Unleashing the temporal-spatial reasoning capacity of gpt for training-free audio and language referenced video object segmentation.arXiv preprint arXiv:2408.15876, 2024

    Shaofei Huang, Rui Ling, Hongyu Li, Tianrui Hui, Zongheng Tang, Xiaoming Wei, Jizhong Han, and Si Liu. Unleashing the temporal-spatial reasoning capacity of gpt for training-free audio and language referenced video object segmentation.arXiv preprint arXiv:2408.15876, 2024. 2

    2024 arXiv

  9. [17]

    Segment anything in high qual- ity.Advances in Neural Information Processing Systems, 36,

    Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi- Keung Tang, Fisher Yu, et al. Segment anything in high qual- ity.Advances in Neural Information Processing Systems, 36,

  10. [18]

    Video object segmentation with language referring expressions

    Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 123–141, 2018. 6

    2018

  11. [19]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 1, 3

    2023

  12. [20]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 3

    2024

  13. [21]

    Learning to learn better for video object segmentation

    Meng Lan, Jing Zhang, Lefei Zhang, and Dacheng Tao. Learning to learn better for video object segmentation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 1205–1212, 2023. 1

    2023

  14. [22]

    Meng Lan, Fu Rong, Hongzan Jiao, Zhi Gao, and Lefei Zhang. Language query-based transformer with multiscale cross-modal alignment for visual grounding on remote sens- ing images.IEEE Transactions on Geoscience and Remote Sensing, 62:1–13, 2024. 2

    2024

  15. [23]

    Bidirectional correlation-driven inter-frame inter- action transformer for referring video object segmentation

    Meng Lan, Fu Rong, Zuchao Li, Wei Yu, and Lefei Zhang. Bidirectional correlation-driven inter-frame inter- action transformer for referring video object segmentation. Pattern Recognition, 153:110535, 2024. 1, 2

    2024

  16. [24]

    You only infer once: Cross-modal meta-transfer for referring video object segmentation

    Dezhuang Li, Ruoqi Li, Lijun Wang, Yifan Wang, Jinqing Qi, Lu Zhang, Ting Liu, Qingquan Xu, and Huchuan Lu. You only infer once: Cross-modal meta-transfer for referring video object segmentation. InProceedings of the AAAI Con- ference on Artificial Intelligence (AAAI), pages 1...

  17. [25]

    Refsam: Efficiently adapting segmenting any- thing model for referring video object segmentation.arXiv preprint arXiv:2307.00997, 2024

    Yonglin Li, Jing Zhang, Xiao Teng, Long Lan, and Xin- wang Liu. Refsam: Efficiently adapting segmenting any- thing model for referring video object segmentation.arXiv preprint arXiv:2307.00997, 2024. 1

    2024 arXiv

  18. [26]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 3

    2024

  19. [27]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023. 2

    2023 arXiv

  20. [28]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2018. 6

    2018

  21. [29]

    Soc: Semantic-assisted object cluster for referring video object segmentation.Advances in Neural Information Processing Systems, 36, 2024

    Zhuoyan Luo, Yicheng Xiao, Yong Liu, Shuyan Li, Yi- tong Wang, Yansong Tang, Xiu Li, and Yujiu Yang. Soc: Semantic-assisted object cluster for referring video object segmentation.Advances in Neural Information Processing Systems, 36, 2024. 1, 2, 7

    2024

  22. [30]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 6

    2016

  23. [31]

    Visual-textual capsule routing for text-based video segmentation

    Bruce McIntosh, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. Visual-textual capsule routing for text-based video segmentation. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9942–9951, 2020. 2

    2020

  24. [32]

    Spectrum-guided multi-granularity referring video object segmentation

    Bo Miao, Mohammed Bennamoun, Yongsheng Gao, and Ajmal Mian. Spectrum-guided multi-granularity referring video object segmentation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 920– 930, 2023. 1, 2, 6, 7, 8, 9

    2023

  25. [33]

    V-net: Fully convolutional neural networks for volumetric medical image segmentation

    Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. InProceedings of the Inter- national Conference on 3D Vision (3DV), pages 565–571,

  26. [34]

    Video object segmentation using space-time memory networks

    Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. InProceedings of the IEEE/CVF international conference on computer vision, pages 9226–9235, 2019. 1

    2019

  27. [35]

    Semantic and sequential alignment for referring video object segmentation

    Feiyu Pan, Hao Fang, Fangkai Li, Yanyu Xu, Yawei Li, Luca Benini, and Xiankai Lu. Semantic and sequential alignment for referring video object segmentation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 19067–19076, 2025. 2

    2025

  28. [36]

    The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017. 6

    2017 arXiv

  29. [37]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2024

  30. [38]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 1, 6, 9

    2024 arXiv

  31. [39]

    Cus- tomized sam 2 for referring remote sensing image segmenta- tion.arXiv preprint arXiv:2503.07266, 2025

    Fu Rong, Meng Lan, Qian Zhang, and Lefei Zhang. Cus- tomized sam 2 for referring remote sensing image segmenta- tion.arXiv preprint arXiv:2503.07266, 2025. 3

    2025

  32. [40]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. InProceedings of the European Con- ference on Computer Vision (ECCV), pages 208–223, 2020. 2, 6, 7, 9, 10

    2020

  33. [41]

    Temporal collection and distribution for referring video object segmentation

    Jiajin Tang, Ge Zheng, and Sibei Yang. Temporal collection and distribution for referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15466–15476, 2023. 2, 7

    2023

  34. [42]

    Samrs: Scaling-up re- mote sensing segmentation dataset with segment anything model.Advances in Neural Information Processing Systems, 36, 2024

    Di Wang, Jing Zhang, Bo Du, Minqiang Xu, Lin Liu, Dacheng Tao, and Liangpei Zhang. Samrs: Scaling-up re- mote sensing segmentation dataset with segment anything model.Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  35. [43]

    Asymmetric cross-guided attention network for actor and ac- tion video segmentation from natural language query

    Hao Wang, Cheng Deng, Junchi Yan, and Dacheng Tao. Asymmetric cross-guided attention network for actor and ac- tion video segmentation from natural language query. InPro- ceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3939–3948, 2019. 2

    2019

  36. [44]

    Image as a foreign language: Beit pretraining for vision and vision- language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision- language tasks. InProceedings of the IEEE/CVF Conferenc...

    2023

  37. [45]

    Hyperseg: Towards univer- sal visual segmentation with large language model.arXiv preprint arXiv:2411.17606, 2024

    Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu Yang. Hyperseg: Towards univer- sal visual segmentation with large language model.arXiv preprint arXiv:2411.17606, 2024. 9

    2024 arXiv

  38. [46]

    Multi-level representation learning with semantic alignment for referring video object segmentation

    Dongming Wu, Xingping Dong, Ling Shao, and Jianbing Shen. Multi-level representation learning with semantic alignment for referring video object segmentation. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4996–5005, 2022. 2

    2022

  39. [47]

    Onlinerefer: A simple online baseline for referring video object segmentation

    Dongming Wu, Tiancai Wang, Yuang Zhang, Xiangyu Zhang, and Jianbing Shen. Onlinerefer: A simple online baseline for referring video object segmentation. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 2761–2770, 2023. 7

    2023

  40. [48]

    Language as queries for referring video object segmen- tation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmen- tation. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4974–4984,

  41. [49]

    Logiczsl: Exploring logic- induced representation for compositional zero-shot learning

    Peng Wu, Xiankai Lu, Hao Hu, Yongqin Xian, Jianbing Shen, and Wenguan Wang. Logiczsl: Exploring logic- induced representation for compositional zero-shot learning. InProceedings of the Computer Vision and Pattern Recogni- tion Conference, pages 30301–30311, 2025. 2

    2025

  42. [50]

    Efficientsam: Leveraged masked image pretraining for efficient segment anything

    Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xi- ang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. In Proceedings of the IEEE/CVF Conference on Compu...

    2024

  43. [51]

    u-llava: Uni- fying multi-modal tasks via large language model.arXiv preprint arXiv:2311.05348, 2023

    Jinjin Xu, Liwu Xu, Yuzhe Yang, Xiang Li, Fanyi Wang, Yanchun Xie, Yi-Jie Huang, and Yaqian Li. u-llava: Uni- fying multi-modal tasks via large language model.arXiv preprint arXiv:2311.05348, 2023. 3

    2023 arXiv

  44. [52]

    Visa: Reasoning video object segmentation via large language models

    Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. InEuropean Conference on Computer Vision, pages 98–115. Springer, 2024. 6, 7, 9

    2024

  45. [53]

    Referred by multi-modality: A unified tem- poral transformer for video object segmentation

    Shilin Yan, Renrui Zhang, Ziyu Guo, Wenchao Chen, Wei Zhang, Hongyang Li, Yu Qiao, Hao Dong, Zhongjiang He, and Peng Gao. Referred by multi-modality: A unified tem- poral transformer for video object segmentation. InProceed- ings of the AAAI Conference on Artificial Intelligen...

    2024

  46. [54]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016. 6

    2016

  47. [55]

    A simple base- line with single-encoder for referring image segmentation

    Seonghoon Yu, Ilchae Jung, Byeongju Han, Taeoh Kim, Yunho Kim, Dongyoon Wee, and Jeany Son. A simple base- line with single-encoder for referring image segmentation. arXiv preprint arXiv:2408.15521, 2024. 4

    2024 arXiv

  48. [56]

    Losh: Long-short text joint prediction network for referring video object segmentation

    Linfeng Yuan, Miaojing Shi, Zijie Yue, and Qijun Chen. Losh: Long-short text joint prediction network for referring video object segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14001–14010, 2024. 2, 6, 7

    2024

  49. [57]

    Surgicalsam: Efficient class prompt- able surgical instrument segmentation

    Wenxi Yue, Jing Zhang, Kun Hu, Yong Xia, Jiebo Luo, and Zhiyong Wang. Surgicalsam: Efficient class prompt- able surgical instrument segmentation. InProceedings of the AAAI Conference on Artificial Intelligence, pages 6890– 6898, 2024. 3

    2024

  50. [58]

    Faster segment anything: Towards lightweight sam for mo- bile applications.arXiv preprint arXiv:2306.14289, 2023

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mo- bile applications.arXiv preprint arXiv:2306.14289, 2023. 1, 3

    2023 arXiv

  51. [59]

    Evf- sam: Early vision-language fusion for text-prompted seg- ment anything model.arXiv preprint arXiv:2406.20076,

    Yuxuan Zhang, Tianheng Cheng, Rui Hu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, Xinggang Wang, et al. Evf- sam: Early vision-language fusion for text-prompted seg- ment anything model.arXiv preprint arXiv:2406.20076,

    arXiv

  52. [60]

    Deformable detr: Deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. InProceedings of the In- ternational Conference on Learning Representations (ICLR),

  53. [61]

    Exploring pre-trained text- to-video diffusion models for referring video object segmen- tation

    Zixin Zhu, Xuelu Feng, Dongdong Chen, Junsong Yuan, Chunming Qiao, and Gang Hua. Exploring pre-trained text- to-video diffusion models for referring video object segmen- tation. InEuropean Conference on Computer Vision, pages 452–469. Springer, 2024. 2, 6, 7

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.