Pith. sign in

REVIEW 3 major objections 6 minor 61 references

SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Upgrading a frozen SAM with three small trainable modules delivers about 90% of SAM 2.1's promptable video segmentation accuracy at 0.2% of its training cost.

desk verdict A solid, useful SAM-to-video efficiency upgrade whose '0.2% cost' headline is metric-bound (real GPU-hour ratio ~0.7–2%), still worth refereeing. read the letter →

arxiv 2506.01304 v1 pith:ZEQPEIC4 submitted 2025-06-02 cs.CV

Haiyang Mei , Pengyu Zhang , Mike Zheng Shou This is my paper
classification cs.CV
keywords promptablevideosegmentationimage-to-videotransferlearningSegmentAnythingModelSAM2temporalfeatureintegrationmemoryfilteringmaskpropagationtrainingcostreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether promptable video segmentation requires training a video model from scratch on massive data, as SAM 2 does at a cost of 256 A100 GPUs for 108 hours. It answers no, and exhibits SAM-I2V, which keeps a pre-trained image-based SAM frozen and bolts on three small trainable modules: a temporal feature integrator, a relevance-filtered memory associator, and a memory-prompt generator. On four benchmarks the upgraded 18.9M-parameter model scores 65.3 online and 66.0 offline in J&F, about 91% and 90% of SAM 2.1's 71.9 and 73.3, while its training bill of 4.6k G-Hour is about 0.2% of SAM 2.1's 2.2 million. If the claim holds, promptable video segmentation becomes trainable in about a day on a small GPU cluster, opening PVS model design to academic labs and resource-constrained settings.

What carries the argument

The load-bearing machinery is a three-module upgrader attached to a frozen SAM. The Temporal Feature Integrator (TFI) adds a 3D-convolutional temporal branch that runs alongside SAM's static image encoder, slicing out current-frame temporal features and fusing them into each stage's spatial features, turning a per-image encoder into a spatiotemporal one. The Memory Selective Associator (MSA) provides relevance-filtered memory: it separates past frames into short-term local and long-term global pools, computes each frame's similarity to the current frame, normalizes the scores into sampling distributions, and selects z frames to attend to via cross-attention, extending usable history to 20 frames. The Memory Prompt Generator (MPG) implements memory-as-prompt: learnable prompt tokens attend to flattened memory features through masked cross-attention that zeros out background regions, then refine through self-attention and an MLP before joining user prompts in the mask decoder. These modules supply the temporal perception, selective memory retrieval, and memory-conditioned decoding that image-based SAM lacks, and only they plus the mask decoder need training.

What would settle it

Recompute both models' training budgets in FLOPs or in A100-equivalent GPU-hours and check whether SAM-I2V's cost is still about 0.2% of SAM 2.1's; if the recomputed ratio is an order of magnitude larger, the headline number is an artifact of the accounting unit rather than of the method. A complementary check: retrain SAM-I2V with SAM 2.1's additional 62.9k proprietary videos replaced by an equivalent public corpus and see whether the 71.9-versus-65.3 online gap closes, which would attribute the residual gap to data rather than architecture.

Watch

Extended reading notes

Core claim

The central discovery is that the hard part of promptable video segmentation — remembering an object and propagating its mask through dynamic scenes — can be supplied as a small upgrade on top of a frozen image segmentation model rather than learned from scratch. SAM-I2V attaches three components to SAM's encoder-decoder: a Temporal Feature Integrator that runs a 3D-convolutional temporal branch alongside SAM's 2D image encoder and fuses current-frame temporal features into each stage's spatial features; a Memory Selective Associator that, instead of attending to the most recent frames as SAM 2 does, splits past frames into local and global pools, scores each frame's similarity to the current frame, samples the most relevant frames, and runs cross-attention between current and selected memory features; and a Memory Prompt Generator that turns object memories into learnable prompt tokens through foreground-masked cross-attention, feeding them into the mask decoder alongside user clicks, boxes, or masks. Trained for 10 epochs on 8 RTX A5000 GPUs using 10,000 SA-1B images and 50,583 SA-V videos, with no SAM 2 weights, the resulting 18.9M-parameter model achieves 65.3 online and 66.0 offline average J&F across ESD, PUMaVOS, LV-VIS, and SA-V Test — about 91% and 90% of SAM 2.1's 71.9 and 73.3 — at 4.6k versus 2.2 million G-Hour of training cost.

Load-bearing premise

The headline saving counts training cost as GPU number times GPU memory times hours, which treats an A5000 GPU-hour as equal in cost to an A100 GPU-hour; if the two are compared by actual compute performed, the 0.2% figure would be larger, though still a large reduction.

Editorial extensions

If this is right

  • Promptable video segmentation becomes trainable at academic scale: 10 epochs on 8 24GB GPUs, roughly one day, instead of 256 A100 GPUs for 108 hours, while keeping 90-91% of SAM 2.1's average accuracy.
  • The upgrade is backbone-agnostic: applying the same three modules to TinySAM, EdgeSAM, MobileSAM, SlimSAM, and SAM-Base each improves on the corresponding temporal-adapter baseline, so the recipe is not tied to one SAM variant.
  • Relevance-sampled memory beats recency: selecting frames by similarity to the current frame outperforms taking the z most recent frames (69.3 vs. 67.1 overall online 3-click) and extends the memory horizon to 20 frames for roughly 158MB of extra GPU memory.
  • The same model transfers zero-shot to semi-supervised video object segmentation: without VOS training data it reaches 70.2 overall, matching TinySAM+Cutie's 70.6 with fewer than half the parameters.
  • Performance scales with training budget: doubling GPU count or hours raises SA-V Test J&F from 59.3 to between 62.6 and 65.2, indicating the approach is not at a performance ceiling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 0.2% ratio is measured in GPU number times memory times hours, a unit that counts an A5000 GPU-hour as equal to an A100 GPU-hour; a FLOP-based or throughput-normalized comparison would make the saving appear smaller, though the qualitative picture of a large cost reduction would likely survive.
  • The three modules form a general recipe — frozen spatial encoder plus temporal branch plus relevance-filtered memory plus memory tokens as prompts — that could plausibly upgrade other interactive segmenters or foundation tracking models; a direct test would swap the SAM backbone for a non-SAM segmentation network and measure the gain.
  • The memory selector decouples memory quality from memory size, suggesting that long-horizon video models in general may gain more from choosing what to attend to than from attending to everything; the paper's 20-frame memory result supports this hypothesis but does not claim it.
  • Because SAM 2.1 also trained on 62.9k proprietary videos, a clean test of the residual accuracy gap is to train SAM-I2V on an equivalent amount of additional public video data; the paper predicts the gap narrows but does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SAM-I2V, an image-to-video upgrade of SAM that adds a Temporal Feature Integrator (TFI), a Memory Selective Associator (MSA), and a Memory Prompt Generator (MPG) to a pre-trained TinySAM backbone, producing a promptable video segmentation (PVS) model. Trained for 10 epochs on 8 RTX A5000 GPUs (24GB) using SA-1B images and SA-V videos, the method reports online and offline J&F scores of 65.3 and 66.0 averaged over ESD, PUMaVOS, LV-VIS, and SA-V-Test, compared with SAM 2.1's 71.9 and 73.3. The paper claims 'over 90%' of SAM 2.1's performance at 'less than 0.2%' of its training cost. The manuscript includes same-budget baselines, ablations for each component, scalability experiments, and qualitative comparisons; code and models are released.

Significance. If the headline claims are taken at face value, the contribution is practically important: it offers a resource-efficient path to PVS by upgrading a pre-trained image SAM, and the released code and models would facilitate replication and downstream use. The paper is clearly organized and the ablations support the individual contributions of TFI, MSA, and MPG. However, the two headline quantitative claims—'over 90%' accuracy and 'less than 0.2%' training cost—require careful qualification: the accuracy ratio is an average that hides a significantly lower ratio on SA-V-Test, and the cost ratio is computed with a nonstandard metric that is not equivalent to compute, energy, or GPU-hours. These issues do not invalidate the approach, but they must be corrected for the paper's central message to be accurate.

major comments (3)
  1. [§4.2, Table 1 caption] The paper's headline 'less than 0.2% training cost' is computed with the nonstandard cost metric 'GPU number × GPU memory × training hours' (Table 1 caption). Under the standard GPU-hours measure, SAM-I2V uses 8×24 = 192 GPU-hours and SAM 2.1 uses 256×108 = 27,648 GPU-hours, giving a ratio of 0.69%, not 0.2%. Since the RTX A5000 has lower throughput and memory bandwidth than the A100 used for SAM 2.1, a FLOP-based or A100-normalized comparison would raise the ratio further. Even with the paper's own metric, 4.6k / 2.2m = 0.21%, which is not 'less than 0.2%'. The abstract and Section 4.2 should state the comparison in standard units or clearly justify the chosen metric.
  2. [§4.2, Table 1] The claim that the method achieves 'over 90%' of SAM 2.1's performance is based on J&F averaged over four datasets and three click settings. On SA-V-Test, the benchmark most directly aligned with the training data, the online ratio is only about 77% (e.g., 58.2 vs 75.2 at 3 clicks; 50.5 vs 66.3 at 1 click; 59.5 vs 77.2 at 5 clicks). The abstract and introduction should not state 'over 90%' without noting that this is an average and that per-dataset performance on SA-V-Test is substantially lower.
  3. [§3.3, Eqs. (4)-(5)] The memory selector is described as sampling x local and y global frames from a categorical distribution, but the manuscript does not specify how this selection is performed at inference time (e.g., top-k versus stochastic sampling, and if stochastic, with what seed). Since all reported results are deterministic-looking point estimates, the test-time procedure must be stated for reproducibility; the released code may clarify this, but the paper itself should define it.
minor comments (6)
  1. [Figures 9-13 captions] The captions of Figures 9-13 refer to 'SAM-12V' instead of 'SAM-I2V'; please correct this typo.
  2. [Throughout] There are several typographical errors, including 'promtable', 'simplifyiing', 'incuring', and 'propogation'; a careful proofread would improve readability.
  3. [§3.2] The equations describing the Temporal Feature Integrator are not numbered, which makes it difficult to refer to them in the text or in future discussions; numbering them would improve clarity.
  4. [§3.3 vs Table 5] The main text introduces x and y as the numbers of selected local and global frames without stating their ranges, while Table 5 uses specific values; please define the ranges (e.g., x, y ≥ 0 and x + y = z) in the main text.
  5. [References] Reference [24] is incomplete; it lists only the first author's last name ('I Loshchilov') and should include the full author list and publication venue.
  6. [Abstract versus §4.2] The abstract states 'less than 0.2%' while Section 4.2 states '0.2%'; the rounding should be aligned, and the abstract should reflect the exact value under the chosen metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: performance is measured on held-out benchmarks, and the '0.2% training cost' ratio is a unit-dependent report rather than an input-equivalent derivation.

full rationale

The paper's central claims are (i) SAM-I2V reaches roughly 91% and 90% of SAM 2.1's online and offline J&F, and (ii) it does so at 0.2% of SAM 2.1's training cost. Neither claim is produced by fitting the target into the input. All J&F numbers in Tables 1 and 2 are computed on held-out benchmarks (ESD, PUMaVOS, LV-VIS, SA-V-Test) using the standard J&F metric, and the authors state that all modules except the TinySAM backbone are randomly initialized and trained from scratch without any SAM 2 weights. The architecture borrows SAM 2's memory-encoder and memory-attention design and trains on the SA-V dataset, but this is transparent reuse of prior work and data, not circular justification. The only questionable number is the headline '0.2% of training cost', which depends on the paper's nonstandard Cost = GPU number × GPU memory × training hours definition; under plain GPU-hours the ratio is about 0.7%. This is a units and methodology caveat rather than a circular step, because the accuracy measurements are independent of the cost definition and the cost ratio is an arithmetic report, not a derivation from the model's own outputs. The paper also candidly discloses in Section 4.4 that its accuracy does not match SAM 2.1 and that SAM 2.1 was trained on additional proprietary videos. No fitted parameter is renamed as a prediction, no load-bearing self-citation is present, and no uniqueness or ansatz claim is smuggled in via citation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central efficiency claim rests on the paper's chosen resource metric (GPU count x memory x hours), on hand-set constants (z, x, y, g) and on a training budget (10k images, 50,583 videos, 10 epochs) that is not shown to be converged. The method introduces no new physical or conceptual entities, so invented_entities is empty. The axioms are standard engineering assumptions that are reasonable but unproven.

free parameters (4)
  • z (number of memory frames) = 6
    Fixed at 6 following SAM 2; not swept.
  • x, y (number of global and local frames selected) = x=3, y=3
    Selected by ablation on the four test benchmarks (Table 5), where this combination gave the highest OA.
  • g (number of memory prompt tokens) = 3
    Selected by ablation on the four test benchmarks (Table 6); g=3 gave the highest OA, though differences are small.
  • training budget (epochs, GPU-hours, data subset) = 10 epochs, 8x A5000 24GB, 24 hours, 10k SA-1B + 50,583 SA-V
    The 0.2% cost ratio is computed from this specific budget; it is a choice, not a derived quantity.
assumptions (4)
  • domain assumption Pre-trained SAM (TinySAM) image features are a sufficient spatial backbone for video segmentation when augmented with lightweight temporal modules.
    The entire method accepts SAM's encoder as the spatial feature extractor; if these features were insufficient for video-level semantics, the approach would fail. Invoked in Section 3.1 (Overview).
  • domain assumption Training on 50,583 SA-V videos and 10,000 SA-1B images for 10 epochs is sufficient to learn promptable video segmentation.
    The reported results depend on this training budget. Table 10 shows that increasing the budget raises SA-V-Test J&F from 59.3 to 65.2, suggesting the model may be undertrained at the default budget. Section 4.1 and Table 10.
  • domain assumption Dot-product similarity between current and past frame features is a valid proxy for memory relevance.
    MSA selects frames using normalized dot-product similarities (Section 3.3); this is a modeling assumption without theoretical or empirical justification outside the paper's own ablations.
  • ad hoc to paper Training cost is properly measured by GPU number x GPU memory x training hours.
    This metric is introduced in the paper (Table 1 caption) and is not a standard measure; it ignores hardware generation and per-GPU throughput, which is load-bearing for the 0.2% claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost." pith.science (2026). https://pith.science/paper/ZEQPEIC4

@misc{pith2026250601304,
  author       = {Pith},
  title        = {Pith review of: SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZEQPEIC4}},
  note         = {Machine review of arXiv:2506.01304}
}
read the original abstract

Foundation models like the Segment Anything Model (SAM) have significantly advanced promptable image segmentation in computer vision. However, extending these capabilities to videos presents substantial challenges, particularly in ensuring precise and temporally consistent mask propagation in dynamic scenes. SAM 2 attempts to address this by training a model on massive image and video data from scratch to learn complex spatiotemporal associations, resulting in huge training costs that hinder research and practical deployment. In this paper, we introduce SAM-I2V, an effective image-to-video upgradation method for cultivating a promptable video segmentation (PVS) model. Our approach strategically upgrades the pre-trained SAM to support PVS, significantly reducing training complexity and resource requirements. To achieve this, we introduce three key innovations: (i) an image-to-video feature extraction upgrader built upon SAM's static image encoder to enable spatiotemporal video perception, (ii) a memory filtering strategy that selects the most relevant past frames for more effective utilization of historical information, and (iii) a memory-as-prompt mechanism leveraging object memory to ensure temporally consistent mask propagation in dynamic scenes. Comprehensive experiments demonstrate that our method achieves over 90% of SAM 2's performance while using only 0.2% of its training cost. Our work presents a resource-efficient pathway to PVS, lowering barriers for further research in PVS model design and enabling broader applications and advancements in the field. Code and model are available at: https://github.com/showlab/SAM-I2V.

Figures

Figures reproduced from arXiv: 2506.01304 by the authors.

Figure 1
Figure 1. Comparison on promptable video segmentation (PVS). [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of SAM-I2V. The input video is fed into a backbone feature extractor, which passes the current frame through a series of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Structural details of temporal feature integrator (TFI). It [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Illustration of our memory selective associator (MSA). It consists of a memory encoder, a memory selector, and memory attention. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the memory prompt generator (MPG). [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The applicability of our proposed SAM-I2V to upgrade different SAM models for promptable video segmentation (PVS) task [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Architecture details of the mask decoder. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Illustration of the memory selective associator (MSA). [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Visual comparison with state-of-the-art methods on a challenging video sequence from PUMaVOS [ [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Visual comparison with state-of-the-art methods on a challenging video sequence from LV-VIS [ [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Visual comparison with state-of-the-art methods on a challenging video sequence from LV-VIS [ [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Visual comparison with state-of-the-art methods on a challenging video sequence from LV-VIS [ [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Visual comparison with state-of-the-art methods on a challenging video sequence from LV-VIS [ [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 37 canonical work pages

  1. [1]

    Xmem++: Production-level video segmenta- tion from few annotated frames

    Maksym Bekuzarov, Ariana Bermudez, Joon-Young Lee, and Hao Li. Xmem++: Production-level video segmenta- tion from few annotated frames. InICCV, 2023. 1, 3, 6, 7, 8, 11, 13, 16, 17, 18, 19, 20

    work page 2023

  2. [2]

    One- shot video object segmentation

    Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taix´e, Daniel Cremers, and Luc Van Gool. One- shot video object segmentation. InCVPR, 2017. 3

    work page 2017

  3. [3]

    Rsprompter: Learning to prompt for remote sensing instance segmenta- tion based on visual foundation model.IEEE Transactions on Geoscience and Remote Sensing, 2024

    Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang, Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. Rsprompter: Learning to prompt for remote sensing instance segmenta- tion based on visual foundation model.IEEE Transactions on Geoscience and Remote Sensing, 2024. 2

    work page 2024

  4. [4]

    Adaptformer: Adapting vision transformers for scalable visual recognition

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. NeurIPS, 2022. 6, 7, 8, 11, 13, 16, 17, 18, 19, 20

    work page 2022

  5. [5]

    Sam2-adapter: Evaluating & adapting segment any- thing 2 in downstream tasks: Camouflage, shadow, medical image segmentation, and more.arXiv:2408.04579, 2024

    Tianrun Chen, Ankang Lu, Lanyun Zhu, Chaotao Ding, Chu- nan Yu, Deyi Ji, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam2-adapter: Evaluating & adapting segment any- thing 2 in downstream tasks: Camouflage, shadow, medical image segmentation, and more.arXiv:2408.04579, 2024. 2

    arXiv 2024

  6. [6]

    0.1% data makes segment anything slim.NeurIPS,

    Zigeng Chen, Gongfan Fang, Xinyin Ma, and Xinchao Wang. 0.1% data makes segment anything slim.NeurIPS,

    work page

  7. [7]

    Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model

    Ho Kei Cheng and Alexander G Schwing. Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model. InECCV, 2022. 3

    work page 2022

  8. [8]

    Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion

    Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. InCVPR, 2021. 2

    work page 2021

Show all 61 references
  1. [9]

    Rethink- ing space-time networks with improved memory coverage for efficient video object segmentation.NeurIPS, 2021

    Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethink- ing space-time networks with improved memory coverage for efficient video object segmentation.NeurIPS, 2021. 3

    2021

  2. [10]

    Tracking anything with de- coupled video segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. Tracking anything with de- coupled video segmentation. InICCV, 2023. 2

    2023

  3. [11]

    Putting the object back into video object segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. InCVPR, 2024. 3, 6, 7, 11, 13, 16, 17, 18, 19, 20

    2024

  4. [12]

    Segment and track anything.arXiv:2305.06558, 2023

    Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything.arXiv:2305.06558, 2023. 2

    2023 arXiv

  5. [13]

    Electra: Pre-training text encoders as discrimina- tors rather than generators.arXiv:2003.10555, 2020

    K Clark. Electra: Pre-training text encoders as discrimina- tors rather than generators.arXiv:2003.10555, 2020. 6

    2003 arXiv

  6. [14]

    Learning the what and how of annotation in video object segmentation

    Thanos Delatolas, Vicky Kalogeiton, and Dim P Papadopou- los. Learning the what and how of annotation in video object segmentation. InWACV, 2024. 2

    2024

  7. [15]

    Segment anything model 2: an ap- plication to 2d and 3d medical images.arXiv:2408.00756,

    Haoyu Dong, Hanxue Gu, Yaqian Chen, Jichen Yang, and Maciej A Mazurowski. Segment anything model 2: an ap- plication to 2d and 3d medical images.arXiv:2408.00756,

    arXiv

  8. [16]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv:2010.11929,

    Alexey Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv:2010.11929,

    2010 arXiv

  9. [17]

    Interactive video object segmentation using global and local transfer modules

    Yuk Heo, Yeong Jun Koh, and Chang-Su Kim. Interactive video object segmentation using global and local transfer modules. InECCV, 2020. 2

    2020

  10. [18]

    A neuromorphic dataset for object segmentation in indoor cluttered environment.arXiv:2302.06301, 2023

    Xiaoqian Huang, Kachole Sanket, Abdulla Ayyad, Fari- borz Baghaei Naeini, Dimitrios Makris, and Yahya Zweiri. A neuromorphic dataset for object segmentation in indoor cluttered environment.arXiv:2302.06301, 2023. 1, 6, 7, 8, 11, 13

    2023 arXiv

  11. [19]

    Prompting visual-language models for efficient video understanding

    Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. InECCV, 2022. 3

    2022

  12. [20]

    Segment anything in high qual- ity.NeurIPS, 2023

    Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi- Keung Tang, Fisher Yu, et al. Segment anything in high qual- ity.NeurIPS, 2023. 2

    2023

  13. [21]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InICCV, 2023. 1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 16, 17, 18, 19, 20

    2023

  14. [22]

    Frozen clip models are efficient video learners

    Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Frozen clip models are efficient video learners. In ECCV, 2022. 3

    2022

  15. [23]

    Surgical sam 2: Real-time segment anything in surgical video by efficient frame pruning.arXiv:2408.07931,

    Haofeng Liu, Erli Zhang, Junde Wu, Mingxuan Hong, and Yueming Jin. Surgical sam 2: Real-time segment anything in surgical video by efficient frame pruning.arXiv:2408.07931,

    arXiv

  16. [24]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv:1711.05101, 2017. 6

    2017 arXiv

  17. [25]

    Segment anything in medical images.Nature Communications, 2024

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images.Nature Communications, 2024. 2

    2024

  18. [26]

    Video object segmentation without temporal information.IEEE TPAMI, 2018

    K-K Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, Laura Leal-Taix´e, Daniel Cremers, and Luc Van Gool. Video object segmentation without temporal information.IEEE TPAMI, 2018. 3

    2018

  19. [27]

    Segment anything model for medical image analysis: an experimental study

    Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 2023. 2

    2023

  20. [28]

    Fast video object segmentation by reference- guided mask propagation

    Seoung Wug Oh, Joon-Young Lee, Kalyan Sunkavalli, and Seon Joo Kim. Fast video object segmentation by reference- guided mask propagation. InCVPR, 2018. 3

    2018

  21. [29]

    St-adapter: Parameter-efficient image-to-video transfer learning.NeurIPS, 2022

    Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hong- sheng Li. St-adapter: Parameter-efficient image-to-video transfer learning.NeurIPS, 2022. 3, 7, 8

    2022

  22. [30]

    Dual- path adaptation from image to video transformers

    Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Dual- path adaptation from image to video transformers. InCVPR,

  23. [31]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019. 6

    2019

  24. [32]

    The 2017 davis challenge on video object segmentation

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbel´aez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017. 3, 6, 11 9

    2017 arXiv

  25. [33]

    Disentangling spatial and temporal learning for efficient image-to-video transfer learning

    Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Yingya Zhang, Changxin Gao, Deli Zhao, and Nong Sang. Disentangling spatial and temporal learning for efficient image-to-video transfer learning. InICCV, 2023. 3

    2023

  26. [34]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 3

    2021

  27. [35]

    Segment anything meets point tracking.arXiv:2307.01197, 2023

    Frano Raji ˇc, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Mar- tin Danelljan, and Fisher Yu. Segment anything meets point tracking.arXiv:2307.01197, 2023. 2

    2023 arXiv

  28. [36]

    Sam 2: Seg- ment anything in images and videos.arXiv:2408.00714,

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Seg- ment anything in images and videos.arXiv:2408.00714,

    arXiv

  29. [37]

    Seg- ment anything, from space? InWACV, 2024

    Simiao Ren, Francesco Luzi, Saad Lahrichi, Kaleb Kassaw, Leslie M Collins, Kyle Bradbury, and Jordan M Malof. Seg- ment anything, from space? InWACV, 2024. 2

    2024

  30. [38]

    Learning fast and robust target models for video object segmentation

    Andreas Robinson, Felix Jaremo Lawin, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg. Learning fast and robust target models for video object segmentation. In CVPR, 2020. 3

    2020

  31. [39]

    Interactive 3d medical image segmentation with sam 2.arXiv:2408.02635, 2024

    Chuyun Shen, Wenhao Li, Yuhang Shi, and Xiangfeng Wang. Interactive 3d medical image segmentation with sam 2.arXiv:2408.02635, 2024. 2

    2024 arXiv

  32. [40]

    Tinysam: Pushing the envelope for efficient segment any- thing model.arXiv:2312.13789, 2023

    Han Shu, Wenshuo Li, Yehui Tang, Yiman Zhang, Yi- hao Chen, Houqiang Li, Yunhe Wang, and Xinghao Chen. Tinysam: Pushing the envelope for efficient segment any- thing model.arXiv:2312.13789, 2023. 1, 2, 6, 7, 8, 11, 12, 13, 16, 17, 18, 19, 20

    2023 arXiv

  33. [41]

    Towards open-vocabulary video instance segmentation

    Haochen Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang, Xu Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. Towards open-vocabulary video instance segmentation. InICCV,

  34. [42]

    F3net: Fu- sion, feedback and focus for salient object detection

    Jun Wei, Shuhui Wang, and Qingming Huang. F3net: Fu- sion, feedback and focus for salient object detection. In AAAI, 2020. 6

    2020

  35. [43]

    Medical sam adapter: Adapting segment anything model for medical image segmentation

    Junde Wu, Wei Ji, Yuanpei Liu, Huazhu Fu, Min Xu, Yanwu Xu, and Yueming Jin. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv:2304.12620, 2023. 2

    2023 arXiv

  36. [44]

    Scalable video object segmentation with simplified frame- work

    Qiangqiang Wu, Tianyu Yang, Wei Wu, and Antoni B Chan. Scalable video object segmentation with simplified frame- work. InICCV, 2023. 3

    2023

  37. [45]

    Cat-sam: Con- ditional tuning for few-shot adaptation of segment anything model

    Aoran Xiao, Weihao Xuan, Heli Qi, Yun Xing, Ruijie Ren, Xiaoqin Zhang, Ling Shao, and Shijian Lu. Cat-sam: Con- ditional tuning for few-shot adaptation of segment anything model. InECCV, 2024. 2

    2024

  38. [46]

    Sam2-unet: Segment anything 2 makes strong encoder for natural and medical image segmentation.arXiv:2408.08870,

    Xinyu Xiong, Zihuang Wu, Shuangyi Tan, Wenxue Li, Fei- long Tang, Ying Chen, Siying Li, Jie Ma, and Guanbin Li. Sam2-unet: Segment anything 2 makes strong encoder for natural and medical image segmentation.arXiv:2408.08870,

  39. [47]

    Efficientsam: Leveraged masked image pretraining for efficient segment anything

    Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xi- ang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. In CVPR, 2024. 2

    2024

  40. [48]

    Youtube-vos: Sequence-to-sequence video object segmentation

    Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Sequence-to-sequence video object segmentation. InECCV, 2018. 3

    2018

  41. [49]

    Track anything: Segment anything meets videos.arXiv:2304.11968, 2023

    Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything meets videos.arXiv:2304.11968, 2023. 2

    2023 arXiv

  42. [50]

    Efficient video object segmen- tation via network modulation

    Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang, and Aggelos K Katsaggelos. Efficient video object segmen- tation via network modulation. InCVPR, 2018. 3

    2018

  43. [51]

    AIM: Adapting image models for efficient video action recognition

    Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. AIM: Adapting image models for efficient video action recognition. InICLR, 2023. 3

    2023

  44. [52]

    Collaborative video object segmentation by foreground-background inte- gration

    Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by foreground-background inte- gration. InECCV, 2020. 3

    2020

  45. [53]

    Faster segment anything: Towards lightweight sam for mo- bile applications.arXiv:2306.14289, 2023

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mo- bile applications.arXiv:2306.14289, 2023. 2, 11, 12

    2023 arXiv

  46. [54]

    Mobilesamv2: Faster segment anything to everything.arXiv:2312.09579,

    Chaoning Zhang, Dongshen Han, Sheng Zheng, Jinwoo Choi, Tae-Ho Kim, and Choong Seon Hong. Mobilesamv2: Faster segment anything to everything.arXiv:2312.09579,

    arXiv

  47. [55]

    Joint modeling of feature, correspondence, and a compressed memory for video object segmentation.arXiv:2308.13505,

    Jiaming Zhang, Yutao Cui, Gangshan Wu, and Limin Wang. Joint modeling of feature, correspondence, and a compressed memory for video object segmentation.arXiv:2308.13505,

    arXiv

  48. [56]

    Fast segment any- thing.arXiv:2306.12156, 2023

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment any- thing.arXiv:2306.12156, 2023. 2

    2023 arXiv

  49. [57]

    Edgesam: Prompt-in-the-loop distillation for on-device de- ployment of sam.arXiv:2312.06660, 2023

    Chong Zhou, Xiangtai Li, Chen Change Loy, and Bo Dai. Edgesam: Prompt-in-the-loop distillation for on-device de- ployment of sam.arXiv:2312.06660, 2023. 8, 11, 12

    2023 arXiv

  50. [58]

    Medical sam 2: Seg- ment medical images as video via segment anything model 2.arXiv:2408.00874, 2024

    Jiayuan Zhu, Yunli Qi, and Junde Wu. Medical sam 2: Seg- ment medical images as video via segment anything model 2.arXiv:2408.00874, 2024. 2 10

    2024 arXiv

  51. [61]

    -” indicates directly combining existing pre-trained models for inference. “Cost

    Appendix In the Appendix, we provide additional experimental re- sults and technical details to complement the main paper. Specifically, we demonstrate the applicability of our pro- posed SAM-I2V across different SAM variants in subsec- tion 6.1. We also present comprehensive ...

  52. [2023]

    1, 6, 7, 8, 11, 13, 17, 18, 19, 20

  53. [2024]

    1, 2, 3, 4, 5, 6, 7, 8, 11, 13, 15, 16, 17, 18, 19, 20

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.