Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read A text prompt can now drive real-time, long-term segmentation of surgical tools and tissues in video.

desk verdict A cleanly ablated two-stage SAM2 adaptation for surgical referring segmentation, but the benchmark comparison may be confounded by reannotated labels and an altered expression protocol. read the letter →

arxiv 2505.08581 v1 pith:ULYYZX7K submitted 2025-05-13 cs.CV eess.IVq-bio.TO

classification cs.CVeess.IVq-bio.TO
keywords referringvideoobjectsegmentationsurgicalsceneSegmentAnythingModel2long-termtrackingMambamemorybankreal-timerobotics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ReSurgSAM2, a two-stage framework for referring segmentation in surgical video: given a free-text description such as "the large needle driver on the right," the system first detects and segments the target in a short window, then tracks it through the rest of the video. The authors argue this fills a practical gap, because existing online methods use only short-term information and lose objects, while offline methods need future frames and cannot run during surgery. The solution rests on two ideas: choose a reliable initial frame using the mask decoder's own quality scores before tracking begins, and augment the nearest-frame memory with a bank of high-confidence, visually diverse long-term frames. If the claims hold, a surgeon could interactively name an instrument or tissue and receive persistent real-time segmentation throughout a long procedure without manual clicks.

What carries the argument

The central machinery is a three-part pipeline built on Segment Anything Model 2. CSTMamba, a cross-modal spatial-temporal Mamba block, combines a 7x7 depthwise convolution with selective scanning and bidirectional text-vision attention to fuse language features with three-frame video features for reliable detection. CIFS, the credible initial frame selection, only switches to tracking after five consecutive frames clear thresholds on the predicted IoU and occlusion scores, then picks the highest-IoU frame as the permanent reference. DLM, the diversity-driven long-term memory, admits frames whose predicted IoU exceeds 0.95 into a candidate pool and, when the pool is full, stores the frame with minimum cosine similarity to the latest long-term memory frame. The mask decoder's self-predicted IoU and occlusion scores are the common currency that decides when tracking starts and what the memory keeps.

What would settle it

Compare the frames CIFS and DLM select using predicted IoU and occlusion scores against selection by oracle mask quality on the same test videos; if the predicted-score selections are no better than random at true mask quality, or if oracle selection does not preserve the reported gains, then credible selection is not what carries the result.

Watch

Extended reading notes

Core claim

ReSurgSAM2 claims that surgical referring video object segmentation is best solved by splitting the problem into two stages: first detect the text-referred object over a short temporal window with a cross-modal spatial-temporal Mamba and select the single most credible frame as the tracking reference, then track with Segment Anything Model 2 whose memory is enriched by high-confidence, diversity-selected long-term frames. On the reannotated Ref-EndoVis17 and Ref-EndoVis18 benchmarks, the pipeline reports J&F of 77.73, 80.62, and 75.09 respectively for tools and tissues, exceeding the best prior online method by up to 14.17 points while running at 61.2 FPS.

Load-bearing premise

The whole design assumes that the model's self-predicted confidence scores — estimated mask quality and object visibility — are calibrated well enough on surgical video to choose trustworthy starting frames and memory frames.

Editorial extensions

If this is right

  • A surgeon can specify a target by text during an operation and the system will keep segmenting that instrument or tissue through occlusions and viewpoint changes for the rest of the video.
  • Because the method runs online at 61.2 FPS, it is usable during surgery rather than after the fact, unlike offline baselines that process whole sequences.
  • Ablations show the full pipeline raises J&F on Ref-EndoVis17 from 61.15 for the framework without the proposed components to 77.73, with CIFS contributing the single largest gain of 6.14 points.
  • The gains extend beyond metal instruments to tissue targets such as kidney parenchyma and small intestine, suggesting the mechanism is not specific to tool appearance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the method needs only a text prompt and no manual first-frame click, it could transfer to other long clinical video tasks where the object of interest appears midway, such as endoscopy navigation or microscope footage.
  • Beyond the paper: the predicted IoU and occlusion scores carry the whole credibility decision, so a learned recalibration of those scores on surgical data is a natural testbed for improving the method's transfer to new instruments or new surgical domains.
  • Beyond the paper: the memory criterion picks the frame least similar to the latest long-term memory frame, but the paper does not isolate whether coverage or raw confidence drives the long-term gain; a factorial experiment varying these two terms would settle that.
  • Beyond the paper: the largest reported margin is 14.17 J&F on Ref-EndoVis17 tools, so an event-level breakdown (occlusion, fast motion, re-entry) would show whether the gains come from better initialization or from long-term memory on particularly hard frames.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ReSurgSAM2, a two-stage framework for referring video object segmentation in surgical scenes. In the first stage, a cross-modal spatial-temporal Mamba (CSTMamba) performs text-referred detection, and a credible initial frame selection (CIFS) strategy picks a reliable frame using the model's predicted IoU and occlusion scores. In the second stage, a diversity-driven long-term memory (DLM) mechanism extends SAM2's memory bank with confident, diverse frames for tracking. The authors report state-of-the-art results on Ref-EndoVis17 and Ref-EndoVis18 (tool and tissue), with J&F improvements of 14.17, 7.76, and 3.19 over prior methods, and real-time inference at 61.2 FPS. The claims are empirical, supported by ablations in Tables 3 and 4, but the comparison protocol raises concerns about fairness because the datasets were reannotated and enriched.

Significance. If the reported gains hold under a fair comparison, ReSurgSAM2 would be a meaningful advance for surgical referring segmentation: it is the first SAM2-based two-stage method for this task, it introduces a credible-frame-selection mechanism and a diversity-driven memory bank, and it achieves real-time speed. The ablations are coherent and attribute each component's contribution, and the qualitative examples support the method's ability to handle long-term tracking. The paper also promises to release code and datasets, which would aid reproducibility. However, the significance is tempered by the lack of a controlled comparison with baselines on the same reannotated data and protocol, and by the absence of statistical validation.

major comments (4)
  1. [Section 3.1 and Table 2] The central comparison in Table 2 is not apples-to-apples. The authors state in Section 3.1 that they 'performed meticulous refinement to address inconsistencies and omissions in instrument labeling' and 'further enriched the datasets by incorporating tissue-specific annotations,' and they also changed the inference protocol by generating text expressions at the first appearance of each object. The paper nowhere states that RSVIS, OnlineRefer, RefSAM, ReferFormer, or MUTR were retrained or re-evaluated on this reannotated, enriched version with the same expressions and split. The reported J&F gains of 14.17, 7.76, and 3.19, and the FPS comparison, therefore conflate method quality with label/protocol artifacts. The authors should re-run all baselines on the refined datasets with identical expressions and splits, or clearly present both original and re-evaluated numbers and quantify the impact of the annotation changes.
  2. [Section 3.2, Tables 2-4] All results are reported as single point estimates without error bars, confidence intervals, or significance tests. Given the small test sets (e.g., 900 frames for Ref-EndoVis17) and the fact that some gains are only around 3 J&F (e.g., 3.03 for DLM in Table 3), the reader cannot judge whether these differences are stable. The authors should provide multiple runs, variance estimates, or at least a paired significance test, and should also report the hyperparameter sensitivity for the thresholds in Eqs. (1)-(3), since the method's behavior depends heavily on them.
  3. [Equations (1)-(3), Sections 2.2-2.3] The credible initial frame selection and the memory candidate admission rely on the model's own predicted IoU and occlusion scores as measures of segmentation quality. The paper provides no calibration analysis or failure-case study demonstrating that these self-predicted scores are trustworthy on surgical data. If the scores are miscalibrated, CIFS and DLM will select unreliable frames, and the two-stage design loses its foundation. A comparison of predicted scores with actual mask quality over frames and objects, and an analysis of failure modes, would strengthen the claim.
  4. [Section 3.1 and Table 2 (tissue rows)] The Ref-EndoVis18 tissue subset is a new task introduced by the authors, combining tissue annotations from EndoVis18 with reannotated instrument labels. There is no indication that any baseline was trained on tissue annotations, so the tissue rows in Table 2 compare ReSurgSAM2 against methods that never saw this class of supervision. This conflates task novelty with method performance. The tissue comparison should include baselines trained on the same tissue-enriched data, or the authors should explicitly discuss and justify this asymmetry.
minor comments (5)
  1. [Section 3.1] The first sentence of Section 3.1 contains a typo: 'Ref-EndoVis17 and Ref-EndoVis17 building upon EndoVis17 [2], EndoVis18 dataset [1] and RSVIS [19]' should refer to Ref-EndoVis17 and Ref-EndoVis18.
  2. [Figure 1] Figure 1 is very dense, with many abbreviations (CSTMamba, CIFS, DLM, T2V, V2T, sensory memory) appearing without a detailed caption; a more explanatory caption would help the reader follow the two-stage pipeline.
  3. [Table 3] In Table 3, the first row (without any checkmarks) appears to be the no-stage-2 baseline, but this is not explicitly explained in the text; the meaning of each row and the baseline setting should be clarified.
  4. [Abstract and Section 3.1] The abstract promises code and datasets at a URL, but the full text does not include the URL; also, for reproducibility, the refined annotations should be released together with the code, and the paper should state the exact annotation changes and how they affect the original RSVIS benchmarks.
  5. [Section 3.1] The hyperparameters delta_o, delta_iou, gamma_iou, N_w, N_p, and N_l are reported, but no sensitivity analysis is given; given their direct influence on CIFS and DLM, a small sensitivity study would help establish robustness.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: architecture is empirical and externally compared; self-citations are not load-bearing; Table 2 protocol mismatch is a comparability risk, not circularity.

full rationale

ReSurgSAM2 is presented as an empirically validated architecture rather than a derivation from first principles. The load-bearing claims are the quantitative comparisons in Table 2 and the ablation study in Table 3, which are evaluated with ground-truth J&F on Ref-EndoVis17/18 against external baselines (ReferFormer, MUTR, RSVIS, OnlineRefer, RefSAM). The headline improvements are therefore not constructed from the method's own definitions. The method's self-confidence scores (predicted IoU and occlusion score) are used as internal signals in CIFS (Eq. 1) and DLM (Eq. 2); this is a design choice that can affect performance if miscalibrated, but it is not circular because success is measured against independent ground truth, not against the same scores. The principal caveat is experimental protocol: Section 3.1 states that the authors refined instrument labels, added tissue annotations, and generate expressions at first object appearance ('For inference, unlike RSVIS...'), while Table 2 does not state that the baselines were retrained or re-evaluated on the same reannotated protocol. If the baseline rows are taken from the original papers, the comparison may conflate method quality with label/task changes; however, this is a comparability/correctness threat, not a circularity of the derivation chain. The only self-citations (RSVIS [19], Surgical SAM2 [9]) are dataset/related-work references and do not supply a load-bearing uniqueness or equivalence argument. Hence no circular step is identified; score 2 reflects minor, non-load-bearing self-citation.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim relies on hand-set thresholds and queue sizes, on the trustworthiness of the model's predicted IoU and occlusion scores, and on the validity of the authors' reannotated datasets. No invented physical entities are introduced.

free parameters (6)
  • delta_o = 0.9
    Occlusion-score threshold for including a frame in the CIFS sliding window (Eq. 1); set by hand, no sensitivity analysis reported.
  • delta_iou = 0.7
    IoU-score threshold for the CIFS sliding window (Eq. 1); set by hand.
  • gamma_iou = 0.95
    IoU threshold for adding a frame to the DLM candidate pool (Eq. 2); set by hand.
  • N_w = 5
    CIFS window size (Eq. 1); set by hand.
  • N_p = 5
    DLM candidate pool capacity; set by hand.
  • N_l = 4
    Long-term memory queue capacity; set by hand.
assumptions (3)
  • domain assumption Predicted IoU and occlusion scores reflect segmentation quality.
    CIFS and DLM use these scores to select frames (Eqs. 1 and 2). No calibration analysis is provided.
  • domain assumption The reannotated Ref-EndoVis17 and Ref-EndoVis18 datasets are valid benchmarks.
    The authors modified existing annotations and merged sequences in Section 3.1; the revised datasets are not released.
  • domain assumption Text expressions generated at the first appearance of each object match deployment conditions.
    Inference uses expressions generated at the first appearance of each object (Section 3.1), which may differ from realistic intraoperative queries.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking." pith.science (2026). https://pith.science/paper/ULYYZX7K

@misc{pith2026250508581,
  author       = {Pith},
  title        = {Pith review of: ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULYYZX7K}},
  note         = {Machine review of arXiv:2505.08581}
}
read the original abstract

Surgical scene segmentation is critical in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, referring surgical segmentation is emerging, given its advantage of providing surgeons with an interactive experience to segment the target object. However, existing methods are limited by low efficiency and short-term tracking, hindering their applicability in complex real-world surgical scenarios. In this paper, we introduce ReSurgSAM2, a two-stage surgical referring segmentation framework that leverages Segment Anything Model 2 to perform text-referred target detection, followed by tracking with reliable initial frame identification and diversity-driven long-term memory. For the detection stage, we propose a cross-modal spatial-temporal Mamba to generate precise detection and segmentation results. Based on these results, our credible initial frame selection strategy identifies the reliable frame for the subsequent tracking. Upon selecting the initial frame, our method transitions to the tracking stage, where it incorporates a diversity-driven memory mechanism that maintains a credible and diverse memory bank, ensuring consistent long-term tracking. Extensive experiments demonstrate that ReSurgSAM2 achieves substantial improvements in accuracy and efficiency compared to existing methods, operating in real-time at 61.2 FPS. Our code and datasets will be available at https://github.com/jinlab-imvr/ReSurgSAM2.

Figures

Figures reproduced from arXiv: 2505.08581 by the authors.

Figure 1
Figure 1. Overview of ReSurgSAM2. The model begins with the text-referred target [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Qualitative comparison in Ref-EndoVis17(left) and Ref-EndoVis18(right). [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Tracking

    cs.CV 2025-11 conditional novelty 5.0 of 10

    SAM2S, a SAM2 variant trained on the new 61k-frame SA-SV surgical benchmark, improves average J&F to 80.42 at 68 FPS for interactive surgical-video object segmentation.

Reference graph

Works this paper leans on

25 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2001.11190 (2020)

    Allan,M.,Kondo,S.,Bodenstedt,S.,Leger,S.,Kadkhodamohammadi,R.,Luengo, I., Fuentes, F., Flouty, E., Mohammed, A., Pedersen, M., et al.: 2018 robotic scene segmentation challenge. arXiv preprint arXiv:2001.11190 (2020)

  2. [2]

    arXiv preprint arXiv:1902.06426 (2019)

    Allan, M., Shvets, A., Kurmann, T., Zhang, Z., Duggal, R., Su, Y.H., Rieke, N., Laina, I., Kalavakonda, N., Bodenstedt, S., et al.: 2017 robotic instrument segmen- tation challenge. arXiv preprint arXiv:1902.06426 (2019)

  3. [3]

    arXiv preprint arXiv:2411.17646 (2024)

    Cuttano, C., Trivigno, G., Rosi, G., Masone, C., Averta, G.: Samwise: Infusing wis- dom in sam2 for text-driven video segmentation. arXiv preprint arXiv:2411.17646 (2024)

  4. [4]

    arXiv preprint arXiv:2312.00752 (2023)

    Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023)

  5. [5]

    In: Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part IV 14

    Khoreva, A., Rohrbach, A., Schiele, B.: Video object segmentation with language referring expressions. In: Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part IV 14. pp. 123–141. Springer (2019)

  6. [6]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023)

  7. [7]

    Surgery170(1), 88–98 (2021)

    Kovoor, J.G., Gupta, A.K., Gladman, M.A.: Validity and effectiveness of aug- mented reality in surgical education: a systematic review. Surgery170(1), 88–98 (2021)

  8. [8]

    arXiv preprint arXiv:2307.00997 (2024)

    Li, Y., Zhang, J., Teng, X., Lan, L., Liu, X.: Refsam: Efficiently adapting seg- menting anything model for referring video object segmentation. arXiv preprint arXiv:2307.00997 (2024)

Show all 25 references
  1. [9]

    In: Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond (2024)

    Liu, H., Zhang, E., Wu, J., Hong, M., Jin, Y.: Surgical sam 2: Real-time segment anything in surgical video by efficient frame pruning. In: Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond (2024)

  2. [10]

    In: European Conference on Computer Vision

    Liu, Y., Yu, R., Yin, F., Zhao, X., Zhao, W., Xia, W., Yang, Y.: Learning quality- aware dynamic memory for video object segmentation. In: European Conference on Computer Vision. pp. 468–486. Springer (2022)

  3. [11]

    International Journal of Surgery 95, 106151 (2021)

    Moglia, A., Georgiou, K., Georgiou, E., Satava, R.M., Cuschieri, A.: A systematic review on artificial intelligence in robot-assisted surgery. International Journal of Surgery 95, 106151 (2021)

  4. [12]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine- Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 724–732 (2016) 10 Liu et al

  5. [13]

    arXiv preprint arXiv:2408.00714 (2024)

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

  6. [14]

    Ryali, C., Hu, Y.T., Bolya, D., Wei, C., Fan, H., Huang, P.Y., Aggarwal, V., Chowdhury, A., Poursaeed, O., Hoffman, J., et al.: Hiera: A hierarchical vision transformerwithoutthebells-and-whistles.In:Internationalconferenceonmachine learning. pp. 29441–29454 (2023)

  7. [15]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: In- verted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510–4520 (2018)

  8. [16]

    Surgical technology international 33 (2019)

    Sheik-Ali, S., Edgcombe, H., Paton, C.: Next-generation virtual and augmented reality in surgical education: a narrative review. Surgical technology international 33 (2019)

  9. [17]

    ACM Computing Surveys55(6), 1–28 (2022)

    Tay, Y., Dehghani, M., Bahri, D., Metzler, D.: Efficient transformers: A survey. ACM Computing Surveys55(6), 1–28 (2022)

  10. [18]

    Advances in neural information pro- cessing systems 30 (2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems 30 (2017)

  11. [19]

    IEEE Transactions on Medical Imaging43(12), 4457–4469 (2024)

    Wang, H., Yang, G., Zhang, S., Qin, J., Guo, Y., Xu, B., Jin, Y., Zhu, L.: Video- instrument synergistic network for referring video instrument segmentation in robotic surgery. IEEE Transactions on Medical Imaging43(12), 4457–4469 (2024)

  12. [20]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wu, D., Wang, T., Zhang, Y., Zhang, X., Shen, J.: Onlinerefer: A simple online baseline for referring video object segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2761–2770 (2023)

  13. [21]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wu, J., Jiang, Y., Sun, P., Yuan, Z., Luo, P.: Language as queries for referring video object segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4974–4984 (2022)

  14. [22]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Yan, S., Zhang, R., Guo, Z., Chen, W., Zhang, W., Li, H., Qiao, Y., Dong, H., He, Z., Gao, P.: Referred by multi-modality: A unified temporal transformer for video object segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 6449–6457 (2024)

  15. [23]

    arXiv preprint arXiv:2401.14168 (2024)

    Yang, Y., Xing, Z., Yu, L., Huang, C., Fu, H., Zhu, L.: Vivim: A video vision mamba for medical video segmentation. arXiv preprint arXiv:2401.14168 (2024)

  16. [24]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Yue, W., Zhang, J., Hu, K., Xia, Y., Luo, J., Wang, Z.: Surgicalsam: Efficient class promptable surgical instrument segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 6890–6898 (2024)

  17. [25]

    Advances in Neural Infor- mation Processing Systems36, 28611–28623 (2023)

    Zhou, Z., Alabi, O., Wei, M., Vercauteren, T., Shi, M.: Text promptable surgical instrument segmentation with vision-language models. Advances in Neural Infor- mation Processing Systems36, 28611–28623 (2023)

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.