{"id":"a9629956-d960-4c4f-867e-7a41996eff8b","arxiv_id":"2505.08581","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ReSurgSAM2 reports large gains in surgical referring video segmentation by adding a cross-modal Mamba detection stage, credible initial frame selection, and a diversity-driven long-term memory to SAM2, at 61.2 FPS.","lead":"A new two-stage framework, ReSurgSAM2, combines text-based detection with long-term memory tracking to perform real-time referring segmentation in surgical videos. On two public surgical datasets it reports large accuracy gains over prior methods, but the code and revised datasets are not yet released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison is not apples-to-apples: Table 2 baseline numbers may be from the original RSVIS labels, while ReSurgSAM2 uses refined and tissue-enriched reannotations (§3.1), so the 14.17 J&F gain could be partly a label/protocol artifact.","rationale":"ReSurgSAM2's design components have internal support: the ablation shows monotonic gains and DLM beats interval and vanilla memory, which is genuine evidence for the mechanism. The vulnerability is the external comparison. A method's SOTA claim is only as strong as the evaluation protocol; modified labels, added tissue classes, and changed text-expression generation alter the task. No ablation can substitute for a controlled comparison. I therefore focus on this rather than the reader's confidence-score calibration concern: even if IoU and occlusion scores are miscalibrated, the method could still rank frames reasonably and the empirical ablations would show it, but a comparison against baselines scored on a different gold standard cannot be salvaged by internal experiments. Since data and code are unavailable and the paper does not document baseline reruns, the quantitative headline is currently unverifiable. I would move the verdict from CONDITIONAL to UNVERDICTED: not because the method is wrong, but because the load-bearing empirical claim cannot be checked as written.","tokens_in":8405,"tokens_out":9733,"duration_ms":101985,"concrete_test":"Release the reannotated Ref-EndoVis17/18 and rerun all five baselines under exactly ReSurgSAM2's protocol (same refined labels, same merged train/test splits, same per-object first-appearance text expressions, same evaluation code), then recompute Table 2. Also report ReSurgSAM2 numbers on the original RSVIS annotations. If the Ref-EndoVis17 gap shrinks materially from 14.17 J&F, the reported advantage is partly a protocol artifact; if it persists, the concern is resolved.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is empirical: ReSurgSAM2 'achieves substantial improvements in accuracy... compared to existing methods' (abstract, Table 2). Section 3.1 says the authors 'performed meticulous refinement to address inconsistencies and omissions in instrument labeling' and 'further enriched the datasets by incorporating tissue-specific annotations from EndoVis18.' They also changed the inference protocol: 'we generate text expressions at the first appearance of each object' (§3.1). The paper never states that RSVIS, OnlineRefer, RefSAM, ReferFormer, or MUTR were retrained or re-evaluated on this reannotated, enriched version with the same expressions and split. If the Table 2 baseline rows are taken from the original publications, the comparison confounds method quality with changes to the gold standard and task definition. This is load-bearing because the headline SOTA improvements (14.17/7.76/3.19 J&F) and the FPS comparison depend entirely on this table. The same issue also affects the tissue subset, which is a new task for the baselines. Internal ablations are coherent, but they cannot establish superiority over methods evaluated under a different protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReSurgSAM2, a two-stage framework for referring video object segmentation in surgical scenes. In the first stage, a cross-modal spatial-temporal Mamba (CSTMamba) performs text-referred detection, and a credible initial frame selection (CIFS) strategy picks a reliable frame using the model's predicted IoU and occlusion scores. In the second stage, a diversity-driven long-term memory (DLM) mechanism extends SAM2's memory bank with confident, diverse frames for tracking. The authors report state-of-the-art results on Ref-EndoVis17 and Ref-EndoVis18 (tool and tissue), with J&F improvements of 14.17, 7.76, and 3.19 over prior methods, and real-time inference at 61.2 FPS. The claims are empirical, supported by ablations in Tables 3 and 4, but the comparison protocol raises concerns about fairness because the datasets were reannotated and enriched.","tokens_in":8615,"tokens_out":3876,"duration_ms":37657,"significance":"If the reported gains hold under a fair comparison, ReSurgSAM2 would be a meaningful advance for surgical referring segmentation: it is the first SAM2-based two-stage method for this task, it introduces a credible-frame-selection mechanism and a diversity-driven memory bank, and it achieves real-time speed. The ablations are coherent and attribute each component's contribution, and the qualitative examples support the method's ability to handle long-term tracking. The paper also promises to release code and datasets, which would aid reproducibility. However, the significance is tempered by the lack of a controlled comparison with baselines on the same reannotated data and protocol, and by the absence of statistical validation.","major_comments":[{"comment":"The central comparison in Table 2 is not apples-to-apples. The authors state in Section 3.1 that they 'performed meticulous refinement to address inconsistencies and omissions in instrument labeling' and 'further enriched the datasets by incorporating tissue-specific annotations,' and they also changed the inference protocol by generating text expressions at the first appearance of each object. The paper nowhere states that RSVIS, OnlineRefer, RefSAM, ReferFormer, or MUTR were retrained or re-evaluated on this reannotated, enriched version with the same expressions and split. The reported J&F gains of 14.17, 7.76, and 3.19, and the FPS comparison, therefore conflate method quality with label/protocol artifacts. The authors should re-run all baselines on the refined datasets with identical expressions and splits, or clearly present both original and re-evaluated numbers and quantify the impact of the annotation changes.","section":"Section 3.1 and Table 2"},{"comment":"All results are reported as single point estimates without error bars, confidence intervals, or significance tests. Given the small test sets (e.g., 900 frames for Ref-EndoVis17) and the fact that some gains are only around 3 J&F (e.g., 3.03 for DLM in Table 3), the reader cannot judge whether these differences are stable. The authors should provide multiple runs, variance estimates, or at least a paired significance test, and should also report the hyperparameter sensitivity for the thresholds in Eqs. (1)-(3), since the method's behavior depends heavily on them.","section":"Section 3.2, Tables 2-4"},{"comment":"The credible initial frame selection and the memory candidate admission rely on the model's own predicted IoU and occlusion scores as measures of segmentation quality. The paper provides no calibration analysis or failure-case study demonstrating that these self-predicted scores are trustworthy on surgical data. If the scores are miscalibrated, CIFS and DLM will select unreliable frames, and the two-stage design loses its foundation. A comparison of predicted scores with actual mask quality over frames and objects, and an analysis of failure modes, would strengthen the claim.","section":"Equations (1)-(3), Sections 2.2-2.3"},{"comment":"The Ref-EndoVis18 tissue subset is a new task introduced by the authors, combining tissue annotations from EndoVis18 with reannotated instrument labels. There is no indication that any baseline was trained on tissue annotations, so the tissue rows in Table 2 compare ReSurgSAM2 against methods that never saw this class of supervision. This conflates task novelty with method performance. The tissue comparison should include baselines trained on the same tissue-enriched data, or the authors should explicitly discuss and justify this asymmetry.","section":"Section 3.1 and Table 2 (tissue rows)"}],"minor_comments":[{"comment":"The first sentence of Section 3.1 contains a typo: 'Ref-EndoVis17 and Ref-EndoVis17 building upon EndoVis17 [2], EndoVis18 dataset [1] and RSVIS [19]' should refer to Ref-EndoVis17 and Ref-EndoVis18.","section":"Section 3.1"},{"comment":"Figure 1 is very dense, with many abbreviations (CSTMamba, CIFS, DLM, T2V, V2T, sensory memory) appearing without a detailed caption; a more explanatory caption would help the reader follow the two-stage pipeline.","section":"Figure 1"},{"comment":"In Table 3, the first row (without any checkmarks) appears to be the no-stage-2 baseline, but this is not explicitly explained in the text; the meaning of each row and the baseline setting should be clarified.","section":"Table 3"},{"comment":"The abstract promises code and datasets at a URL, but the full text does not include the URL; also, for reproducibility, the refined annotations should be released together with the code, and the paper should state the exact annotation changes and how they affect the original RSVIS benchmarks.","section":"Abstract and Section 3.1"},{"comment":"The hyperparameters delta_o, delta_iou, gamma_iou, N_w, N_p, and N_l are reported, but no sensitivity analysis is given; given their direct influence on CIFS and DLM, a small sensitivity study would help establish robustness.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the comparison protocol: if the baselines were not re-evaluated on the reannotated, enriched datasets, the headline improvements may be largely an artifact of label changes and the new inference protocol. This is fixable by re-running baselines, but it is load-bearing. The paper would also benefit from statistical validation and a calibration analysis of the self-predicted confidence scores."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know first: ReSurgSAM2 is a thoughtful two-stage architecture for text-driven segmentation in surgical video, and the ablations are unusually clean. But the headline numbers in Table 2 may be comparing a reannotated, tissue-enriched dataset against baselines evaluated under the original RSVIS protocol. If that is true, the 14.17 J&F jump is partly a label artifact.\n\nWhat is new: CSTMamba extends Mamba with cross-modal attention for detection; CIFS picks a reliable starting frame using the model's own IoU/occlusion scores; DLM replaces SAM2's greedy memory with a diversity-based long-term bank. Each component is ablated and contributes meaningfully on Ref-EndoVis17, with increments from 2.64 to 6.14 J&F. The system runs at 61.2 FPS, which matters for intraoperative use. The observation that SAM2's nearest-frame memory and unverified confidence scores are weak for long surgical videos is accurate, and the proposed fixes are reasonable.\n\nWhere it wobbles: the paper says it performed meticulous refinement of instrument labels, added tissue annotations, and changed inference to generate expressions at each object's first appearance. Nowhere does it state that ReferFormer, MUTR, RSVIS, OnlineRefer, or RefSAM were retrained or re-evaluated on this modified data with the same expression schedule. The line \"All experiments utilized the same training data\" is too vague. This is load-bearing, since the claimed SOTA gains and the FPS comparison rest entirely on Table 2. The internal ablations are coherent, but they cannot establish superiority over methods possibly run under a different protocol. Also, there are no error bars or significance tests; that is common in this subfield but still worth flagging. The mask decoder's confidence scores are trusted as quality measures; the ablation suggests they work, but calibration on surgical data is not examined. Code and datasets are promised but not released, so independent verification is impossible right now.\n\nWho this is for: someone working on surgical video understanding or SAM2 adaptations will find the two-stage tracking design and the memory diversity idea useful, even if the comparison issue is unresolved.\n\nRecommendation: send it to a serious referee. The architecture is thoughtful, the ablations support the components, and the application is practical. But the authors must clarify the baseline evaluation protocol, release code and data, and ideally report variance before the empirical claims are fully credible.","headline":"A cleanly ablated two-stage SAM2 adaptation for surgical referring segmentation, but the benchmark comparison may be confounded by reannotated labels and an altered expression protocol.","tokens_in":9205,"tokens_out":2677,"would_cite":false,"duration_ms":25939,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A text prompt can now drive real-time, long-term segmentation of surgical tools and tissues in video.","keywords":["referring video object segmentation","surgical scene segmentation","Segment Anything Model 2","long-term tracking","Mamba","memory bank","real-time segmentation","surgical robotics"],"falsifier":"Compare the frames CIFS and DLM select using predicted IoU and occlusion scores against selection by oracle mask quality on the same test videos; if the predicted-score selections are no better than random at true mask quality, or if oracle selection does not preserve the reported gains, then credible selection is not what carries the result.","tokens_in":8176,"feed_emoji":"🩺","tokens_out":7994,"duration_ms":70995,"temperature":0.7,"pith_summary":"The paper introduces ReSurgSAM2, a two-stage framework for referring segmentation in surgical video: given a free-text description such as \"the large needle driver on the right,\" the system first detects and segments the target in a short window, then tracks it through the rest of the video. The authors argue this fills a practical gap, because existing online methods use only short-term information and lose objects, while offline methods need future frames and cannot run during surgery. The solution rests on two ideas: choose a reliable initial frame using the mask decoder's own quality scores before tracking begins, and augment the nearest-frame memory with a bank of high-confidence, visually diverse long-term frames. If the claims hold, a surgeon could interactively name an instrument or tissue and receive persistent real-time segmentation throughout a long procedure without manual clicks.","feed_headline":"Surgical tool tracking from text prompts hits 61 FPS","feed_subtitle":"A two-stage Segment Anything design beats prior online referring-segmentation methods by up to 14.2 J&F.","key_machinery":"The central machinery is a three-part pipeline built on Segment Anything Model 2. CSTMamba, a cross-modal spatial-temporal Mamba block, combines a 7x7 depthwise convolution with selective scanning and bidirectional text-vision attention to fuse language features with three-frame video features for reliable detection. CIFS, the credible initial frame selection, only switches to tracking after five consecutive frames clear thresholds on the predicted IoU and occlusion scores, then picks the highest-IoU frame as the permanent reference. DLM, the diversity-driven long-term memory, admits frames whose predicted IoU exceeds 0.95 into a candidate pool and, when the pool is full, stores the frame with minimum cosine similarity to the latest long-term memory frame. The mask decoder's self-predicted IoU and occlusion scores are the common currency that decides when tracking starts and what the memory keeps.","core_discovery":"ReSurgSAM2 claims that surgical referring video object segmentation is best solved by splitting the problem into two stages: first detect the text-referred object over a short temporal window with a cross-modal spatial-temporal Mamba and select the single most credible frame as the tracking reference, then track with Segment Anything Model 2 whose memory is enriched by high-confidence, diversity-selected long-term frames. On the reannotated Ref-EndoVis17 and Ref-EndoVis18 benchmarks, the pipeline reports J&F of 77.73, 80.62, and 75.09 respectively for tools and tissues, exceeding the best prior online method by up to 14.17 points while running at 61.2 FPS.","pith_inferences":["Beyond the paper: because the method needs only a text prompt and no manual first-frame click, it could transfer to other long clinical video tasks where the object of interest appears midway, such as endoscopy navigation or microscope footage.","Beyond the paper: the predicted IoU and occlusion scores carry the whole credibility decision, so a learned recalibration of those scores on surgical data is a natural testbed for improving the method's transfer to new instruments or new surgical domains.","Beyond the paper: the memory criterion picks the frame least similar to the latest long-term memory frame, but the paper does not isolate whether coverage or raw confidence drives the long-term gain; a factorial experiment varying these two terms would settle that.","Beyond the paper: the largest reported margin is 14.17 J&F on Ref-EndoVis17 tools, so an event-level breakdown (occlusion, fast motion, re-entry) would show whether the gains come from better initialization or from long-term memory on particularly hard frames."],"forward_implications":["A surgeon can specify a target by text during an operation and the system will keep segmenting that instrument or tissue through occlusions and viewpoint changes for the rest of the video.","Because the method runs online at 61.2 FPS, it is usable during surgery rather than after the fact, unlike offline baselines that process whole sequences.","Ablations show the full pipeline raises J&F on Ref-EndoVis17 from 61.15 for the framework without the proposed components to 77.73, with CIFS contributing the single largest gain of 6.14 points.","The gains extend beyond metal instruments to tissue targets such as kidney parenchyma and small intestine, suggesting the mechanism is not specific to tool appearance."],"supporting_citations":[{"why":"Supplies the Segment Anything Model 2 backbone, pretrained weights, and the mask decoder whose IoU and occlusion scores drive both selection stages.","marker":"[13]"},{"why":"Introduces the surgical referring video instrument segmentation task and the RSVIS reannotations of EndoVis17 and EndoVis18 used for training and testing.","marker":"[19]"},{"why":"RefSAM is the strongest prior online SAM-based referring video segmentation baseline that ReSurgSAM2 compares against.","marker":"[8]"},{"why":"OnlineRefer provides the other online query-propagation baseline for long-term tracking comparisons.","marker":"[20]"},{"why":"ReferFormer is the offline full-sequence baseline whose future-frame dependence motivates the online design.","marker":"[21]"},{"why":"MUTR is the offline multi-modality temporal transformer baseline included in the comparison table.","marker":"[22]"},{"why":"Supplies the video vision Mamba design that CSTMamba extends with depthwise convolution and cross-modal attention.","marker":"[23]"},{"why":"Motivates replacing greedy nearest-frame memory with quality-aware memory selection for video object segmentation.","marker":"[10]"},{"why":"Provides the EndoVis18 challenge dataset whose instrument and tissue annotations underlie Ref-EndoVis18.","marker":"[1]"},{"why":"Provides the EndoVis17 challenge dataset whose instrument annotations underlie Ref-EndoVis17.","marker":"[2]"}],"fun_headline_variants":["Text-prompted surgical tracking: 61 FPS with SAM2","ReSurgSAM2: real-time referring segmentation in surgery","Two-stage tracking boosts surgical referring segmentation by 14 J&F","SAM2-based tracker for surgical tools hits 61.2 FPS","From text to tracked tools: ReSurgSAM2 at 61 FPS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole design assumes that the model's self-predicted confidence scores — estimated mask quality and object visibility — are calibrated well enough on surgical video to choose trustworthy starting frames and memory frames.","fun_headline_variants_meta":{"raw":{"variants":["Text-prompted surgical tracking: 61 FPS with SAM2","ReSurgSAM2: real-time referring segmentation in surgery","Two-stage tracking boosts surgical referring segmentation by 14 J&F","SAM2-based tracker for surgical tools hits 61.2 FPS","From text to tracked tools: ReSurgSAM2 at 61 FPS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1487,"prompt_tokens":922,"completion_tokens":565,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":470}},"tokens_in":538,"tokens_out":565,"duration_ms":4822,"temperature":1.0,"reasoning_tokens":470,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:50:41.046302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the frames CIFS and DLM select using predicted IoU and occlusion scores against selection by oracle mask quality on the same test videos; if the predicted-score selections are no better than random at true mask quality, or if oracle selection does not preserve the reported gains, then credible selection is not what carries the result.","supporting_citations":[{"cited_title":"IEEE Transactions on Medical Imaging43(12), 4457–4469 (2024)","cited_arxiv_id":null,"evidence_quote":"Introduces the surgical referring video instrument segmentation task and the RSVIS reannotations of EndoVis17 and EndoVis18 used for training and testing."},{"cited_title":"In: Proceedings of the IEEE/CVF International Conference on Computer Vision","cited_arxiv_id":null,"evidence_quote":"OnlineRefer provides the other online query-propagation baseline for long-term tracking comparisons."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"ReferFormer is the offline full-sequence baseline whose future-frame dependence motivates the online design."},{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"MUTR is the offline multi-modality temporal transformer baseline included in the comparison table."}],"review_version":1}