{"id":"fe24cc4d-9212-4242-b4a4-e97a1223100e","arxiv_id":"2506.19658","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SAM2-SGP automatically prompts SAM2 using support-set-derived pseudo-masks and achieves higher Dice scores than nnUNet, SwinUNet, SAM2, and MedSAM2 across eight medical datasets.","lead":"A team proposes SAM2-SGP, a medical image segmentation framework that removes the need for human prompts by generating pseudo-masks from small support sets of labeled examples. It builds on SAM2's video memory mechanism and adds modules for pseudo-mask attention and LoRA fine-tuning, reporting improved Dice scores on 2D and 3D medical datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random-click prompts for SAM2/MedSAM2 baselines make the claimed consistent superiority over SAM2-based methods unsubstantiated; proper prompt comparison is required.","rationale":"The reader's verdict is CONDITIONAL, and the rationale includes the random-click baseline as one of several evaluation weaknesses. However, the reader's 'weakest_assumption' field points to the unspecified similarity measure for support-set selection, which we consider secondary. The load-bearing issue for the central claim is the unfair baseline protocol: SAM2 and MedSAM2 are deliberately given suboptimal prompts while the proposed method receives a strong automatically generated box prompt. This directly undermines the claim of 'consistently outperformed' SAM2-based approaches, because the comparison does not isolate the contribution of the proposed architecture. The similarity-measure gap is a reproducibility and potential data-leakage issue, but even a perfectly specified similarity function would not salvage the headline comparison if the baselines are improperly prompted. Since the reader's verdict is already conditional pending fairer evaluation, our concern does not move the verdict to a different category; it sharpens the conditions under which acceptance would be warranted. We therefore recommend UNCHANGED (CONDITIONAL), with the concrete test as a required revision.","tokens_in":14524,"tokens_out":3943,"duration_ms":41586,"concrete_test":"On REFUGE and AMOS22 CT, run the released SAM2 and MedSAM2 checkpoints with (a) a single click at the ground-truth centroid, (b) the ground-truth bounding box, and (c) the bounding box generated by the proposed method's pseudo-mask but without any of the proposed fine-tuned modules (i.e., standard SAM2 decoding). Record Dice for each. If any of these prompts yields Dice equal to or higher than the reported SAM2-SGP Dice, the claim of consistent superiority over SAM2-based methods is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 1) is that SAM2-SGP 'consistently outperformed both fully supervised segmentation models and SAM2-based approaches.' The evaluation of SAM2 and MedSAM2, however, is explicitly handicapped: Section 5.1 states 'Note that for SAM-based methods, a random click was used as the prompt.' SAM2 and MedSAM2 are promptable segmenters whose outputs depend strongly on prompt quality. A single random click frequently lands outside the target structure or on a boundary, especially in medical images with small or elongated anatomies. The proposed method instead automatically derives a bounding-box prompt from the PMG pseudo-mask, which is a substantially more informative prompt. The reported Dice gaps over SAM2/MedSAM2 (e.g., 0.753 vs 0.865 on REFUGE; 0.627 vs 0.976 on WBC) may therefore reflect the prompt advantage rather than any architectural benefit of the proposed modules. No error bars or significance tests are given, so even the remaining differences are not established as non-noise. The central claim thus rests on a comparison protocol that is unfair to the baselines; this is the least secure condition for the claim to hold. The unspecified similarity measure for support-set selection (Section 3.1) is a reproducibility concern, but it does not directly invalidate the comparison; the random-click baseline does.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAM2-SGP, a framework for automatic (prompt-free) medical image segmentation built on SAM2. A support set of image-mask pairs is used by a Pseudo-mask Generation (PMG) module, which adapts SAM2's memory attention to produce a pseudo-mask for each query image. A Pseudo-mask Attention (PMA) module converts the pseudo-mask into a bounding-box prompt and uses the pseudo-mask to gate attention, while LoRA adapts the frozen image encoder to the medical domain. For 3D volumes, a 3D memory module propagates predictions slice-to-slice. The method is evaluated on five 2D datasets and three 3D datasets against UNETR, SwinUNet, nnUNet, SAM2, and MedSAM2, and is reported to achieve the highest Dice in most settings. Ablations examine the contributions of the PMG, PMA, and 3DM modules, support-set size, and SAM2 pretrained weights. The central claim is that SAM2-SGP consistently outperforms both fully supervised models and SAM2-based approaches across modalities without manual prompting.","tokens_in":14794,"tokens_out":3578,"duration_ms":38214,"significance":"If the results hold under a fair comparison protocol, the paper makes a useful empirical contribution: it shows a concrete way to remove manual prompting from SAM2-style medical segmentation while retaining the benefits of foundation-model priors, through in-context support sets and pseudo-mask-guided attention. The paper also ships code and includes ablations for each proposed module, which are commendable for reproducibility. The architectural description is backed by equations and figures, and the use of LoRA to adapt a frozen encoder is sensible. However, the significance is currently tempered by a comparison protocol that appears unfair to the promptable baselines, by an undefined support-set similarity measure that undermines exact reproducibility, and by the absence of any error bars or significance tests. These issues affect the paper's central claim of consistent superiority and must be addressed before the contribution can be fully assessed.","major_comments":[{"comment":"The evaluation of SAM2 and MedSAM2 uses a random click as the prompt, as stated in Section 5.1, while SAM2-SGP automatically derives a bounding-box prompt from its pseudo-mask. Because SAM2 and MedSAM2 are prompt-dependent segmenters, a random click that often lands outside or on the boundary of the target structure cannot be considered a comparable baseline. The large reported gaps (e.g., REFUGE 0.753 vs. 0.865 and WBC 0.627 vs. 0.976) may therefore reflect the prompt information advantage rather than the contribution of the proposed modules. To substantiate the central claim, the authors should compare against SAM2/MedSAM2 using equivalent automatic prompts (e.g., ground-truth or predicted box prompts, or a multi-click protocol) and report both the prompt protocol and the resulting scores.","section":"Section 5.1, Tables 1-4"},{"comment":"The support-set selection is not reproducible: the paper states that 'we selected a subset of the support set consisting of the K most similar samples' but never defines the similarity function. Since the PMG module's pseudo-mask quality depends directly on this ranking, and for 3D data the memory bank also stores the model's own previous predictions (Eqs. 5-6), an undefined selection procedure makes the method impossible to re-implement exactly and leaves the failure mode of early poor selections unexamined. The authors should specify the similarity metric (e.g., feature distance, image-level Dice of pseudo-masks, or DINOv2 features) and describe how the selection is performed during training and inference.","section":"Section 3.1, Eq. (5)-(6)"},{"comment":"The dataset descriptions do not match the cited references. The head-and-neck PET/CT dataset is introduced with reference [36] as '100 3D PET images with a resolution of 124x124 pixels per slice,' but Section 5.2 evaluates on 'the PET/CT dataset [42],' which is described in the bibliography as a PET image denoising dataset. Similarly, the 'Panoramic dental dataset' is cited to reference [32], which is titled 'Automatic segmentation of mandible in panoramic x-ray' and does not describe an over-1,000-image tooth-and-bone segmentation dataset as stated. These mismatches make the exact evaluation conditions ambiguous and should be corrected with the proper dataset references and scanning protocols.","section":"Section 4.1.2 and Section 5.2"},{"comment":"No error bars, confidence intervals, or statistical significance tests are reported for any of the Dice comparisons. Several reported differences are small (e.g., CAMUS 0.932 vs. 0.925; ACDC average 0.917 vs. 0.916; Table 5 right kidney 0.952 vs. 0.953 across ablations), so without variance estimates the 'consistent' superiority claim is not established even under a fair prompt protocol. The authors should report results over multiple random seeds or cross-validation folds and, where appropriate, a paired significance test.","section":"Tables 1-5"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: 'Pesudo Mask Decoder' in Figure 2, 'Pandental' in Figure 4, 'outputed' in Section 3.2, and inconsistent capitalization of 'medSAM2' versus 'MedSAM2' between text and tables.","section":"Throughout"},{"comment":"The word 'significant' is used to describe performance improvements, but no statistical significance is established; please replace with 'substantial' or add significance tests.","section":"Abstract and Section 1"},{"comment":"The hyperparameter details ('learning rates and batch sizes were tuned') are too vague for reproduction; please list the actual values per dataset, including LoRA rank and alpha, loss weights in Eq. (9), and the memory bank size.","section":"Section 4.2"},{"comment":"The support-set-size ablation is described as duplicating a subset of training samples so that 'the training distribution remained unchanged'; please clarify whether the support set is disjoint from the query set used for training, and whether any samples in the support set also appear among the training queries, to rule out trivial memorization.","section":"Section 5.3.2, Table 6"},{"comment":"The organ-wise results show that SAM2-SGP is not best on all organs (e.g., pancreas 0.851 vs. nnUnet 0.874; esophagus 0.852 vs. nnUnet 0.857), which is fine, but the text should acknowledge this nuance when claiming consistent superiority.","section":"Table 3"},{"comment":"The ACDC results show lower Dice for left ventricle and myocardium than some baselines; the text mentions this but should also explain whether this is due to the unidirectional 3D propagation limitation mentioned in Section 6.","section":"Section 5.2, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is readable and the proposed architecture is plausible, but the evaluation is currently not convincing because of the random-click prompt for SAM2/MedSAM2 and the absence of error bars. The dataset reference mismatches also suggest the experimental section was not finalized carefully. Once the prompt protocol is made fair and the similarity metric is specified, the paper may be a solid contribution. I would like the editor to ensure the revised version includes a point-by-point response on the random-click issue, since it directly bears on the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper has a genuinely interesting mechanism—using SAM2's video memory machinery as an in-context support-set pseudo-mask generator, then deriving box prompts and mask attention from those pseudo-masks—and the 3D memory extension is a natural fit. But the paper's central claim that it beats SAM2 and MedSAM2 is not established by the current experiments, because those baselines were given a single random click as prompt while the proposed method gets a bounding box derived from its own pseudo-mask. That is not a fair comparison for a promptable segmenter. The gap on some datasets (WBC: 0.627 vs 0.976) is almost certainly mostly prompt advantage.\n\nWhat is actually new: combining the memory mechanism of SAM2 with support-set pseudo-masks, pseudo-mask attention, LoRA fine-tuning, and a 3D memory path. I don't see that exact combination in the cited work (ICL-SAM, MedSAM2, FS-MedSAM2), so the architecture contribution is real. The method is described with equations, the ablation on AMOS22 kidneys shows each module adds something, and the code is promised. The limitation discussion is honest about unidirectional 3D propagation and support-set dependence.\n\nSoft spots, in order:\n\n1. The baseline protocol is the load-bearing problem. Random click vs auto-derived box. To support the claim, they should re-run SAM2/MedSAM2 with an equally informative prompt—oracle box, best of several clicks, or at least the same pseudo-mask box. Without that, the comparison against SAM2-based methods is unsubstantiated. Against nnUNet/SwinUNet/UNETR the comparison is fine, though those are fully supervised, so the context differs.\n\n2. No error bars or significance tests anywhere. Several dice differences are small (CAMUS 0.932 vs 0.925; ACDC average 0.917 vs 0.916; AMOS MRI 0.715 vs 0.676). Some may be noise. They should report multiple seeds and variance.\n\n3. Section 3.1 says the K most similar support samples are selected but never defines the similarity function. That is a reproducibility gap; it should be specified.\n\n4. Minor: the support-set-size ablation is non-monotonic (K=4 slightly worse than K=2 on CAMUS/BUSI), and K=4 is then chosen as default. Also the PET/CT dataset citation jumps between [36] and [42]; the PanDental description doesn't quite match the cited mandible paper. These are easy fixes.\n\nThe math itself is fine—no circularity; the KL term between pseudo-mask and final prediction is a co-training objective, not evidence. The paper is for people adapting SAM2 to medical images and wanting a prompt-free workflow. It deserves a serious referee, but the comparison needs to be redone before the headline claim stands. My recommendation: send to peer review with a request for a fairer SAM2 baseline and error bars; conditional acceptance is the right call if those come out clean.","headline":"Useful SAM2 adaptation idea, but the claimed win over SAM2 baselines rests on a random-click prompt disadvantage and needs a fair rerun before the headline is credible.","tokens_in":15347,"tokens_out":2377,"would_cite":true,"duration_ms":23754,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAM2-SGP claims that a SAM2-based model can segment medical images without user-supplied prompts by generating pseudo-masks from a support set, and that this prompt-free design outperforms fully supervised models and prior SAM2…","keywords":["SAM2","medical image segmentation","support-set prompting","pseudo-mask generation","in-context learning","LoRA","3D medical segmentation","foundation model"],"falsifier":"Run the reported 2D benchmarks again with the K support samples chosen by a random or deliberately poor ranking instead of the paper's similarity criterion. If Dice does not drop substantially, then the claimed support-set-guided prompting mechanism is not what drives the gains; if it does drop, the missing similarity definition becomes the decisive implementation detail.","tokens_in":14288,"feed_emoji":"🩺","tokens_out":6882,"duration_ms":63059,"temperature":0.7,"pith_summary":"This paper tries to establish that SAM2, a foundation model trained on natural images and videos, can be made to segment medical images without any human-provided prompts. The proposed framework, SAM2-SGP, feeds a small support set of image-mask pairs into SAM2's memory mechanism to generate a pseudo-mask for each query image, derives a bounding-box prompt from that pseudo-mask, and uses the pseudo-mask to focus attention on relevant regions. A low-rank adaptation layer adapts the frozen encoder to the medical domain. If the claim holds, organ and lesion segmentation no longer requires a radiologist to click points or draw boxes at inference time, and the same model can be applied to fundus, X-ray, ultrasound, CT, MRI, and PET images. Across the reported 2D and 3D benchmarks, the framework reports higher Dice scores than fully supervised baselines such as nnUNet and SwinUNet and than prompt-based SAM2 baselines such as MedSAM2.","feed_headline":"Prompt-free SAM2 beats fully supervised medical baselines","feed_subtitle":"Pseudo-masks from a support set generate box prompts, beating nnUNet and MedSAM2 on 2D and 3D scans.","key_machinery":"The load-bearing mechanism is the support-set-guided pseudo-mask loop built on SAM2's memory attention. A support-memory encoder adds downsampled support masks to support-image embeddings ($M_s = \\phi(y_s) + F_s$); support-memory attention applies self-attention and then cross-attention between query embeddings and these support memories to form query-specific features; a lightweight decoder with an empty prompt produces the pseudo-mask $\\tilde{y}_q$. That pseudo-mask then does double duty: it is resized and element-wise multiplied into the attention output of the Pseudo-mask Attention module ($\\hat{F}_q = \\tilde{y}'_q \\odot \\mathrm{softmax}(QK^T/\\sqrt{d})V + \\tilde{F}_q$), and it supplies the bounding-box prompt $B_{\\mathrm{box}}$ for the prompt encoder. For 3D data, the same memory machinery is extended with a queue-based volumetric memory bank that appends encoded previous-slice predictions to the support memory.","core_discovery":"On the paper's own terms, the central discovery is that SAM2's memory mechanism, built for tracking objects across video frames, can be repurposed as an in-context learner for medical segmentation. The Pseudo-mask Generation module encodes support-set masks together with support-image features, cross-attends these support memories against query-image features, and decodes an initial pseudo-mask with an empty prompt. The Pseudo-mask Attention module turns that pseudo-mask into a bounding-box prompt and multiplies a resized pseudo-mask into the attention output, so the final decoder concentrates on the region the pseudo-mask highlights. For 3D volumes, a third module treats slices as frames, storing previously predicted slices in a memory bank that is concatenated with the support memory, and the memory bank pops the least similar entry as the model propagates along the axial direction. The paper reports that this prompt-free design consistently achieves the highest Dice among all compared methods on REFUGE, PanDental, WBC, CAMUS, BUSI, AMOS22 CT/MRI, PET/CT, and ACDC.","pith_inferences":["Because the K-most-similar selection criterion is never specified, the reported results should be read as conditional on an implicit retrieval quality; a natural extension is to quantify how Dice degrades as the similarity ranking is corrupted or replaced by random selection.","The unidirectional slice propagation uses only half of the available 3D context; extending the memory bank to bidirectional propagation is a direct algorithmic next step that could raise accuracy on anisotropic volumes.","The framework's prompt-free design suggests a practical deployment pattern: one frozen SAM2 with per-organ or per-modality support sets, with LoRA adapters swapped per domain, rather than retraining a full segmentation network for each task.","The pseudo-mask attention's restriction to relevant regions may reduce compute on high-resolution inputs like fundus and panoramic X-ray, though the paper does not report runtime or FLOPs to confirm this."],"forward_implications":["Automatic inference: once trained, the model needs only the query image plus the stored support set, so no clicks or boxes are required at test time.","Domain adaptation via LoRA: freezing SAM2's image encoder and tuning only low-rank layers is enough to transfer natural-image pretraining to medical modalities, per the reported Dice gains.","3D handled as video: treating axial slices as a temporal sequence makes SAM2's video memory directly applicable to CT, MRI, and PET volumes, with adjacent-slice context stored in a memory bank.","Support-set size is a tunable cost-accuracy dial: larger support sets improve reported Dice and IoU up to a point, with size 4 chosen as the default trade-off.","Stronger than fully supervised baselines on these benchmarks: the reported Dice scores exceed nnUNet, SwinUNet, and UNETR as well as SAM2 and MedSAM2 on all evaluated 2D and 3D datasets."],"supporting_citations":[{"why":"Supplies the SAM2 architecture, including the image encoder, prompt encoder, mask decoder, and the memory mechanism that the PMG and 3D modules adapt.","marker":"[12]"},{"why":"Introduces the original SAM promptable segmentation paradigm that SAM2 extends.","marker":"[2]"},{"why":"Supplies the low-rank adaptation strategy used to fine-tune the frozen image encoder and mitigate domain shift.","marker":"[27]"},{"why":"Supplies the mask-attention design that the Pseudo-mask Attention module adapts by replacing the coarse mask with the generated pseudo-mask.","marker":"[29]"},{"why":"Provides additional mask-attention inspiration used in the pseudo-mask attention module.","marker":"[28]"},{"why":"Supplies the hierarchical pixel decoder that recovers full-resolution predictions, addressing SAM2's 4x downsampled decoder output.","marker":"[30]"},{"why":"Demonstrates combining an in-context learning model with SAM for medical segmentation, motivating the support-set conditioning approach.","marker":"[26]"},{"why":"nnUNet is the strongest fully supervised baseline that the framework claims to outperform.","marker":"[41]"},{"why":"MedSAM2 is the primary prompt-based SAM2 medical segmentation baseline compared in the experiments.","marker":"[16]"}],"fun_headline_variants":["SAM2's video memory powers prompt-free medical segmentation","Support-set pseudo-masks replace manual prompts in SAM2","SAM2-SGP: no prompts, just support sets for medical imaging","Medical segmentation: SAM2 memory banks beat nnUNet","Prompt-free SAM2 uses support memories to outscore MedSAM2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the K most similar support images can be selected reliably, but the paper never defines the similarity function it uses, so the pseudo-mask, the box prompt, and the final segmentation all inherit whatever errors that unspecified ranking introduces.","fun_headline_variants_meta":{"raw":{"variants":["SAM2's video memory powers prompt-free medical segmentation","Support-set pseudo-masks replace manual prompts in SAM2","SAM2-SGP: no prompts, just support sets for medical imaging","Medical segmentation: SAM2 memory banks beat nnUNet","Prompt-free SAM2 uses support memories to outscore MedSAM2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1479,"prompt_tokens":1068,"completion_tokens":411,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":326}},"tokens_in":684,"tokens_out":411,"duration_ms":4035,"temperature":1.0,"reasoning_tokens":326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:28:43.839533+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the reported 2D benchmarks again with the K support samples chosen by a random or deliberately poor ranking instead of the paper's similarity criterion. If Dice does not drop substantially, then the claimed support-set-guided prompting mechanism is not what drives the gains; if it does drop, the missing similarity definition becomes the decisive implementation detail.","supporting_citations":[{"cited_title":"Universeg: Universal medical image segmentation,","cited_arxiv_id":null,"evidence_quote":"Supplies the low-rank adaptation strategy used to fine-tune the frozen image encoder and mitigate domain shift."},{"cited_title":"Masked-attention mask transformer for universal image segmenta- tion,","cited_arxiv_id":null,"evidence_quote":"Supplies the mask-attention design that the Pseudo-mask Attention module adapts by replacing the coarse mask with the generated pseudo-mask."},{"cited_title":"Mask3d: Mask transformer for 3d semantic instance segmentation,","cited_arxiv_id":null,"evidence_quote":"Provides additional mask-attention inspiration used in the pseudo-mask attention module."},{"cited_title":"Icl-sam: Synergizing in-context learning model and sam in medical image segmentation,","cited_arxiv_id":null,"evidence_quote":"Demonstrates combining an in-context learning model with SAM for medical segmentation, motivating the support-set conditioning approach."},{"cited_title":"Medical sam 2: Segment medical images as video via segment anything model 2,","cited_arxiv_id":null,"evidence_quote":"MedSAM2 is the primary prompt-based SAM2 medical segmentation baseline compared in the experiments."}],"review_version":2}