Pith. sign in

REVIEW 4 major objections 4 minor 64 references

Proxy Prompt: Endowing SAM and SAM 2 with Auto-Interactive-Prompt for Medical Segmentation

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A learned 'proxy prompt' from annotated non-target images can steer a frozen SAM or SAM 2 to segment medical images and video as well as manual point, box, or mask prompts, matching fully trained models with 16 labeled image-mask pairs.

desk verdict A useful and genuinely novel plug-in for reference-prompted SAM/SAM2, but the SOTA claim needs head-to-head comparisons and the generalization claim needs a proper cross-dataset test. read the letter →

arxiv 2502.03501 v3 pith:NMQ5D7B3 submitted 2025-02-05 eess.IV cs.LG

Wang Xinyi , Kang Hongyu , Wei Peishan , Shuai Li , Yu Sun , Sai Kit Lam , Yongping Zheng This is my paper · ORCID
classification eess.IVcs.LG
keywords proxypromptSAM2medicalimagesegmentationfew-shotvisionmambaautomaticpromptingcross-attentionultrasoundvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SAM and SAM 2 segment whatever a user points at, but in a busy clinic pointing at every image or frame is exactly the bottleneck. This paper claims that bottleneck can be removed: a 'proxy prompt' — a high-dimensional embedding computed from an annotated image of another subject, not the one being examined — can replace manual point, box, or mask prompts entirely, with SAM's and SAM 2's original weights left frozen. The authors build a Proxy Prompt Generator that selects the most target-relevant context from the support image-mask pair (Contextual Selective Module) and then 'colorizes' it with the user's object choice via dual cross-attention (Contextual Colorization Module). Trained on only 16 image-mask pairs, the method reports 85.9% average Dice across five medical segmentation objects, above both the few-shot baselines (best 80.7%) and a fully trained upper-bound model (83.3%); on ultrasound video it reports 80.9% average Dice with 0.3% standard deviation across five different support pairs. If these results hold, clinicians could pre-load a prompt once and then switch targets or cases by swapping an annotated support pair, with no per-case prompting and no retraining.

What carries the argument

The load-bearing object is the proxy prompt $P \in \mathbb{R}^{N \times ((H \times W)/16) \times C}$: $N$ high-dimensional embeddings, one per requested object, computed from a support set of image-mask pairs and fed into the frozen decoder. The selection machinery is the Selective Map (Eq. 6), a channel-normalized correlation $$\mathrm{Selective} = \frac{2\left(\hat{F}_{\mathrm{sup}}^{T}\hat{F}_x\right) - \hat{F}_{\mathrm{sup}}^{2}}{\sqrt{C}},$$ computed between flattened support features $\hat{F}_{\mathrm{sup}}$ and target features $\hat{F}_x$; after softmax normalization it weights the aggregated support features $A_{\mathrm{agg}}$ to yield the contextual embedding $E_{ctx}$ (Eq. 7), so only support content relevant to the target survives. The interpretation machinery is the Contextual Colorization Module: four blocks in which target features read from the context by cross-attention and the context then reads back from the refined target features by reversed cross-attention, so the embedding absorbs the user's object choice from the support mask along with target appearance — conceptually, 'coloring' the target with the support's annotation. A Vision Mamba encoder supplies the first selection step, its input-dependent state-space parameters serving as the input-driven filter; a bridge unit with CBAM channel-spatial attention supplies the second step, communicating features across objects before the Selective Map applies the final target-relevance filter.

What would settle it

Give the trained generator a target image and, on a fixed support image, two masks denoting different objects (e.g., optic disc vs. vessels, or fetal head vs. pubic symphysis): if the output does not switch cleanly with the mask, the prompt is not carrying user intent. The sharper probe is already half-run by the authors — a support image from a different modality drops Dice to 65.25% — so a systematic sweep over modality, anatomy, and scanner mismatch would show exactly where the 'non-target data' claim fails and whether 16 training images are enough to survive any of those shifts.

Watch

Extended reading notes

Core claim

The paper's claim, stated sympathetically, is that prompting a segment-anything model is a representation problem rather than a user-interface problem: the information a user would put into a point, box, or mask can instead be learned and delivered as an embedding, and that embedding can be produced from data that never touches the target. The same mechanism is claimed to work for images (SAM) and video (SAM 2), because both models accept prompt inputs through a prompt encoder or memory-attention channel; the Proxy Prompt Generator feeds the refined embedding directly into the SAM decoder (Eq. 12) and into SAM 2's memory attention. The experimental claim is that this yields state-of-the-art few-shot performance — 85.9% average Dice with 16 training image-mask pairs on REFUGE2 (disc/cup), STARE (vessels), and FPA (fetal head and pubic symphysis), versus 80.7% for the best prior few-shot method and 83.3% for fully trained models — that it transfers to 3D MRI (87.4% vs. 88.1% fully supervised), and that on real-time ultrasound video it is both accurate (80.9%) and stable (0.3% standard deviation across support pairs vs. 10.9% for the comparison method). The paper further claims the design is model-agnostic: it improved SAM-ViT-B, SAM-ViT-H, and MedSAM alike, and can be reattached as newer foundation models appear.

Load-bearing premise

The load-bearing premise is that a support image-mask pair from the same modality and roughly the same anatomy carries all the object information a manual point or box would carry, and that this information survives compression into a high-dimensional embedding — the paper trains only on 16 image-mask pairs, tests only within matching modalities, and documents the premise failing for an unrelated support (a retinal scan guiding an ultrasound target, 65.25% Dice).

Editorial extensions

If this is right

  • A single annotated support pair replaces per-image prompting: with the proxy prompt, the same frozen model segments every image or video frame of a study, which is what makes real-time ultrasound guidance feasible.
  • Task switching without retraining: because the support mask encodes the object choice, clinicians can move from segmenting optic disc/cup to vessels, or from fetal head to pubic symphysis, by changing the support pair rather than training a new model.
  • Few-shot medical segmentation reaches full-data parity: 16 image-mask pairs (58 MRI slices) suffice for average Dice scores comparable to or better than models trained on the complete datasets (85.9% vs. 83.3% on images; 87.4% vs. 88.1% on MRI).
  • The prompting strategy is transferable across foundation models: applying the same generator to SAM-ViT-B, SAM-ViT-H, and MedSAM improved all three, with the largest gains on models without a conflicting prompt prior.
  • Video segmentation becomes pre-loadable: because the prompt derives from non-target data, it can be prepared before the exam starts, then applied as memory prompts frame after frame.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper trains one generator per modality rather than per object, so the natural next test is whether the support-pair machinery transfers across modalities; the authors' own experiment with an unrelated support (65.25% Dice) marks a boundary worth mapping systematically, e.g., MRI-supported ultrasound or fundus-supported endoscopy.
  • The finding that MedSAM gains less than SAM (and loses ground on FPA, its pretraining domain) implies the proxy prompt is most effective on models without a strong learned prompt prior; attaching this module to foundation models heavily trained on box or point prompts should be expected to yield diminishing returns.
  • Because the prompt enters through SAM's standard prompt channel and SAM 2's memory attention, the same generator could in principle steer any promptable segmenter with a comparable interface — a generalization the paper gestures at with its 'plug-and-play' framing but does not demonstrate.
  • The near-zero variance across support pairs on video (0.3% std) hints that the Selective Map actively discards support-specific appearance; a testable extension is whether stability survives when the support image is a different subject, a different scanner, or a compressed frame, conditions under which manual point prompts typically degrade.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Proxy Prompt (PP), a plug-in module called the Proxy Prompt Generator (PPG) that converts a support image-mask pair drawn from non-target data into a high-dimensional embedding prompt for frozen SAM and SAM 2. The PPG consists of a Contextual Selective Module (CSM), which uses a Vision Mamba encoder, a bridge unit, and a selective map to extract relevant contextual information from the support pair, and a Contextual Colorization Module (CCM), which refines the contextual embedding through dual reverse cross-attention with target features. The method is evaluated on five public datasets (REFUGE2, STARE, FPA, PROMISE12, JNU-IFM) using a few-shot setup with 16 training images or videos, and the authors report state-of-the-art performance and results comparable to fully trained models, along with extensive ablations of the modules, prompt quality, training size, and support-pair quantity.

Significance. If the central claim is established, the method addresses a real clinical need by replacing manual per-target prompting with a pre-annotated support pair and by enabling task switching without retraining. The paper is strong in its breadth of experiments and internal consistency: the ablation tables are detailed, the stability analysis on video data with multiple support pairs is valuable, and the training strategy that alternates support and target roles within the same dataset is a sensible way to avoid obvious target-leakage or circularity. The low trainable-parameter count of the Vision Mamba based CSM is also an attractive feature. The main weakness is that the central generalization claim, namely that prompts from non-target data transfer broadly, is only demonstrated for support pairs drawn from the same training distribution as the target.

major comments (4)
  1. [Sec. 7.1 and Sec. 11.2, Fig. 10] The evidence for the central claim that non-target data can guide target segmentation is entirely within-distribution. At inference, support pairs are randomly selected from the same training dataset (Sec. 7.1), and during training, support and target images are alternated within the same dataset. The only out-of-distribution test, Fig. 10 (SI5), changes both modality and anatomy, and the Dice score collapses from about 95.5% to 65.25%. The realistic retrospective-data scenario, namely a different scanner, protocol, or population within the same modality and anatomy, is never tested. This is load-bearing because the abstract promises guidance from non-target data generally. Please add same-modality cross-dataset support experiments (for example, a fundus support pair from REFUGE2 used for STARE targets, or an ultrasound support pair from one scan protocol used for FPA targets) and temper the abstract and conclusion to the demonstrated scope until such evidence is available.
  2. [Sec. 6.1 and Table 1] The comparison protocol for point and box baselines gives those baselines near-ideal prompts computed from ground-truth masks (one random point inside the target mask, or the minimum bounding rectangle of the target), while the proposed method uses a support pair that contains no target-specific ground truth. This is conservative with respect to the proposed method and should be acknowledged explicitly in the main text. In addition, the 'Upper' rows in Tables 1 and 2 are taken from results reported in the original papers (BEAL, nnUnet, SegNet, DSD-FCN) under their own protocols, not re-run on the same test splits. To support the claim of being 'comparable with fully-trained models', either re-run the Upper methods on the same split or state clearly that the comparison is against literature-reported numbers.
  3. [Eq. (6), Sec. 3.1] The selective-map equation has a shape inconsistency. The first term, Fsup^T * Fx, has shape (K*H*W) x (H*W), while the second term, Fsup^2, is not defined in a way that permits subtraction from that matrix. If Fsup^2 is intended to denote a broadcast row-wise squared norm as in STCN, the notation should make that explicit; if it is elementwise squaring, the operation is dimensionally invalid. Because Eq. (6) defines the core selective map, the formula needs to be unambiguous and must match the implementation.
  4. [Sec. 2 and Tables 1-3] The closest prompt-from-reference methods, including VRP-SAM, ProtoSAM, EviPrompt, and PerSAM, are discussed in the related work and Sec. 10 but do not appear in any quantitative comparison. Since the abstract claims state-of-the-art performance, the manuscript should include at least one quantitative comparison with these methods under the same few-shot protocol, or provide an explicit and justified statement of why such a comparison is not applicable.
minor comments (4)
  1. [Abstract] The standalone abstract says 'four public datasets' while the main-text abstract says 'five public datasets'; the experiments actually cover five datasets (REFUGE2, STARE, FPA, PROMISE12, JNU-IFM). Please reconcile the count.
  2. [Sec. 11.6] The text says 'To evaluate the impact of training set size K' but then defines M in {2, 4, 8, 16}; the notation for training-set size and inference support-set size is used inconsistently and should be fixed.
  3. [Eqs. (3)-(5), Sec. 3.1] The tensor notation for the multi-object case is confusing: V is written as R^{K x Cv x H x W}, but the text says N feature matrices are produced, and Eq. (4) introduces Fcat with both K and N. Please clarify the roles of K and N consistently in the equations and in Fig. 3.
  4. [Figure numbering] The main-text ablation refers to Fig. 13 for training-size results, but the same figure appears as Fig. 6 in the body; supplementary figures and main-text figures should have consistent numbering.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proxy-prompt derivation is self-contained; support and target are distinct data and the target mask is only a training label, not an input.

full rationale

The derivation chain is not circular. The PPG takes support image-mask pairs and a target image, and the training loss in Eq. 13 supervises the target prediction against the target ground-truth mask; the support mask is an input prompt, while the target mask is a label, so the prediction is not self-definitional. At inference, Sec. 7.1 states that support pairs are randomly selected from the training dataset, but targets are held-out test samples from different subjects or videos, so the reported Dice values are not forced by construction. The selective-map mechanism in Eqs. 6-7 is an attention-like weighting of support features by target-feature similarity, not an identity map from the target label to the output, and no fitted parameter is renamed as a prediction. I found no load-bearing self-citation or imported uniqueness theorem; the citation to [5] is only for the selective-map formula, and the dataset citations are external. The paper itself documents the relevant limitation at Sec. 11.2 and Fig. 10: replacing the support with an unrelated retinal scan drops Dice to 65.25%, confirming that support-target relevance is required; this narrows the practical scope of the 'non-target data' claim but does not make the central few-shot claim circular. The only same-image case (SI1, MSE=0, Fig. 10) is an ablation extreme used to illustrate the selective map, not a claimed prediction, and the surrounding SI2-SI4 results use non-identical support images. Overall, the main derivation is self-contained against external benchmarks.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities or forces. The only new objects are the CSM and CCM modules, which are standard neural components. The free parameters are mostly standard training hyperparameters, but the LoRA rank and the Vision Mamba size are load-bearing for the parameter efficiency claim, and neither is varied in the ablations.

free parameters (3)
  • LoRA rank and position = Not stated
    LoRA is applied to the SAM encoder and decoder but the rank, alpha, and target modules are unspecified; the few-shot results depend on these choices.
  • Vision Mamba encoder patch size and embedding dimension = Patch 16, embedding 192
    Reported in Section 7.1, but without a sensitivity analysis, so the contribution of these specific choices is not isolated.
  • Number of CCM blocks = 4
    Stated as four identical blocks with no ablation showing that four is optimal.
assumptions (3)
  • domain assumption Vision Mamba's input-dependent SSM selection improves medical feature extraction over a plain ViT at lower parameter count.
    The CSM relies on this assumption; the comparison against ViT and ResNet-50 uses different parameter counts, and the gain over ViT is 1.4% Dice, so the assumption is plausible but not established as causal.
  • domain assumption A frozen SAM or SAM 2 decoder can consume a high-dimensional embedding prompt that is not a point, box, or mask in the training distribution.
    The entire method depends on this. The paper shows it works empirically on five datasets, but does not analyze whether the Decoder's prompt encoder assumption is violated.
  • domain assumption Training with alternating support and target roles on 16 images generalizes to new patients and modalities within the same dataset.
    The method relies on this few-shot generalization claim; the reported gains are consistent but the test sets are small in some cases (STARE has 20 images).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Proxy Prompt: Endowing SAM and SAM 2 with Auto-Interactive-Prompt for Medical Segmentation." pith.science (2026). https://pith.science/paper/NMQ5D7B3

@misc{pith2026250203501,
  author       = {Pith},
  title        = {Pith review of: Proxy Prompt: Endowing SAM and SAM 2 with Auto-Interactive-Prompt for Medical Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMQ5D7B3}},
  note         = {Machine review of arXiv:2502.03501}
}
read the original abstract

In this paper, we aim to address the unmet demand for automated prompting and enhanced human-model interactions of SAM and SAM2 for the sake of promoting their widespread clinical adoption. Specifically, we propose Proxy Prompt (PP), auto-generated by leveraging non-target data with a pre-annotated mask. We devise a novel 3-step context-selection strategy for adaptively selecting the most representative contextual information from non-target data via vision mamba and selective maps, empowering the guiding capability of non-target image-mask pairs for segmentation on target image/video data. To reinforce human-model interactions in PP, we further propose a contextual colorization module via a dual-reverse cross-attention to enhance interactions between target features and contextual-embedding with amplifying distinctive features of user-defined object(s). Via extensive evaluations, our method achieves state-of-the-art performance on four public datasets and yields comparable results with fully-trained models, even when trained with only 16 image masks.

Figures

Figures reproduced from arXiv: 2502.03501 by the authors.

Figure 1
Figure 1. Illustration of comparison without/with PP in (a-b) SAM [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Schematic differences of traditional prompt encoder and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Designed Proxy Prompt Generator for both SAM 2 and SAM. Our key designed focus on the Contextual Selective Module and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Visualization comparison results of nine models across five objects. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Selective Map visualization on support images with dif [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The boxes show performance for each M with varying K, while in the rightmost plots (G < M−1), higher peaks refer to better performance and narrower ranges reflect improved stability. Optimum is achieved when both M and K are maximized. 5. Conclusion We present the PPG,…
Figure 7
Figure 7. Figure 7: Quantitative comparison results on four representative examples. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Comparison on the video dataset using five different support pairs, showing the average results for two objects. Results show that [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Comparison on one representative video. The top left corner of each subplot displays the Dice scores for the segmentation of the [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Visualization of Selective Map across different support images. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Line chart of model segmentation performance across varying prompt quality. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Six levels of support masks for 16 support images, with the gray area denoting the disc object and the black area indicating the [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Enlarged box plots and ridge plots (rightmost column) [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 45 canonical work pages

  1. [1]

    Proto- sam: One-shot medical image segmentation with founda- tional models

    Lev Ayzenberg, Raja Giryes, and Hayit Greenspan. Proto- sam: One-shot medical image segmentation with founda- tional models. ArXiv, abs/2407.07042, 2024. 3

    arXiv 2024

  2. [2]

    Segnet: A deep convolutional encoder-decoder architecture for image segmentation

    Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern anal- ysis and machine intelligence, 39(12):2481–2495, 2017. 12

    work page 2017

  3. [3]

    Uni- verseg: Universal medical image segmentation

    Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Uni- verseg: Universal medical image segmentation. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 21438–21451, 2023. 15

    work page 2023

  4. [4]

    Seg- mentation by registration-enabled sam prompt engineering using five reference images

    Yaxi Chen, Aleksandra Ivanova, Shaheer U Saeed, Rikin Hargunani, Jie Huang, Chaozong Liu, and Yipeng Hu. Seg- mentation by registration-enabled sam prompt engineering using five reference images. In International Workshop on Biomedical Image Registration , pages 241–252. Springer,

    work page

  5. [5]

    Rethink- ing space-time networks with improved memory coverage for efficient video object segmentation

    Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethink- ing space-time networks with improved memory coverage for efficient video object segmentation. Advances in neural information processing systems, 34:11781–11794, 2021. 5

    work page 2021

  6. [6]

    Refuge2 challenge: A treasure trove for multi- dimension analysis and evaluation in glaucoma screening

    Huihui Fang, Fei Li, Junde Wu, Huazhu Fu, Xu Sun, Jaemin Son, Shuang Yu, Menglu Zhang, Chenglang Yuan, Cheng Bian, et al. Refuge2 challenge: A treasure trove for multi- dimension analysis and evaluation in glaucoma screening. arXiv preprint arXiv:2202.08994, 2022. 6, 11

    arXiv 2022

  7. [7]

    Isuog practice guide- lines: intrapartum ultrasound

    T Ghi, T Eggebø, C Lees, K Kalache, P Rozenberg, A Youssef, LJ Salomon, and B Tutschek. Isuog practice guide- lines: intrapartum ultrasound. Ultrasound in Obstetrics & Gynecology, 52(1):128–139, 2018. 2

    work page 2018

  8. [8]

    Modeling Sequences with Structured State Spaces

    Albert Gu. Modeling Sequences with Structured State Spaces. Stanford University, 2023. 3

    work page 2023

Show all 64 references
  1. [9]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 3

    2023 arXiv

  2. [10]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5

    2016

  3. [11]

    Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response

    AD Hoover, Valentina Kouznetsova, and Michael Gold- baum. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Trans- actions on Medical imaging, 19(3):203–210, 2000. 6, 11

    2000

  4. [12]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 3, 4, 6

    2021 arXiv

  5. [13]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Pe- tersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation. Nature methods, 18(2):203–211, 2021. 12

    2021

  6. [14]

    Pubic Symphysis-Fetal Head Segmentation and Angle of Progression, 2023

    Bai Jieyun and Ou ZhanHong. Pubic Symphysis-Fetal Head Segmentation and Angle of Progression, 2023. 6, 11

    2023

  7. [15]

    Vm-ddpm: Vision mamba diffusion for medical image synthesis

    Zhihan Ju and Wanting Zhou. Vm-ddpm: Vision mamba diffusion for medical image synthesis. arXiv preprint arXiv:2405.05667, 2024. 3

    2024 arXiv

  8. [16]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 1, 6

    2023

  9. [17]

    Visual in-context prompting

    Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Jianwei Yang, Chunyuan Li, et al. Visual in-context prompting. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12861–12871, 2024. 15

    2024

  10. [18]

    Evaluation of prostate segmentation algorithms for mri: the promise12 challenge

    Geert Litjens, Robert Toth, Wendy Van De Ven, Caroline Hoeks, Sjoerd Kerkstra, Bram Van Ginneken, Graham Vin- cent, Gwenael Guillard, Neil Birbeck, Jindang Zhang, et al. Evaluation of prostate segmentation algorithms for mri: the promise12 challenge. Medical image analysis , 1...

    2014

  11. [19]

    The jnu-ifm dataset for segmenting pubic symphysis-fetal head

    Yaosheng Lu, Mengqiang Zhou, Dengjiang Zhi, Minghong Zhou, Xiaosong Jiang, Ruiyu Qiu, Zhanhong Ou, Huijin Wang, Di Qiu, Mei Zhong, Xiaoxing Lu, Gaowen Chen, and Jieyun Bai. The jnu-ifm dataset for segmenting pubic symphysis-fetal head. Data in Brief, 41:107904, 2022. 6, 11

    2022

  12. [20]

    Semi-supervised medical image segmentation through dual- task consistency

    Xiangde Luo, Jieneng Chen, Tao Song, and Guotai Wang. Semi-supervised medical image segmentation through dual- task consistency. In Proceedings of the AAAI conference on artificial intelligence, pages 8801–8809, 2021. 6 9

    2021

  13. [21]

    Fer-yolo-mamba: Facial expression detection and classi- fication based on selective state space

    Hui Ma, Sen Lei, Turgay Celik, and Heng-Chao Li. Fer-yolo-mamba: Facial expression detection and classi- fication based on selective state space. arXiv preprint arXiv:2405.01828, 2024. 3

    2024 arXiv

  14. [22]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024. 3

    2024

  15. [23]

    U-mamba: Enhancing long-range dependency for biomedical image segmentation

    Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722, 2024. 3

    2024 arXiv

  16. [24]

    V-net: Fully convolutional neural networks for volumetric medical image segmentation

    Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016. 6

    2016

  17. [25]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 1

    2024 arXiv

  18. [26]

    Auto- mated localisation of optic disk and fovea in retinal fundus images

    S Sekhar, Waleed Al-Nuaimy, and Asoke K Nandi. Auto- mated localisation of optic disk and fovea in retinal fundus images. In 2008 16th European Signal Processing Confer- ence, pages 1–5. IEEE, 2008. 2

    2008

  19. [27]

    Autosam: Adapting sam to medical images by overloading the prompt encoder

    Tal Shaharabany, Aviad Dahan, Raja Giryes, and Lior Wolf. Autosam: Adapting sam to medical images by overloading the prompt encoder. arXiv preprint arXiv:2306.06370, 2023. 2, 3

    2023 arXiv

  20. [28]

    Vrp-sam: Sam with visual reference prompt

    Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, Gang Zhang, Errui Ding, Jingdong Wang, and Zechao Li. Vrp-sam: Sam with visual reference prompt. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 23565–23574, 2024. 13, 15

    2024

  21. [29]

    Survey on segmentation and classification approaches of optic cup and optic disc for diagnosis of glaucoma

    Niharika Thakur and Mamta Juneja. Survey on segmentation and classification approaches of optic cup and optic disc for diagnosis of glaucoma. Biomedical Signal Processing and Control, 42:162–189, 2018. 2

    2018

  22. [30]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 5

    2017

  23. [31]

    Deeply supervised 3d fully convolutional networks with group dilated convolution for automatic mri prostate segmentation

    Bo Wang, Yang Lei, Sibo Tian, Tonghe Wang, Yingzi Liu, Pretesh Patel, Ashesh B Jani, Hui Mao, Walter J Curran, Tian Liu, et al. Deeply supervised 3d fully convolutional networks with group dilated convolution for automatic mri prostate segmentation. Medical physics, 46(4):1707–1718,

  24. [32]

    Review of large vision models and visual prompt engineering

    Jiaqi Wang, Zhengliang Liu, Lin Zhao, Zihao Wu, Chong Ma, Sigang Yu, Haixing Dai, Qiushi Yang, Yiheng Liu, Songyao Zhang, et al. Review of large vision models and visual prompt engineering. Meta-Radiology, page 100047,

  25. [33]

    Boundary and entropy-driven ad- versarial learning for fundus image segmentation

    Shujun Wang, Lequan Yu, Kang Li, Xin Yang, Chi-Wing Fu, and Pheng-Ann Heng. Boundary and entropy-driven ad- versarial learning for fundus image segmentation. In Medi- cal Image Computing and Computer Assisted Intervention– MICCAI 2019: 22nd International Conference, Shenzhen, ...

    2019

  26. [34]

    Consistency-guided meta- learning for bootstrapping semi-supervised medical image segmentation

    Qingyue Wei, Lequan Yu, Xianhang Li, Wei Shao, Cihang Xie, Lei Xing, and Yuyin Zhou. Consistency-guided meta- learning for bootstrapping semi-supervised medical image segmentation. In International conference on medical image computing and computer-assisted intervention , pages 183–

  27. [35]

    Cbam: Convolutional block attention module

    Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018. 5

    2018

  28. [36]

    One-prompt to segment all med- ical images

    Junde Wu and Min Xu. One-prompt to segment all med- ical images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11302– 11312, 2024. 15

    2024

  29. [37]

    Medical sam adapter: Adapting seg- ment anything model for medical image segmentation.arXiv preprint arXiv:2304.12620, 2023

    Junde Wu, Wei Ji, Yuanpei Liu, Huazhu Fu, Min Xu, Yanwu Xu, and Yueming Jin. Medical sam adapter: Adapting seg- ment anything model for medical image segmentation.arXiv preprint arXiv:2304.12620, 2023. 3

    2023 arXiv

  30. [38]

    Self-prompting large vision models for few-shot medical image segmenta- tion

    Qi Wu, Yuyao Zhang, and Marawan Elbatel. Self-prompting large vision models for few-shot medical image segmenta- tion. In MICCAI workshop on domain adaptation and rep- resentation transfer, pages 156–167. Springer, 2023. 2, 3

    2023

  31. [39]

    Eviprompt: A training-free evidential prompt generation method for segment anything model in medical images

    Yinsong Xu, Jiaqi Tang, Aidong Men, and Qingchao Chen. Eviprompt: A training-free evidential prompt generation method for segment anything model in medical images. arXiv preprint arXiv:2311.06400, 2023. 3, 13

    2023 arXiv

  32. [40]

    Tavp: Task-adaptive visual prompt for cross-domain few-shot segmentation

    Jiaqi Yang, Yaning Zhang, Jingxi Hu, Xiangjian He, Linlin Shen, and Guoping Qiu. Tavp: Task-adaptive visual prompt for cross-domain few-shot segmentation. arXiv preprint arXiv:2409.05393, 2024. 15

    2024 arXiv

  33. [41]

    Customized segment any- thing model for medical image segmentation

    Kaidong Zhang and Dong Liu. Customized segment any- thing model for medical image segmentation. arXiv preprint arXiv:2304.13785, 2023. 2, 3

    2023 arXiv

  34. [42]

    Personalize segment anything model with one shot

    Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junt- ing Pan, Xianzheng Ma, Hao Dong, Peng Gao, and Hong- sheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023. 3, 15

    2023 arXiv

  35. [43]

    Fd-vision mamba for endoscopic exposure correction

    Zhuoran Zheng and Jun Zhang. Fd-vision mamba for endoscopic exposure correction. arXiv preprint arXiv:2402.06378, 2024. 3

    2024 arXiv

  36. [44]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 3, 4, 5, 11

    2024 arXiv

  37. [45]

    Segment everything everywhere all at once

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36:19769–19782,

  38. [48]

    Settings for Image Dataset

    Prompt Strategy 6.1. Settings for Image Dataset. For the models that require prompts in the comparison ex- periments on image dataset, the following prompt condi- tions are provided

  39. [49]

    Point: Since the center point of the disc and vessel mask is not on the target, one random point within the target mask as a positive point prompt

  40. [51]

    We select the prediction that the highest overlaps with ground truth to calculate the model’s Dice score

    Everything: Automatically segment multiple targets with everything mode. We select the prediction that the highest overlaps with ground truth to calculate the model’s Dice score

  41. [52]

    Others: The SAMed and AutoSAM models are de- signed to perform automatic segmentation without manual- given prompt, while we use the support image-mask pair as a prompt. 6.2. Settings for Video Dataset. Considering the high demand for timely operation in real-time image-guided...

  42. [53]

    Box: Minimum bounding rectangle of the target as a bounding box prompt

  43. [54]

    Point: Center point of the object as a positive point

  44. [55]

    Mask: Target mask of the patient under examination

  45. [56]

    non-target

    Support Image-Mask Pair: Image of the first frame and the corresponding target mask from “non-target” data (i.e., image/video frame of subjects other than the one under examination, such as from retrospective datasets). Dataset Modality Segmentation Objects Samples REFUGE2 [6]...

    2000

  46. [57]

    Datasets Introduction The type of image modalities, segmentation objects, and number of samples for the four included datasets are sum- marized in the table below. REFUGE2 [6], STARE [11] and FPA [14] are image datasets evaluated in the image segmen- tation task, whereas PROMI...

  47. [58]

    3D DSD-FCN [31] and MLB-Seg are existing SOTA models for PROMISE12 [18] dataset under fully supervised and semi-supervised settings, respectively

    Experiment Details: Comparison with SOTA on MRI Dataset We extensively evaluated various methods, including semi- supervised models (DTC, MLB-Seg), SAM-based mod- els (Med-SA, SAMed, AutoSAM), traditional segmentation models (nnUNet), and 3D DSD-FCN [31], trained on the full d...

  48. [59]

    Given that SAM2 is pretrained on a large-scale video dataset, it was a natural choice to integrate our method into SAM2 for video segmentation

    Experiment Details: Comparison with SOTA on Video Dataset Beyond evaluating our model on image and 3D datasets, we further investigated its performance on video data. Given that SAM2 is pretrained on a large-scale video dataset, it was a natural choice to integrate our method ...

  49. [60]

    #𝐹𝑟𝑎𝑚𝑒!"$%𝐹𝑟𝑎𝑚𝑒!

    Discussion of related work We also note that in the domain of natural image datasets, some works share a similar paradigm to ours [28]. However, our approach differs significantly from VRP-SAM [28] in both motivation and methodology. Specifically, our study addresses a clinica...

  50. [61]

    For our proposed modules, we conducted ablations on CSM and CCM, visualized the effectiveness of the Selec- tive Map , and analyzed different encoder architectures within CSM

    Detailed settings and results: Ablation study We structured our ablation studies into three key aspects: our proposed modules , the retrospective image-mask pair prompt strategy, and methodological parameters. For our proposed modules, we conducted ablations on CSM and CCM, vi...

  51. [62]

    Using box prompts, SAM-ViT-B, SAM-ViT-H, and MedSAM demonstrate a progressive improvement (56.1% → 60.3% → 74.5%) in average Dice scores. This trend reflects MedSAM’s advantage in medical segmenta- tion due to extensive domain-specific training and SAM- ViT-H’s superior genera...

  52. [63]

    The most notable improvement is on STARE-Vessel, where Dice increased by 39.3%

    Despite MedSAM’s medical pretraining, our method further improves its segmentation performance on unseen datasets using only 16 support images. The most notable improvement is on STARE-Vessel, where Dice increased by 39.3%. This is because MedSAM strug- gles with vessel-like b...

  53. [64]

    According to MedSAM’s supplementary mate- rials, its pretraining included FPA and REFUGE datasets

    The extent of our method’s improvement on Med- SAM correlates with its pretraining exposure to simi- lar data. According to MedSAM’s supplementary mate- rials, its pretraining included FPA and REFUGE datasets. REFUGE, REFUGE2, and STARE all belong to the fun- dus imaging modal...

  54. [65]

    Surprisingly, Our Method Boosts SAM-ViT-H Be- yond MedSAM:Although our prompt improves MedSAM, it benefits SAM-ViT-H even more, enabling it to outperform MedSAM in segmentation (85.9% v.s. 81.3%). This con- tradicts our initial assumption that a medically pretrained model shou...

  55. [193]

    Springer, 2023. 6, 7

    2023

  56. [2023]

    Quantitative comparison results on four representative examples

    15 10 Support Pair OutputGround Truth SupportPair OutputGround Truth GroundTruth SupportPair Output Support Pair OutputGroundTruth Figure 7. Quantitative comparison results on four representative examples

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.