Pith. sign in

REVIEW 4 major objections 7 minor 24 references

SAMed-2: Selective Memory Enhanced Medical Segment Anything Model

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAMed-2 claims a temporal adapter and a confidence-driven memory bank yield state-of-the-art medical segmentation on 21 internal and 10 external tasks.

desk verdict A sensible SAM-2 adaptation for medical segmentation with a promising memory mechanism, but an image-level split and missing baselines leave the headline gains unproven. read the letter →

arxiv 2507.03698 v1 pith:JTS37AZV submitted 2025-07-04 cs.CV

Zhiling Yan , Sifan Song , Dingjie Song , Yiwei Li , Rong Zhou , Weixiang Sun , Zhennong Chen , Sekeun Kim
show 6 more authors
Hui Ren Tianming Liu Quanzheng Li Xiang Li Lifang He Lichao Sun
This is my paper
classification cs.CV
keywords medicalimagesegmentationSAM-2segmentanythingtemporaladapterconfidence-drivenmemorymulti-taskzero-shottransferMedBank-100k
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single general-purpose medical image segmentation model can be built by taking SAM-2 and adding two mechanisms: a temporal adapter that lets the image encoder use correlations across slices or frames, and a confidence-driven memory bank that stores only high-certainty features during training and retrieves the most similar, most confident entries at inference. The authors argue that this combination handles noise in large-scale medical annotations, transfers to unseen datasets, and mitigates catastrophic forgetting—the loss of previously learned abilities when new tasks or modalities are added. To support the claim, they build MedBank-100k, a dataset of 122,594 frame-mask pairs spanning seven imaging modalities and 21 segmentation tasks, and report the highest average Dice scores among the compared methods on internal and external benchmarks, including a 10.53% improvement over MedSAM on external zero-shot tasks and an 87.61% reduction in annotation time in a human user study. If the claim holds, a single model could serve many medical segmentation tasks without per-task retraining.

What carries the argument

The load-bearing object is the confidence-driven memory mechanism, a bank of tuples storing mask features, positional encodings, predicted IoU confidences, and unconditioned image embeddings. Retrieval ranks stored entries by cosine similarity to the current image embedding plus a confidence bonus, then feeds the top-K mask features into SAM-2's memory attention to produce the conditioned embedding used by the mask decoder. Replacement keeps the bank fresh by swapping out the most similar stored entry whenever the incoming frame has higher confidence. A secondary mechanism is the temporal adapter, a residual module with a 3D convolution and down/up projection layers inserted in each image-encoder block, which lets the encoder exploit inter-slice or inter-frame context. Together these are what the paper credits for the gains over baselines.

What would settle it

Evaluate SAMed-2 on a held-out task whose anatomy and modality appear in neither the 21 internal tasks nor the 10 external tasks; if the confidence-driven memory retrieval does not beat a no-memory baseline or MedSAM there, the claimed transfer and forgetting benefits are not established.

Watch

Extended reading notes

Core claim

The paper's central claim is that selective memory, rather than more parameters or more data, is what lets a promptable segmentation model survive noisy, multi-modality medical pre-training. During training, SAMed-2 writes into its memory base only features whose predicted IoU confidence is high; when a new image arrives, it retrieves the K stored entries that maximize the sum of cosine similarity to the current embedding and a confidence bonus, then fuses them through memory attention. It also replaces a stored entry when a new frame is more similar to it and has higher confidence, so the bank stays diverse and clean. On the paper's measurements this yields the best average Dice on both the 21-task internal benchmark (0.7118) and the 10-task external benchmark (0.6938), with external zero-shot results 10.53% above MedSAM's.

Load-bearing premise

The whole approach rests on the assumption that high-confidence features stored from MedBank-100k training images are still the right reference points for never-seen test images, so retrieving them by similarity improves segmentation instead of pulling predictions toward the training distribution.

Editorial extensions

If this is right

  • A single pre-trained model can be applied to at least 21 medical segmentation tasks across seven modalities without task-specific retraining, with the best average Dice on both the internal and external benchmarks.
  • External zero-shot performance improves by 10.53% over MedSAM, suggesting that storing and retrieving high-confidence training features transfers to new patients and somewhat different imaging conditions.
  • The temporal adapter alone raises internal spleen Dice from 0.7656 to 0.8448 and external liver Dice from 0.7000 to 0.7680, indicating inter-slice context is a substantial part of the gain for volumetric data.
  • Larger memory banks (640 entries) beat smaller ones (0 or 16), implying that feature diversity in the bank, not just confidence filtering, helps multi-task generalization.
  • With SAMed-2 assistance, an expert's annotation time per frame drops from about 609 seconds to about 75 seconds, an 87.61% reduction, pointing toward a practical clinical workflow benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the paper never runs a true sequential continual-learning protocol, so the claim that memory mitigates catastrophic forgetting is untested; a natural experiment would train tasks one after another and measure old-task Dice.
  • Inference: because each external task shares a name with an internal task (Liver, Spleen, and others) and dataset sources are not reported, the 10.53% zero-shot gain may reflect inter-dataset variation rather than transfer to genuinely new anatomies or modalities; a stricter hold-out of an entire modality would settle this.
  • Inference: the retrieval ranking relies on the model's predicted IoU confidence, which may be miscalibrated on out-of-distribution images, so calibrating confidence on a small external sample could make the memory ranking more reliable.
  • Inference: the same selective-memory recipe could be lifted onto other promptable segmentation backbones or applied to video object segmentation, where the temporal adapter is already natural; this is not tested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes SAMed-2, a medical image segmentation model built on the SAM-2 architecture, with two main modifications: a temporal adapter inserted into the image encoder to exploit inter-slice or inter-frame correlations, and a confidence-driven memory mechanism that stores high-confidence training features and retrieves them at inference by similarity and confidence. The authors also introduce MedBank-100k, a collection of 122,594 frame-mask pairs spanning 21 tasks and seven modalities, and report internal and external Dice results, component ablations, few-shot scaling, and an annotation-time user study. The headline claims are state-of-the-art performance on internal and external benchmarks and a 10.53% improvement over MedSAM on external zero-shot tasks.

Significance. If the evaluation were properly controlled, the contributions would be interesting: a parameter-efficient temporal adapter for volumetric medical data, a confidence-gated memory bank that is methodologically distinct from simply fine-tuning SAM-2, and a new large-scale medical segmentation benchmark. The paper also releases code, which supports reproducibility. However, the current evaluation does not establish patient-level generalization or forgetting mitigation, and the external-validation claims rest on unverified dataset disjointness. These issues are fixable, but they are central to the paper's main claims.

major comments (4)
  1. [Sec. 4.2, Eqs. (3)-(7)] The 9:1 split in Section 4.2 is performed at the image level, not at the patient or volume level. For CT and MR volumes, slices from the same scan can therefore appear in both training and test sets, and because the memory mechanism stores training features and retrieves them by cosine similarity at inference (Eqs. 3-7), a test slice can retrieve near-duplicate training slices from the same volume. This would inflate the reported Dice scores through memorization rather than generalization, affecting both the internal results (Table 3) and the external zero-shot gains (Table 2). Please re-run the evaluation with patient-level or volume-level splits and report the resulting numbers, or justify why image-level splitting is sufficient for these volumetric datasets.
  2. [Sec. 4.3, Tables 1-2] The external validation set is described as representing 'new patients, imaging conditions, and tasks,' but no dataset-source table is provided, and seven of the ten external tasks (Liver, LiverTumor, Spleen, Prostate, Hippocampus, BrainTumor, ColonTumor) are also task names that appear in MedBank-100k's CT/MR counts in Table 1. Without evidence that the external datasets and patients are disjoint from MedBank-100k, the 10.53% improvement over MedSAM and the zero-shot claim cannot be distinguished from retrieval of already-seen task distributions. Please provide per-task dataset sources, patient-level disjointness verification, and an explicit overlap analysis between the external datasets and MedBank-100k.
  3. [Secs. 1 and 3.3, Eq. (7)] The paper repeatedly claims that the confidence-driven memory mechanism 'mitigates catastrophic forgetting' (Abstract, Section 1, Section 3.3, Conclusion), but no continual learning experiment is run. The model is trained jointly on all MedBank-100k tasks, and there is no sequential task-order protocol that evaluates retention of earlier tasks after learning new ones. The replacement rule in Eq. (7) is therefore untested as a forgetting-mitigation mechanism. Please add a continual-learning experiment with a defined task order and retention metrics, or remove the forgetting-mitigation claim from the paper.
  4. [Secs. 4.2-4.3, Tables 2-3] The claim of state-of-the-art performance is not supported by the current comparison protocol. The paper omits nnU-Net, which is cited in the related work as [8] and is the standard strong baseline for medical image segmentation; comparing only against a per-task U-Net understates the baseline strength. In addition, all reported Dice scores appear to come from a single run with no error bars, standard deviations, or significance tests. The internal average gap over MedSAM is 1.71%, which may be within run-to-run variability. Please add nnU-Net and report repeated-seed or bootstrap confidence intervals for the main tables.
minor comments (7)
  1. [Sec. 3.3, Eq. (3)] The symbol sigma in Eq. (3) is not defined; if it denotes the sigmoid function or a temperature, please state this explicitly and specify how the similarity score and the confidence term are weighted when selecting the top-K memory entries.
  2. [Sec. 4.1] The preprocessing description says that '2D images are randomly shuffled while preserving temporal/spatial relationships of video/3D slices.' Random shuffling would destroy temporal or spatial relationships; please clarify what is actually done to the ordering of slices.
  3. [Tables 2-3] The column header 'SAMU-Net' appears to be a missing space between 'SAM' and 'U-Net'; the comparison text in Section 4.2 lists SAM, SAM-2, MedSAM, MedSAM-2, and U-Net, so the tables should have six clearly labeled method columns.
  4. [Fig. 3] The few-shot scaling experiment is shown for only one external task (prostate) and without error bars or number of runs; please indicate the variance and state whether the trend holds on other external tasks.
  5. [Sec. 4.5, Fig. 4] The human user study is based on a single expert annotator and a single in-house CMR dataset. Please report the dataset size, annotation protocol, number of frames, and whether the time difference was tested statistically, or describe the result as a pilot study.
  6. [Table 5] The memory-base-size ablation is unclear for the '0' setting, which is described as using no pre-trained memory while 'the validation task's own memory guides prediction.' Please clarify what memory entries are stored in each setting and how the default size of 640 was chosen.
  7. [Table 1] The row 'Others - 35911' lists 'Others' as if it were an imaging modality; please rename this to something like 'Other/aggregate' and explain which sources it aggregates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported Dice gains are empirical and do not reduce by construction to a fitted parameter or a self-citation chain.

full rationale

The paper's derivation chain is method-engineering: a temporal adapter (Eqs. 1-2), a confidence-similarity memory retrieval (Eqs. 3-5), and a confidence-driven memory replacement (Eqs. 6-7) are proposed, trained on MedBank-100k, and then evaluated on internal and external tasks by Dice. The confidence scores used to gate memory storage and retrieval are the SAM-2 predicted IoU values, which are supervised during training; selecting high-confidence stored features is an internal design choice rather than a circular reduction, because the measured Dice does not equal the selection criterion by construction. The temporal adapter is adapted from the cited MA-SAM work by partially overlapping authors, but the citation is used as inspiration for an architectural component, not as a load-bearing theorem that forces the reported improvement. The main concerns in the paper are evaluation-validity risks rather than circularity: the 9:1 split is stated to be 'at the image level' (Section 4.2), which for CT/MR volumes may not prevent the same patient or scan from contributing to both training and test, and Table 2's 'external' tasks (Liver, Spleen, Prostate, Hippocampus, BrainTumor, LiverTumor, ColonTumor, etc.) share task names with the CT/MR components of MedBank-100k without a source-disjointness table. Additionally, the catastrophic-forgetting claim in Sections 1 and 3.3 is never tested under a sequential learning protocol. These are correctness, generalization, and reporting issues, not reductions of the paper's predictions to its own inputs, so under the hard rules they do not raise the circularity score.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method relies on existing SAM-2 pretrained weights, a curated dataset, and several hand-set hyperparameters (memory size, K, confidence weighting). No new physical entities are introduced. The main unverified premise is the reliability of self-confidence as a gating signal.

free parameters (4)
  • Memory base size N = 640
    Chosen as the default; ablation shows size 640 outperforms 0 and 16, but no principled selection method is reported.
  • Number of retrieved memories K = not reported
    The selection in Eq. 3 picks K entries, but K is not stated anywhere.
  • Temporal adapter bottleneck dimension = not reported
    The down/up projection dimensions and 3D conv kernel are not specified, despite being learned parameters that affect capacity.
  • Confidence-similarity combination weights = fixed at 1 and 1
    Eq. 3 sums similarity and sigmoid confidence with equal weight; no ablation or learning of the combination is reported.
assumptions (3)
  • domain assumption SAM-2's predicted IoU confidence is a reliable proxy for mask quality.
    Used throughout the confidence-driven memory replacement (Eq. 7) and retrieval (Eq. 3), but no calibration study is provided.
  • domain assumption The 3D convolution in the temporal adapter can be applied to 2D image features without architectural changes that break the pretrained SAM-2 encoder.
    Section 3.2 introduces Conv3D without explaining how it handles 2D inputs or what padding/stride is used.
  • domain assumption Cosine similarity between unconditioned image embeddings reflects semantic relevance for segmentation.
    The retrieval (Eq. 4) and replacement (Eq. 6) rely on this, but the embedding space is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAMed-2: Selective Memory Enhanced Medical Segment Anything Model." pith.science (2026). https://pith.science/paper/JTS37AZV

@misc{pith2026250703698,
  author       = {Pith},
  title        = {Pith review of: SAMed-2: Selective Memory Enhanced Medical Segment Anything Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTS37AZV}},
  note         = {Machine review of arXiv:2507.03698}
}
read the original abstract

Recent "segment anything" efforts show promise by learning from large-scale data, but adapting such models directly to medical images remains challenging due to the complexity of medical data, noisy annotations, and continual learning requirements across diverse modalities and anatomical structures. In this work, we propose SAMed-2, a new foundation model for medical image segmentation built upon the SAM-2 architecture. Specifically, we introduce a temporal adapter into the image encoder to capture image correlations and a confidence-driven memory mechanism to store high-certainty features for later retrieval. This memory-based strategy counters the pervasive noise in large-scale medical datasets and mitigates catastrophic forgetting when encountering new tasks or modalities. To train and evaluate SAMed-2, we curate MedBank-100k, a comprehensive dataset spanning seven imaging modalities and 21 medical segmentation tasks. Our experiments on both internal benchmarks and 10 external datasets demonstrate superior performance over state-of-the-art baselines in multi-task scenarios. The code is available at: https://github.com/ZhilingYan/Medical-SAM-Bench.

Figures

Figures reproduced from arXiv: 2507.03698 by the authors.

Figure 1
Figure 1. Workflow of SAMed-2. It integrates a temporal adapter in the image [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Qualitative visualization of segmentation results [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Few-shot scaling result of our method and compared methods on ex￾ternal prostate segmentation task. 0 100 200 300 400 500 Manual SAMed-2 Assistant Manual Prompt Given SAMed-2 Inference Refinement Time (s) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 7 canonical work pages

  1. [8]

    Nature methods 18(2), 203–211 (2021)

    Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)

    2021

  2. [1]

    Nature communications13(1), 4128 (2022)

    Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., et al.: The medical segmentation decathlon. Nature communications13(1), 4128 (2022)

    2022

  3. [2]

    In: European conference on computer vision

    Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M.: Swin- unet: Unet-like pure transformer for medical image segmentation. In: European conference on computer vision. pp. 205–218. Springer (2022)

    2022

  4. [3]

    Medical Image Analysis98, 103310 (2024)

    Chen, C., Miao, J., Wu, D., Zhong, A., Yan, Z., Kim, S., Hu, J., Liu, Z., Sun, L., Li, X., et al.: Ma-sam: Modality-agnostic sam adaptation for 3d medical image segmentation. Medical Image Analysis98, 103310 (2024)

    2024

  5. [4]

    arXiv preprint arXiv:2307.01187 (2023)

    Dai, H., Ma, C., Yan, Z., Liu, Z., Shi, E., Li, Y., Shu, P., Wei, X., Zhao, L., Wu, Z., et al.: Samaug: Point prompt augmentation for segment anything model. arXiv preprint arXiv:2307.01187 (2023)

    arXiv 2023

  6. [5]

    Biomedical & Pharmacology Journal11(3), 1227 (2018)

    Goyal, B., Agrawal, S., Sohi, B.: Noise issues prevailing in various types of medical images. Biomedical & Pharmacology Journal11(3), 1227 (2018)

    work page 2018

  7. [6]

    arXiv preprint arXiv:1605.01397 (2016)

    Gutman, D., Codella, N.C., Celebi, E., Helba, B., Marchetti, M., Mishra, N., Halpern, A.: Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the inter- national skin imaging collaboration (isic). arXiv preprint arXiv:1605.01397 (2016)

    arXiv 2016

  8. [7]

    Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., et al.: Segment anything model for medical images? Medical Image Analysis 92, 103061 (2024)

    2024

Show all 24 references
  1. [9]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)

    2023

  2. [10]

    arXiv preprint arXiv:2312.17004 (2023)

    Kumari, P., Chauhan, J., Bozorgpour, A., Huang, B., Azad, R., Merhof, D.: Con- tinual learning in medical image analysis: A comprehensive review of recent ad- vancements and future prospects. arXiv preprint arXiv:2312.17004 (2023)

    2023 arXiv

  3. [11]

    In: Medical Imaging 2024: Computer-Aided Diagnosis

    Li, Y., Hu, M., Yang, X.: Polyp-sam: Transfer sam for polyp segmentation. In: Medical Imaging 2024: Computer-Aided Diagnosis. vol. 12927, pp. 759–765. SPIE (2024)

    2024

  4. [12]

    Sustainability13(3), 1224 (2021)

    Liu, X., Song, L., Liu, S., Zhang, Y.: A review of deep-learning-based medical image segmentation methods. Sustainability13(3), 1224 (2021)

    2021

  5. [13]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  6. [14]

    Nature Communications15(1), 654 (2024)

    Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)

    2024

  7. [15]

    https://www.kaggle.com/datasets/eiraoi/ thyroidultrasound (2023)

    OI,E.:ThyroidUltrasoundDataset. https://www.kaggle.com/datasets/eiraoi/ thyroidultrasound (2023)

    2023

  8. [16]

    arXiv preprint arXiv:1511.08458 (2015)

    O’shea, K., Nash, R.: An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458 (2015)

    2015 arXiv

  9. [17]

    arXiv preprint arXiv:2408.00714 (2024)

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

    2024 arXiv

  10. [18]

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Medical image computing and computer-assisted Selective Memory Enhanced Medical Segment Anything Model 11 intervention–MICCAI 2015: 18th international conference, Mun...

    2015

  11. [19]

    In: 2014 IEEE 11th international symposium on biomedical imaging (ISBI)

    Sivaswamy, J., Krishnadas, S., Joshi, G.D., Jain, M., Tabish, A.U.S.: Drishti-gs: Retinal image dataset for optic nerve head (onh) segmentation. In: 2014 IEEE 11th international symposium on biomedical imaging (ISBI). pp. 53–56. IEEE (2014)

    2014

  12. [20]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    Wang, L., Zhang, X., Su, H., Zhu, J.: A comprehensive survey of continual learn- ing: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  13. [21]

    arXiv preprint arXiv:2304.13785 (2023)

    Zhang, K., Liu, D.: Customized segment anything model for medical image seg- mentation. arXiv preprint arXiv:2304.13785 (2023)

    2023 arXiv

  14. [22]

    arXiv preprint arXiv:2305.03678 (2023)

    Zhang, Y., Jiao, R.: Towards segment anything model (sam) for medical image segmentation: a survey. arXiv preprint arXiv:2305.03678 (2023)

    2023 arXiv

  15. [23]

    arXiv preprint arXiv:2408.00874 (2024)

    Zhu, J., Qi, Y., Wu, J.: Medical sam 2: Segment medical images as video via segment anything model 2. arXiv preprint arXiv:2408.00874 (2024)

    2024 arXiv

  16. [24]

    Advances in neural information processing systems 36, 19769–19782 (2023)

    Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Wang, J., Wang, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. Advances in neural information processing systems 36, 19769–19782 (2023)

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.