Pith. sign in

REVIEW 3 major objections 4 minor 19 references

MAIS: Memory-Attention for Interactive Segmentation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read MAIS shows that remembering past clicks and masks lets interactive segmentation keep improving where memoryless models plateau.

desk verdict A useful memory-attention twist on SAM-Med3D with real gains on three datasets, but the paper's 'consistently outperform' claim is contradicted by its own ACDC numbers and no significance testing backs it. read the letter →

arxiv 2505.07511 v1 pith:HUIPNBXH submitted 2025-05-12 cs.CV cs.AIcs.LG

classification cs.CVcs.AIcs.LG
keywords interactivesegmentationmemoryattentionSAM-Med3Dfoundationmodelsmedicalimage3Dlow-datafine-tuninguserclicks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the plateau in interactive medical segmentation comes from missing memory: models such as SAM-Med3D treat every user click as a fresh event, discarding the history of corrections. To fix this, the authors propose a memory-attention mechanism (MAIS) that stores click embeddings and previous mask embeddings in a FIFO bank and conditions the current image embedding on that bank before mask decoding. Across CT and MRI datasets covering head-and-neck, heart, and abdomen, they report that MAIS consistently outperforms fine-tuned SAM-Med3D, with the gap widening as the number of clicks grows, and that it approaches or exceeds a nn-UNet oracle even when trained on 10% of the data. If this is right, interactive annotation with foundation models becomes a genuinely incremental process: each correction compounds earlier ones instead of being wasted.

What carries the argument

The load-bearing object is the memory-attention block paired with a FIFO memory bank of the latest $N=60$ interaction embeddings, split into sparse click embeddings and dense mask embeddings. The block performs self-attention over the two memory stacks with a convolutional transformer, adds the self-attended dense output to the current image embedding, and then runs cross-attention between the sparse and dense memories; only after this conditioning does the mask decoder see the embedding. The design omits the separate memory encoder used in SAM2, which the paper reports keeps computational overhead low while preserving the refinement gains. This mechanism is what lets each new click draw on all earlier corrections.

What would settle it

Run the same four-dataset protocol with clicks generated by a boundary-oriented or clinician-in-the-loop strategy instead of random samples from error regions; if MAIS's advantage over Ft-SAM3D disappears or narrows sharply, the claim that memory drives refinement in real workflows is falsified. A second test: replace the stored past interactions with random but equally sized memory stacks; if Dice gains persist, the improvement comes from extra capacity rather than from remembering what the user corrected.

Watch

Extended reading notes

Core claim

The central claim is that temporal context, not extra parameters or more data, is what makes repeated user corrections pay off. MAIS keeps the SAM-Med3D image encoder, prompt encoders, and mask decoder, and inserts a memory bank holding the latest $N=60$ sparse (click) and dense (mask) embeddings. A memory-attention block self-attends over the memory stacks, adds the resulting dense output to the image embedding, then cross-attends between the sparse and dense memories before the decoder produces the mask. In the paper's experiments this configuration beats the same architecture with the memory module removed on all four datasets, continues to improve through 50 to 150 clicks where the memoryless baseline flattens around 10, and reaches Dice scores comparable to an nn-UNet oracle trained on the full target data even when MAIS is fine-tuned on a tenth of it.

Load-bearing premise

The load-bearing assumption is that simulated clicks sampled from regions where the previous prediction was wrong behave like real human corrections; if real users click less often, on boundaries, or in more informative locations, the measured refinement gains may shrink.

Editorial extensions

If this is right

  • Interactive refinement no longer plateaus: MAIS keeps gaining up to 50–150 clicks, while the memoryless baseline saturates near 10 clicks.
  • Task-specific accuracy can be reached with dramatically less labeled data: with 10% of training data MAIS lands near or above oracle nn-UNet on several datasets.
  • Combining sparse and dense memory is essential; click-only memory underperforms and mask-only memory is weaker than the fused bank.
  • The memory module is cheap enough to retrofit: 2.84M parameters and roughly 6 seconds extra inference per image in the heaviest configuration.
  • The benefit transfers across CT and MRI and across head-neck, abdomen, and heart tasks, suggesting the mechanism is not dataset-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run is to apply the same memory bank to 2D SAM and to SAM2-style video object segmentation, since the mechanism is defined on embeddings and is backbone-agnostic; the paper only demonstrates it on SAM-Med3D.
  • The FIFO bank implies a session-boundary test: if memory persists across separate annotation sessions, clinicians could resume work where they left off; the paper evaluates only within a single simulated session.
  • Because gains appear largest in low-data regimes, a testable corollary is that memory attention acts partly as a regularizer or as a curriculum over the model's own mistakes; this could be checked by ablating memory during training versus inference only.
  • The simulated-click assumption is the main gap between the reported numbers and clinical use; a user study with real radiologists' clicks would be the direct validity test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MAIS, a memory-attention mechanism for interactive 3D medical image segmentation built on top of SAM-Med3D. A FIFO memory bank stores sparse click embeddings and dense previous-mask embeddings, and a memory attention block conditions the image embedding on these stored memories before the mask decoder produces the next segmentation. The authors fine-tune only the prompt encoder, mask decoder, and memory module on HaN-SEG, ACDC, AMOS-CT, and AMOS-MR, and compare MAIS against fine-tuned SAM-Med3D (Ft-SAM3D), zero-shot SAM3D, and oracle nn-UNet under one-shot and partial-data settings. They claim that MAIS consistently outperforms Ft-SAM3D, especially with more interactions and in low-data regimes, and that it approaches oracle performance even with 10% training data. The paper also reports ablation studies on memory bank size, sparse versus dense memory, and computational costs.

Significance. If the central claim holds, the paper would make a useful contribution: it introduces a lightweight, modular way to add temporal context to interactive segmentation, with clear potential to reduce redundant corrections in clinical annotation workflows. The strength of the paper is its focus on a practical setting (fine-tuning a foundation model on small target datasets) and its explicit accounting of parameter counts, GPU memory, and training and inference time. The reported gains on HaN-SEG and AMOS are encouraging. However, the paper's own Table 1 does not support the 'consistently outperform' claim, and the absence of error bars or significance tests makes the performance differences difficult to trust. The core idea is plausible and worth publishing, but the evidence as presented needs revision.

major comments (3)
  1. [Section 3.2, Table 1] The statement that MAIS 'consistently outperform[s] Ft-SAM3D fine-tuning' is not supported by the data in Table 1. On ACDC in the one-shot setting, MAIS is below Ft-SAM3D at the 1-, 10-, 20-, and 50-click rows (61.38 vs. 62.65, 70.46 vs. 72.61, 71.85 vs. 73.49, and 74.09 vs. 74.66), and it is also lower at the first click in the 10%, 50%, and 70% data columns. MAIS exceeds Ft-SAM3D at 150 clicks in the one-shot case, but that value (76.54) is marked as underperforming the zero-shot SAM3D baseline (77.75). The 'consistently' claim therefore needs to be narrowed, or supported by a dataset-by-dataset analysis.
  2. [Section 3.2 and all experimental tables] No error bars, confidence intervals, or repeated-seed statistics are reported anywhere in the paper. Given that the text uses the word 'significantly outperforms' to describe the comparison with Ft-SAM3D, the authors should either add statistical testing (e.g., paired tests across test subjects or multiple seeds) or remove the significance claim and present the numbers as point estimates.
  3. [Section 2.3 and Conclusions] The user-interaction simulation samples clicks from regions where previous predictions were incorrect, but the paper does not validate this against real human correction behavior. Since the conclusions are framed in terms of clinical workflow benefits, the transfer of the measured gains to actual annotation sessions is untested. This limitation should be stated explicitly, and the clinical claims should be tempered accordingly.
minor comments (4)
  1. [Keywords] The keyword 'Fundation Models' should be spelled 'Foundation Models'.
  2. [Section 2.1.2] There is a missing space in 'vanillaSAM-Med3D' and a missing space after the colon in 'Sparse Memory: this memory bank'; these should be corrected.
  3. [Section 2.3] The sentence beginning 'we simulate sparse visual prompts...' should start with a capital letter, and the simulation details (how many clicks per iteration, how incorrect regions are sampled, and how positive versus negative clicks are assigned) should be described precisely enough for reproduction.
  4. [Table 1] The table formatting is dense and contains missing entries (dashes). The caption should also explain the asterisk notation more clearly, since the current text says 'underperforming SAM3D' but the reader must infer the comparison row by row.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: MAIS is an empirical comparison against matched and external baselines, not a result derived from fitted inputs.

full rationale

MAIS is an empirical contribution. The memory-attention module is added to the SAM-Med3D backbone, and the paper reports Dice scores against a matched no-memory baseline (Ft-SAM3D), the zero-shot backbone (SAM3D), and an external nn-UNet oracle on four public datasets. No prediction in the paper is obtained by algebraic reduction from a fitted quantity: the memory bank size (N=60) and the sparse-plus-dense configuration are selected by ablations on HaN-Seg, and the oracle is an external upper-bound baseline, not a fitted input to MAIS. The only author-adjacent citation is Diaz-Pinto et al. 2024 (MONAI Label), used in the introduction as background on interactive segmentation; it is not load-bearing for any result. The simulated-click protocol is shared by all compared methods, so it does not make the MAIS-versus-Ft-SAM3D gap circular. The paper's own Table 1 shows MAIS below Ft-SAM3D on ACDC at most interaction counts, and no significance tests are reported; this weakens the strength of the 'consistently outperform' claim, but that is an evidentiary and statistical issue, not a circularity of derivation. Therefore no circular step is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical entities or theoretical constructs are introduced; the memory bank is a standard FIFO queue of embeddings. The main fitted quantity is the memory size, and the key assumptions concern the backbone, the click simulation, and the effectiveness of the attention design.

free parameters (1)
  • memory bank size N = 60
    Selected from experiments on HaN-Seg 'Cavity Oral' in Section 3.1, where larger sizes generally improve accuracy; N=60 is used for all main experiments.
assumptions (3)
  • domain assumption SAM-Med3D provides a valid frozen visual encoder whose features remain useful after fine-tuning only the prompt encoder, mask decoder, and memory module.
    The paper freezes the image encoder to avoid large compute (Section 2.3 and Appendix B), so the central claim depends on SAM-Med3D features being adequate for new modalities.
  • domain assumption Simulated clicks sampled from error regions are a faithful proxy for human interaction in interactive segmentation.
    Section 2.3 states this simulation 'mirrors human correction behavior'; if real user clicks differ, the measured gains may not transfer.
  • ad hoc to paper The memory attention module described in Section 2.1.3 (self-attention on memory stacks plus cross-attention with the image embedding) is a sufficient mechanism for temporal context, with no formal guarantee or comparison to simpler aggregation methods.
    The architecture is proposed without a theoretical or empirical comparison to alternative memory designs such as concatenating all past clicks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MAIS: Memory-Attention for Interactive Segmentation." pith.science (2026). https://pith.science/paper/HUIPNBXH

@misc{pith2026250507511,
  author       = {Pith},
  title        = {Pith review of: MAIS: Memory-Attention for Interactive Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HUIPNBXH}},
  note         = {Machine review of arXiv:2505.07511}
}
read the original abstract

Interactive medical segmentation reduces annotation effort by refining predictions through user feedback. Vision Transformer (ViT)-based models, such as the Segment Anything Model (SAM), achieve state-of-the-art performance using user clicks and prior masks as prompts. However, existing methods treat interactions as independent events, leading to redundant corrections and limited refinement gains. We address this by introducing MAIS, a Memory-Attention mechanism for Interactive Segmentation that stores past user inputs and segmentation states, enabling temporal context integration. Our approach enhances ViT-based segmentation across diverse imaging modalities, achieving more efficient and accurate refinements.

Figures

Figures reproduced from arXiv: 2505.07511 by the authors.

Figure 1
Figure 1. Segmentation accuracy (Dice %) vs. user inter [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed segmentation model incorporating memory attention: A [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Memory bank and prompt type analysis : Subplot [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative result on AMOS-MR dataset For [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Qualitative result on the HaN-SEG dataset (CT images) for [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 7 canonical work pages

  1. [1]

    Olivier Bernard and Pierre-Marc Lalande, ... Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Transactions on Medical Imaging, 37 0 (11): 0 2514--2525, 2018. doi:10.1109/TMI.2018.2837502

  2. [2]

    Sam3d: Segment anything model in volumetric medical images, 2024

    Nhat-Tan Bui, Dinh-Hieu Hoang, Minh-Triet Tran, Gianfranco Doretto, Donald Adjeroh, Brijesh Patel, Arabinda Choudhary, and Ngan Le. Sam3d: Segment anything model in volumetric medical images, 2024. URL https://arxiv.org/abs/2309.03493

  3. [3]

    Sam on medical images: A comprehensive study on three prompt modes

    Dongjie Cheng, Ziyuan Qin, Zekun Jiang, Shaoting Zhang, Qicheng Lao, and Kang Li. Sam on medical images: A comprehensive study on three prompt modes. arXiv preprint arXiv:2305.00035, 2023

  4. [4]

    Monai label: A framework for ai-assisted interactive labeling of 3d medical images

    Andres Diaz-Pinto, Sachidanand Alle, Vishwesh Nath, Yucheng Tang, Alvin Ihsani, Muhammad Asad, Fernando P \'e rez-Garc \' a, Pritesh Mehta, Wenqi Li, Mona Flores, et al. Monai label: A framework for ai-assisted interactive labeling of 3d medical images. Medical Image Analysis, 95: 0 103207, 2024

  5. [5]

    Segvol: Universal and interactive volumetric medical image segmentation

    Yuxin Du, Fan Bai, Tiejun Huang, and Bo Zhao. Segvol: Universal and interactive volumetric medical image segmentation. arXiv preprint arXiv:2311.13385, 2023

  6. [6]

    3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation

    Shizhan Gong, Yuan Zhong, Wenao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Ann Heng, and Qi Dou. 3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation. arXiv preprint arXiv:2306.13465, 2023

  7. [7]

    Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images

    Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI brainlesion workshop, pages 272--284. Springer, 2021

  8. [8]

    Unetr: Transformers for 3d medical image segmentation

    Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574--584, 2022

Show all 19 references
  1. [9]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmentation

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18 0 (2): 0 203--211, 2021

  2. [10]

    Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation

    Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. Advances in neural information processing systems, 35: 0...

  3. [11]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015--4026, 2023

  4. [12]

    Promise: Prompt-driven 3d medical image segmentation using pretrained image foundation models

    Hao Li, Han Liu, Dewei Hu, Jiacheng Wang, and Ipek Oguz. Promise: Prompt-driven 3d medical image segmentation using pretrained image foundation models. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pages 1--5. IEEE, 2024

  5. [13]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15 0 (1): 0 654, 2024

  6. [14]

    Segment anything model for medical image analysis: an experimental study

    Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 89: 0 102918, 2023

  7. [15]

    Han-seg: The head and neck organ-at-risk ct and mr segmentation challenge

    Ga s per Podobnik, Bulat Ibragimov, Elias Tappeiner, Chanwoong Lee, Jin Sung Kim, Zacharia Mesbah, Romain Modzelewski, Yihao Ma, Fan Yang, Miko aj Rudecki, et al. Han-seg: The head and neck organ-at-risk ct and mr segmentation challenge. Radiotherapy and Oncology, 198: 0 110410, 2024

  8. [16]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R \"a dle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

  9. [17]

    Sam-clip: Merging vision foundation models towards semantic and spatial understanding

    Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. In Proceedings of the...

  10. [18]

    Sam-med3d: Towards general-purpose segmentation models for volumetric medical images, 2024 b

    Haoyu Wang, Sizheng Guo, Jin Ye, Zhongying Deng, Junlong Cheng, Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, Bin Fu, Shaoting Zhang, Junjun He, and Yu Qiao. Sam-med3d: Towards general-purpose segmentation models for volumetric medical images, 2024 b . URL ht...

  11. [19]

    Totalsegmentator: robust segmentation of 104 anatomic structures in ct images

    Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. Totalsegmentator: robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intell...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.