REVIEW 3 major objections 4 minor 19 references
MAIS: Memory-Attention for Interactive Segmentation
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read MAIS shows that remembering past clicks and masks lets interactive segmentation keep improving where memoryless models plateau.
desk verdict A useful memory-attention twist on SAM-Med3D with real gains on three datasets, but the paper's 'consistently outperform' claim is contradicted by its own ACDC numbers and no significance testing backs it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the memory-attention block paired with a FIFO memory bank of the latest $N=60$ interaction embeddings, split into sparse click embeddings and dense mask embeddings. The block performs self-attention over the two memory stacks with a convolutional transformer, adds the self-attended dense output to the current image embedding, and then runs cross-attention between the sparse and dense memories; only after this conditioning does the mask decoder see the embedding. The design omits the separate memory encoder used in SAM2, which the paper reports keeps computational overhead low while preserving the refinement gains. This mechanism is what lets each new click draw on all earlier corrections.
What would settle it
Run the same four-dataset protocol with clicks generated by a boundary-oriented or clinician-in-the-loop strategy instead of random samples from error regions; if MAIS's advantage over Ft-SAM3D disappears or narrows sharply, the claim that memory drives refinement in real workflows is falsified. A second test: replace the stored past interactions with random but equally sized memory stacks; if Dice gains persist, the improvement comes from extra capacity rather than from remembering what the user corrected.
Extended reading notes
Core claim
The central claim is that temporal context, not extra parameters or more data, is what makes repeated user corrections pay off. MAIS keeps the SAM-Med3D image encoder, prompt encoders, and mask decoder, and inserts a memory bank holding the latest $N=60$ sparse (click) and dense (mask) embeddings. A memory-attention block self-attends over the memory stacks, adds the resulting dense output to the image embedding, then cross-attends between the sparse and dense memories before the decoder produces the mask. In the paper's experiments this configuration beats the same architecture with the memory module removed on all four datasets, continues to improve through 50 to 150 clicks where the memoryless baseline flattens around 10, and reaches Dice scores comparable to an nn-UNet oracle trained on the full target data even when MAIS is fine-tuned on a tenth of it.
Load-bearing premise
The load-bearing assumption is that simulated clicks sampled from regions where the previous prediction was wrong behave like real human corrections; if real users click less often, on boundaries, or in more informative locations, the measured refinement gains may shrink.
Editorial extensions
If this is right
- Interactive refinement no longer plateaus: MAIS keeps gaining up to 50–150 clicks, while the memoryless baseline saturates near 10 clicks.
- Task-specific accuracy can be reached with dramatically less labeled data: with 10% of training data MAIS lands near or above oracle nn-UNet on several datasets.
- Combining sparse and dense memory is essential; click-only memory underperforms and mask-only memory is weaker than the fused bank.
- The memory module is cheap enough to retrofit: 2.84M parameters and roughly 6 seconds extra inference per image in the heaviest configuration.
- The benefit transfers across CT and MRI and across head-neck, abdomen, and heart tasks, suggesting the mechanism is not dataset-specific.
Reading between the lines
- A natural extension the paper does not run is to apply the same memory bank to 2D SAM and to SAM2-style video object segmentation, since the mechanism is defined on embeddings and is backbone-agnostic; the paper only demonstrates it on SAM-Med3D.
- The FIFO bank implies a session-boundary test: if memory persists across separate annotation sessions, clinicians could resume work where they left off; the paper evaluates only within a single simulated session.
- Because gains appear largest in low-data regimes, a testable corollary is that memory attention acts partly as a regularizer or as a curriculum over the model's own mistakes; this could be checked by ablating memory during training versus inference only.
- The simulated-click assumption is the main gap between the reported numbers and clinical use; a user study with real radiologists' clicks would be the direct validity test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MAIS, a memory-attention mechanism for interactive 3D medical image segmentation built on top of SAM-Med3D. A FIFO memory bank stores sparse click embeddings and dense previous-mask embeddings, and a memory attention block conditions the image embedding on these stored memories before the mask decoder produces the next segmentation. The authors fine-tune only the prompt encoder, mask decoder, and memory module on HaN-SEG, ACDC, AMOS-CT, and AMOS-MR, and compare MAIS against fine-tuned SAM-Med3D (Ft-SAM3D), zero-shot SAM3D, and oracle nn-UNet under one-shot and partial-data settings. They claim that MAIS consistently outperforms Ft-SAM3D, especially with more interactions and in low-data regimes, and that it approaches oracle performance even with 10% training data. The paper also reports ablation studies on memory bank size, sparse versus dense memory, and computational costs.
Significance. If the central claim holds, the paper would make a useful contribution: it introduces a lightweight, modular way to add temporal context to interactive segmentation, with clear potential to reduce redundant corrections in clinical annotation workflows. The strength of the paper is its focus on a practical setting (fine-tuning a foundation model on small target datasets) and its explicit accounting of parameter counts, GPU memory, and training and inference time. The reported gains on HaN-SEG and AMOS are encouraging. However, the paper's own Table 1 does not support the 'consistently outperform' claim, and the absence of error bars or significance tests makes the performance differences difficult to trust. The core idea is plausible and worth publishing, but the evidence as presented needs revision.
major comments (3)
- [Section 3.2, Table 1] The statement that MAIS 'consistently outperform[s] Ft-SAM3D fine-tuning' is not supported by the data in Table 1. On ACDC in the one-shot setting, MAIS is below Ft-SAM3D at the 1-, 10-, 20-, and 50-click rows (61.38 vs. 62.65, 70.46 vs. 72.61, 71.85 vs. 73.49, and 74.09 vs. 74.66), and it is also lower at the first click in the 10%, 50%, and 70% data columns. MAIS exceeds Ft-SAM3D at 150 clicks in the one-shot case, but that value (76.54) is marked as underperforming the zero-shot SAM3D baseline (77.75). The 'consistently' claim therefore needs to be narrowed, or supported by a dataset-by-dataset analysis.
- [Section 3.2 and all experimental tables] No error bars, confidence intervals, or repeated-seed statistics are reported anywhere in the paper. Given that the text uses the word 'significantly outperforms' to describe the comparison with Ft-SAM3D, the authors should either add statistical testing (e.g., paired tests across test subjects or multiple seeds) or remove the significance claim and present the numbers as point estimates.
- [Section 2.3 and Conclusions] The user-interaction simulation samples clicks from regions where previous predictions were incorrect, but the paper does not validate this against real human correction behavior. Since the conclusions are framed in terms of clinical workflow benefits, the transfer of the measured gains to actual annotation sessions is untested. This limitation should be stated explicitly, and the clinical claims should be tempered accordingly.
minor comments (4)
- [Keywords] The keyword 'Fundation Models' should be spelled 'Foundation Models'.
- [Section 2.1.2] There is a missing space in 'vanillaSAM-Med3D' and a missing space after the colon in 'Sparse Memory: this memory bank'; these should be corrected.
- [Section 2.3] The sentence beginning 'we simulate sparse visual prompts...' should start with a capital letter, and the simulation details (how many clicks per iteration, how incorrect regions are sampled, and how positive versus negative clicks are assigned) should be described precisely enough for reproduction.
- [Table 1] The table formatting is dense and contains missing entries (dashes). The caption should also explain the asterisk notation more clearly, since the current text says 'underperforming SAM3D' but the reader must infer the comparison row by row.
Circularity Check
No significant circularity: MAIS is an empirical comparison against matched and external baselines, not a result derived from fitted inputs.
full rationale
MAIS is an empirical contribution. The memory-attention module is added to the SAM-Med3D backbone, and the paper reports Dice scores against a matched no-memory baseline (Ft-SAM3D), the zero-shot backbone (SAM3D), and an external nn-UNet oracle on four public datasets. No prediction in the paper is obtained by algebraic reduction from a fitted quantity: the memory bank size (N=60) and the sparse-plus-dense configuration are selected by ablations on HaN-Seg, and the oracle is an external upper-bound baseline, not a fitted input to MAIS. The only author-adjacent citation is Diaz-Pinto et al. 2024 (MONAI Label), used in the introduction as background on interactive segmentation; it is not load-bearing for any result. The simulated-click protocol is shared by all compared methods, so it does not make the MAIS-versus-Ft-SAM3D gap circular. The paper's own Table 1 shows MAIS below Ft-SAM3D on ACDC at most interaction counts, and no significance tests are reported; this weakens the strength of the 'consistently outperform' claim, but that is an evidentiary and statistical issue, not a circularity of derivation. Therefore no circular step is present.
Assumptions & free parameters
free parameters (1)
- memory bank size N =
60
assumptions (3)
- domain assumption SAM-Med3D provides a valid frozen visual encoder whose features remain useful after fine-tuning only the prompt encoder, mask decoder, and memory module.
- domain assumption Simulated clicks sampled from error regions are a faithful proxy for human interaction in interactive segmentation.
- ad hoc to paper The memory attention module described in Section 2.1.3 (self-attention on memory stacks plus cross-attention with the image embedding) is a sufficient mechanism for temporal context, with no formal guarantee or comparison to simpler aggregation methods.
Cite this review
Pith. "Pith review of MAIS: Memory-Attention for Interactive Segmentation." pith.science (2026). https://pith.science/paper/HUIPNBXH
@misc{pith2026250507511,
author = {Pith},
title = {Pith review of: MAIS: Memory-Attention for Interactive Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/HUIPNBXH}},
note = {Machine review of arXiv:2505.07511}
}
read the original abstract
Interactive medical segmentation reduces annotation effort by refining predictions through user feedback. Vision Transformer (ViT)-based models, such as the Segment Anything Model (SAM), achieve state-of-the-art performance using user clicks and prior masks as prompts. However, existing methods treat interactions as independent events, leading to redundant corrections and limited refinement gains. We address this by introducing MAIS, a Memory-Attention mechanism for Interactive Segmentation that stores past user inputs and segmentation states, enabling temporal context integration. Our approach enhances ViT-based segmentation across diverse imaging modalities, achieving more efficient and accurate refinements.
Figures
Reference graph
Works this paper leans on
-
[1]
Olivier Bernard and Pierre-Marc Lalande, ... Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Transactions on Medical Imaging, 37 0 (11): 0 2514--2525, 2018. doi:10.1109/TMI.2018.2837502
arXiv 2018
-
[2]
Sam3d: Segment anything model in volumetric medical images, 2024
Nhat-Tan Bui, Dinh-Hieu Hoang, Minh-Triet Tran, Gianfranco Doretto, Donald Adjeroh, Brijesh Patel, Arabinda Choudhary, and Ngan Le. Sam3d: Segment anything model in volumetric medical images, 2024. URL https://arxiv.org/abs/2309.03493
arXiv 2024
-
[3]
Sam on medical images: A comprehensive study on three prompt modes
Dongjie Cheng, Ziyuan Qin, Zekun Jiang, Shaoting Zhang, Qicheng Lao, and Kang Li. Sam on medical images: A comprehensive study on three prompt modes. arXiv preprint arXiv:2305.00035, 2023
arXiv 2023
-
[4]
Monai label: A framework for ai-assisted interactive labeling of 3d medical images
Andres Diaz-Pinto, Sachidanand Alle, Vishwesh Nath, Yucheng Tang, Alvin Ihsani, Muhammad Asad, Fernando P \'e rez-Garc \' a, Pritesh Mehta, Wenqi Li, Mona Flores, et al. Monai label: A framework for ai-assisted interactive labeling of 3d medical images. Medical Image Analysis, 95: 0 103207, 2024
work page 2024
-
[5]
Segvol: Universal and interactive volumetric medical image segmentation
Yuxin Du, Fan Bai, Tiejun Huang, and Bo Zhao. Segvol: Universal and interactive volumetric medical image segmentation. arXiv preprint arXiv:2311.13385, 2023
arXiv 2023
-
[6]
3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation
Shizhan Gong, Yuan Zhong, Wenao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Ann Heng, and Qi Dou. 3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation. arXiv preprint arXiv:2306.13465, 2023
arXiv 2023
-
[7]
Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images
Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI brainlesion workshop, pages 272--284. Springer, 2021
work page 2021
-
[8]
Unetr: Transformers for 3d medical image segmentation
Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574--584, 2022
2022
Show all 19 references
-
[9]
nnu-net: a self-configuring method for deep learning-based biomedical image segmentation
Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18 0 (2): 0 203--211, 2021
2021
-
[10]
Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation
Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. Advances in neural information processing systems, 35: 0...
2022
-
[11]
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015--4026, 2023
2023
-
[12]
Promise: Prompt-driven 3d medical image segmentation using pretrained image foundation models
Hao Li, Han Liu, Dewei Hu, Jiacheng Wang, and Ipek Oguz. Promise: Prompt-driven 3d medical image segmentation using pretrained image foundation models. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pages 1--5. IEEE, 2024
2024
-
[13]
Segment anything in medical images
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15 0 (1): 0 654, 2024
2024
-
[14]
Segment anything model for medical image analysis: an experimental study
Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 89: 0 102918, 2023
2023
-
[15]
Han-seg: The head and neck organ-at-risk ct and mr segmentation challenge
Ga s per Podobnik, Bulat Ibragimov, Elias Tappeiner, Chanwoong Lee, Jin Sung Kim, Zacharia Mesbah, Romain Modzelewski, Yihao Ma, Fan Yang, Miko aj Rudecki, et al. Han-seg: The head and neck organ-at-risk ct and mr segmentation challenge. Radiotherapy and Oncology, 198: 0 110410, 2024
2024
-
[16]
Sam 2: Segment anything in images and videos
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R \"a dle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024
2024 arXiv
-
[17]
Sam-clip: Merging vision foundation models towards semantic and spatial understanding
Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. In Proceedings of the...
2024
-
[18]
Sam-med3d: Towards general-purpose segmentation models for volumetric medical images, 2024 b
Haoyu Wang, Sizheng Guo, Jin Ye, Zhongying Deng, Junlong Cheng, Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, Bin Fu, Shaoting Zhang, Junjun He, and Yu Qiao. Sam-med3d: Towards general-purpose segmentation models for volumetric medical images, 2024 b . URL ht...
2024 arXiv
-
[19]
Totalsegmentator: robust segmentation of 104 anatomic structures in ct images
Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. Totalsegmentator: robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intell...
2023
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.