REVIEW 5 major objections 5 minor 1 cited by
SAM-Guided Robust Representation Learning for One-Shot 3D Medical Image Segmentation
T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that a SAM-guided framework, RRL-MedSAM, achieves state-of-the-art one-shot 3D medical segmentation on OASIS and Lung-CT without manual prompts, using an encoder with about 3% of SAM-Base's parameters.
desk verdict Solid incremental SAM-distillation framework for one-shot 3D segmentation, but the missing warped-atlas-label control means the headline Dice gains may just be better registration. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a two-encoder mutual-supervision loop. A general lightweight encoder is trained to match SAM's image-encoder output through a reconstruction loss; a medical-specific encoder is trained with a shared encoder and task-specific decoders for joint registration and segmentation. Dual-stage knowledge distillation (DSKD) first transfers common image knowledge from SAM, then applies mutual exponential moving average (mutual-EMA) updates in both directions so the general and medical encoders teach each other. The auto-prompting (AP) segmentation decoder takes the general branch's own predicted mask as an automatic prompt, replacing the manual point, box, or mask prompts SAM normally requires. The registration loss combines smoothness, appearance similarity, and a Dice term that uses the predicted segmentation, while the segmentation losses use pseudo-labels generated by warping the atlas label.
What would settle it
Run the same architecture on a third dataset whose target organ was never seen during pre-training, for example abdominal CT with a single atlas, and check whether one-shot Dice stays near the reported 82.4–93.6% range. A separate targeted test is to replace the auto-prompting mask with a constant or atlas-derived prompt: the paper's own ablation assigns 1.4–1.6 Dice points to the prompt-guided decoder, so a much larger drop would indicate that the auto-prompting, not the mutual-supervision loop, is the main driver of the reported gains.
Extended reading notes
Core claim
The central discovery claimed is that SAM's encoder, despite being trained on natural images, carries transferable feature knowledge for 3D medical volumes, and that this knowledge can be captured by a small encoder using about 3% of SAM-Base's parameters. The framework couples two branches: the general branch learns to reproduce SAM's features, the medical branch performs joint registration and segmentation, and a mutual-EMA loop makes each branch teach the other. Pseudo-labels from the registration network (the warped atlas labels) supervise both segmentation heads, and the auto-prompting decoder feeds the general branch's coarse mask into the prompt encoder to refine the medical branch's final prediction. On the OASIS brain MRI and Lung-CT benchmarks the paper reports the highest Dice scores among compared one-shot and SAM-based methods, 82.4% on OASIS and 93.6% on Lung-CT, while using far fewer parameters than the full SAM model.
Load-bearing premise
The framework assumes that training the segmentation head on pseudo-labels produced by the registration network, while the registration loss is in turn supervised by the predicted segmentation, converges to correct anatomy instead of both branches collapsing to the single warped atlas label.
Editorial extensions
If this is right
- One-shot 3D segmentation can be performed without any manual point, box, or mask prompt at test time, since the general branch's own mask serves as the auto-prompt.
- A clinical deployment no longer needs SAM's full encoder: the distilled lightweight encoder uses about 3% of SAM-Base's parameters, reducing memory and inference cost.
- Because the segmentation Dice term feeds the registration loss, the two tasks improve together; the paper's ablations attribute gains to the dual-stage distillation, the mutual-EMA, and the auto-prompting decoder.
- The pre-trained lightweight encoder is intended as a reusable feature extractor for fine-tuning on other datasets, so the one-shot setup may transfer beyond the two evaluated benchmarks.
Reading between the lines
- An untested corollary is that the same two-encoder mutual-EMA recipe may work for other single-atlas segmentation targets such as abdominal organs or cardiac structures, since nothing in the design is brain- or lung-specific; a cross-organ experiment would be needed to confirm.
- The comparison hands SAM manual prompts derived from ground-truth masks, so the auto-prompting advantage is established against a hand-annotated oracle; testing against fully automatic SAM variants without any oracle would show how much of the gain comes from removing prompt dependence.
- If the mutual-supervision loop is the real driver, the framework could be extended from one-shot to semi-supervised settings with many unlabeled volumes, where the EMA-averaged encoders could aggregate pseudo-labels over the whole training set rather than a single atlas.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RRL-MedSAM, a one-shot 3D medical image segmentation framework that adapts the SAM encoder through dual-stage knowledge distillation, mutual exponential moving average updates, and an auto-prompting segmentation decoder. The authors report state-of-the-art Dice scores of 82.4% on OASIS and 93.6% on Lung-CT, along with registration metrics, and emphasize that their lightweight encoder uses only 3% of the parameters of SAM-Base. The central empirical claim is that this prompt-free, lightweight framework outperforms existing one-shot MIS methods as well as SAM-based approaches.
Significance. If fully validated, the framework would be a practically valuable contribution: it addresses two real limitations of SAM for medical imaging, namely the need for manual prompts and the computational cost of the large encoder, by distilling SAM knowledge into a lightweight joint registration-segmentation model. The parameter efficiency and the proposal to unify registration and segmentation knowledge are worthwhile ideas. However, the current manuscript does not yet establish the central claim because the segmentation supervision is derived from registration pseudo-labels in a loop that could collapse to copying the warped atlas label, and the paper lacks the controlling baseline needed to rule out that case.
major comments (5)
- [Section 2.4, Eqs. (4)-(6)] The segmentation heads are supervised exclusively with the registration pseudo-label ya∘φ in Eqs. (5) and (6), and the same Dice term λ3·ldice(ya∘φ, ŷu^m) appears in the registration loss of Eq. (4). This creates a positive-feedback loop whose degenerate fixed point is ŷu^m = ya∘φ, meaning the segmentation network could simply copy the warped atlas label. The paper never reports the obvious control: the test-set Dice of the warped atlas label ya∘φ alone. Without this baseline, the headline Dice improvements in Table 1 (82.4 and 93.6) could reflect only registration accuracy rather than any learned representation in the segmentation head. Please add this control and show that the proposed method improves upon the pseudo-label itself.
- [Section 3.3, Table 1 and Abstract] The abstract claims that the proposed method outperforms state-of-the-art one-shot MIS methods 'for both segmentation and registration tasks,' yet Table 1 shows that on OASIS the proposed method's NCC (0.363±0.004) is lower than SST's NCC (0.364±0.005). Since NCC is the reported registration metric, this contradicts the claim. Please either revise the claim to reflect the registration results accurately or provide additional registration metrics that support superiority in registration.
- [Section 2.2 Eq. (2) vs. Section 3.2] The EMA smoothing coefficient is set to α=0.99 in Eq. (2), citing reference [6], but Section 3.2 states that an EMA decay rate of 0.999 is applied to smooth model weight updates. These values are inconsistent, and the mutual-EMA mechanism is a central component of the proposed DSKD strategy. Please clarify the exact value used for all reported experiments and explain the discrepancy.
- [Section 3.3, Table 2] The text after Table 2 states that removing the prompt-guided strategy lowers Dice by '1.4% and 1.6%,' but the table reports 82.2 versus 80.8 (delta 1.4) and 80.1 versus 79.5 (delta 0.6). The second delta is therefore misreported as 1.6 instead of 0.6. Please correct this number and also specify whether the S-Dice and R-Dice columns refer to the OASIS or Lung-CT dataset, or to an average over both.
- [Section 2.4, Eq. (4)] The total registration loss in Eq. (4) is a weighted sum with coefficients λ1, λ2, and λ3, but the values of these weights are never reported anywhere in the manuscript. Since this loss jointly drives registration and segmentation, the experimental results are not reproducible without these values. Please report the loss weights and, ideally, a small sensitivity analysis showing that the conclusions do not hinge on a particular tuning.
minor comments (5)
- [Abstract] The abstract says experiments were conducted on 'three public datasets, i.e., OASIS, CT-lung,' but only two datasets are named and only two are used in the evaluation. Please correct the count or add the missing dataset.
- [Section 3.1] The Lung-CT data are attributed to the Learn2Reg challenge, but reference [12] is 'Self-tuning networks: Bilevel optimization of hyperparameters...', which does not appear to be the Learn2Reg dataset source. Please verify the citation for the data source.
- [Section 2.2, Eq. (1)] The notation in Eq. (1) is inconsistent: fsam(x) is first defined as fsam({xi}i=1^N), but then written as fsam without the explicit batch index. Please make the indexing consistent.
- [Section 3.1] The paper does not specify how the single annotated atlas is selected for the one-shot setting, nor whether the same atlas is used for all test subjects. Please clarify the one-shot protocol.
- [Table 2] The column headers 'S-Dice' and 'R-Dice' are not defined in the text. Please define them (presumably segmentation Dice and registration Dice) so the ablation table is self-contained.
Circularity Check
No circularity found: the mutual-supervision loop is a joint-training design, not an equivalence, and held-out Dice is measured against independent ground truth.
full rationale
The paper's training objective is self-referential in the sense that Eqs. (5)-(6) supervise both segmentation heads with the registration pseudo-label y_a∘φ and Eq. (4) includes the same Dice term in the registration loss, so segmentation and registration co-train and could in principle converge to the degenerate fixed point ŷ=y_a∘φ. However, this is a joint-training design, not a derivation: the reported test Dice in Sec. 3.3 is computed against ground-truth labels on held-out OASIS and Lung-CT test sets, so the headline result is not defined as, nor statistically forced to equal, the pseudo-label. The absence of a warped-atlas-only control is a real empirical limitation—segmentation gains could partly reflect registration quality—but that is a correctness/validation risk, not circular equivalence. The only overlapping-author citation is [4], used among three references for shared encoder-decoder structure and as a baseline; it is not load-bearing for any claim, and no 'uniqueness theorem' or ansatz is imported solely from self-citation. Internal ablations (Table 2) compare variants on held-out data. Therefore no circular step is exhibited under the required standard.
Assumptions & free parameters
free parameters (3)
- Loss weights λ1, λ2, λ3 =
Not reported
- EMA decay α =
0.99 (Eq. 2) / 0.999 (Sec. 3.2)
- Lightweight encoder architecture size =
3% of SAM-Base encoder (claimed)
assumptions (4)
- domain assumption MSE distillation from SAM features transfers useful general visual knowledge to the lightweight encoder (Eq. 1).
- domain assumption Pseudo-labels ya∘φ generated by registration are reliable enough to supervise the segmentation head (Eqs. 5, 6).
- domain assumption The mutual EMA update between the two encoders (Eqs. 2-3) converges to a stable and better representation than either alone.
- domain assumption The coarse mask from the general branch can serve as an effective auto-prompt without manual interaction.
Cite this review
Pith. "Pith review of SAM-Guided Robust Representation Learning for One-Shot 3D Medical Image Segmentation." pith.science (2026). https://pith.science/paper/JDPZB6DY
@misc{pith2026250420501,
author = {Pith},
title = {Pith review of: SAM-Guided Robust Representation Learning for One-Shot 3D Medical Image Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/JDPZB6DY}},
note = {Machine review of arXiv:2504.20501}
}
read the original abstract
One-shot medical image segmentation (MIS) is crucial for medical analysis due to the burden of medical experts on manual annotation. The recent emergence of the segment anything model (SAM) has demonstrated remarkable adaptation in MIS but cannot be directly applied to one-shot medical image segmentation (MIS) due to its reliance on labor-intensive user interactions and the high computational cost. To cope with these limitations, we propose a novel SAM-guided robust representation learning framework, named RRL-MedSAM, to adapt SAM to one-shot 3D MIS, which exploits the strong generalization capabilities of the SAM encoder to learn better feature representation. We devise a dual-stage knowledge distillation (DSKD) strategy to distill general knowledge between natural and medical images from the foundation model to train a lightweight encoder, and then adopt a mutual exponential moving average (mutual-EMA) to update the weights of the general lightweight encoder and medical-specific encoder. Specifically, pseudo labels from the registration network are used to perform mutual supervision for such two encoders. Moreover, we introduce an auto-prompting (AP) segmentation decoder which adopts the mask generated from the general lightweight model as a prompt to assist the medical-specific model in boosting the final segmentation performance. Extensive experiments conducted on three public datasets, i.e., OASIS, CT-lung demonstrate that the proposed RRL-MedSAM outperforms state-of-the-art one-shot MIS methods for both segmentation and registration tasks. Especially, our lightweight encoder uses only 3\% of the parameters compared to the encoder of SAM-Base.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 1 Pith paper
-
SAMOT: State-Aware Step Modulation and Optimal Transport Matching for Audio-Visual Instance Segmentation
SAMOT uses adaptive Mamba step modulation and optimal transport matching to set a new state-of-the-art on the AVISeg audio-visual instance segmentation benchmark.
Reference graph
Works this paper leans on
-
[6]
IEEE Transactions on Medical Imaging (2023)
He, A., Li, T., Yan, J., Wang, K., Fu, H.: Bilateral supervision network for semi- supervised medical image segmentation. IEEE Transactions on Medical Imaging (2023)
2023
-
[1]
International Journal of Computer Assisted Radiology and Surgery17(4), 699–710 (2022)
Andresen, J., Kepp, T., Ehrhardt, J., Burchard, C.v.d., Roider, J., Han- dels, H.: Deep learning-based simultaneous registration and unsupervised non- correspondence segmentation of medical images with pathologies. International Journal of Computer Assisted Radiology and Surgery17(4), 699–710 (2022)
work page 2022
-
[2]
In: Proceedings of the AAAI con- ference on artificial intelligence
Ding, Y., Yu, X., Yang, Y.: Modeling the probabilistic distribution of unlabeled data for one-shot medical image segmentation. In: Proceedings of the AAAI con- ference on artificial intelligence. vol. 35, pp. 1246–1254 (2021)
work page 2021
-
[3]
IEEE Access9, 95551–95568 (2021)
Elmahdy, M.S., Beljaards, L., Yousefi, S., Sokooti, H., Verbeek, F., Van Der Heide, U.A., Staring, M.: Joint registration and segmentation via multi-task learning for adaptive radiotherapy of prostate cancer. IEEE Access9, 95551–95568 (2021)
work page 2021
-
[4]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Fan, X., Wang, X., Gao, J., Wang, J., Luo, Z., Liu, R.: Bi-level learning of task- specific decoders for joint registration and one-shot medical image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11726–11735 (2024)
2024
-
[5]
arXiv preprint arXiv:2306.13465 (2023)
Gong, S., Zhong, Y., Ma, W., Li, J., Wang, Z., Zhang, J., Heng, P.A., Dou, Q.: 3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation. arXiv preprint arXiv:2306.13465 (2023)
arXiv 2023
-
[7]
IEEE Transactions on Neural Networks and Learning Systems35(2), 2588–2601 (2022)
He, Y., Ge, R., Qi, X., Chen, Y., Wu, J., Coatrieux, J.L., Yang, G., Li, S.: Learning betterregistrationtolearnbetterfew-shotmedicalimagesegmentation:Authentic- ity, diversity, and robustness. IEEE Transactions on Neural Networks and Learning Systems35(2), 2588–2601 (2022)
work page 2022
-
[8]
Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., et al.: Segment anything model for medical images? Medical Image Analysis92, 103061 (2024)
work page 2024
Show all 25 references
-
[9]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023)
2023
-
[10]
arXiv preprint arXiv:2309.06824 (2023)
Lin, X., Xiang, Y., Zhang, L., Yang, X., Yan, Z., Yu, L.: Samus: Adapting segment anything model for clinically-friendly and generalizable ultrasound image segmen- tation. arXiv preprint arXiv:2309.06824 (2023)
2023 arXiv
-
[11]
Nature Communications15(1), 654 (2024)
Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)
2024
-
[12]
arXiv preprint arXiv:1903.03088 (2019)
MacKay, M., Vicol, P., Lorraine, J., Duvenaud, D., Grosse, R.: Self-tuning net- works: Bilevel optimization of hyperparameters using structured best-response functions. arXiv preprint arXiv:1903.03088 (2019)
2019 arXiv
-
[13]
Journal of cognitive neuroscience22(12), 2677–2684 (2010) 10 Jia Wang, Yunan Mei, Jiarui Liu, and Xin Fan
Marcus, D.S., Fotenos, A.F., Csernansky, J.G., Morris, J.C., Buckner, R.L.: Open access series of imaging studies: longitudinal mri data in nondemented and de- mented older adults. Journal of cognitive neuroscience22(12), 2677–2684 (2010) 10 Jia Wang, Yunan Mei, Jiarui Liu, an...
2010
-
[14]
Sinclair, M., Schuh, A., Hahn, K., Petersen, K., Bai, Y., Batten, J., Schaap, M., Glocker, B.: Atlas-istn: joint segmentation, registration and atlas construction with image-and-spatialtransformernetworks.MedicalImageAnalysis78,102383(2022)
2022
-
[15]
In: 2023 IEEE/CVF International Con- ference on Computer Vision (ICCV)
Sun, Y., Wang, F., Shu, J., Wang, H., Wang, L., Meng, D., Lian, C.: Dual meta- learning with longitudinally generalized regularization for one-shot brain tissue segmentation across the human lifespan. In: 2023 IEEE/CVF International Con- ference on Computer Vision (ICCV). pp. ...
2023
-
[16]
In: Proceedings of the IEEE/CVF winter conference on applications of computer vision
Tomar, D., Bozorgtabar, B., Lortkipanidze, M., Vray, G., Rad, M.S., Thiran, J.P.: Self-supervised generative style transfer for one-shot medical image segmentation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 1998–2008 (2022)
2022
-
[17]
Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 8(5), 519–528 (2020)
Vandewinckele, L., Willems, S., Robben, D., Van Der Veen, J., Crijns, W., Nuyts, S., Maes, F.: Segmentation of head-and-neck organs-at-risk in longitudinal ct scans combining deformable registrations and convolutional neural networks. Computer Methods in Biomechanics and Biome...
2020
-
[18]
arXiv preprint arXiv:2304.12620 (2023)
Wu, J., Ji, W., Liu, Y., Fu, H., Xu, M., Xu, Y., Jin, Y.: Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620 (2023)
2023 arXiv
-
[19]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Xiong, Y., Varadarajan, B., Wu, L., Xiang, X., Xiao, F., Zhu, C., Dai, X., Wang, D., Sun, F., Iandola, F., et al.: Efficientsam: Leveraged masked image pretraining for efficient segment anything. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...
2024
-
[20]
arXiv preprint arXiv:2407.14153 (2024)
Xu, Q., Li, J., He, X., Liu, Z., Chen, Z., Duan, W., Li, C., He, M.M., Tesema, F.B., Cheah, W.P., et al.: Esp-medsam: Efficient self-prompting sam for universal domain-generalized medical image segmentation. arXiv preprint arXiv:2407.14153 (2024)
2024 arXiv
-
[21]
arXiv preprint arXiv:2306.14289 (2023)
Zhang, C., Han, D., Qiao, Y., Kim, J.U., Bae, S.H., Lee, S., Hong, C.S.: Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289 (2023)
2023 arXiv
-
[22]
arXiv preprint arXiv:2402.05008 (2024)
Zhang, Z., Cai, H., Han, S.: Efficientvit-sam: Accelerated segment anything model without performance loss. arXiv preprint arXiv:2402.05008 (2024)
2024 arXiv
-
[23]
Zhao, F., Wu, Z., Wang, L., Lin, W., Xia, S., Li, G., Consortium, U.B.C.P.: A deep network for joint registration and parcellation of cortical surfaces. In: Med- ical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, Fra...
2021
-
[24]
In: International Conference on Medical Image Computing and Computer- Assisted Intervention
Zhao, X., Shen, Z., Chen, D., Wang, S., Zhuang, Z., Wang, Q., Zhang, L.: One-shot traumatic brain segmentation with adversarial training and uncertainty rectifica- tion. In: International Conference on Medical Image Computing and Computer- Assisted Intervention. pp. 120–129. S...
2023
-
[25]
arXiv preprint arXiv:2312.06660 (2023)
Zhou, C., Li, X., Loy, C.C., Dai, B.: Edgesam: Prompt-in-the-loop distillation for on-device deployment of sam. arXiv preprint arXiv:2312.06660 (2023)
2023 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.