REVIEW 4 major objections 4 minor 31 references
Register Anything: Estimating "Corresponding Prompts" for Segment Anything Model
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read PromptReg claims that image registration can be reduced to finding corresponding point prompts for a frozen Segment Anything Model, and reports results that beat trained baselines.
desk verdict Genuinely new registration framing via SAM prompts, but the central inverse-prompt derivation is dimensionally ill-posed and the auxiliary boundary-fitting loop carries much of the reported performance. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the inverse-prompt identity of Eq. (1), which claims to invert the SAM segmentation function through first-order Taylor expansion: the prototype $G_k$ is linearized as $f_0 + J \cdot Z_y^k$, the similarity volume $S'_y$ is linearized as $(f_{\mathrm{sim}})_0 + J_{\mathrm{sim}} \cdot G_k$, and the two Jacobians are composed and inverted to map a spatial similarity map back to a single point prompt. Around it, the method adds an auxiliary-prompt loop (Eq. (2)) that iteratively places positive or negative points at maximum Hausdorff-distance discrepancies to fit the target ROI boundary, and a marginalization step that applies random spatial transforms to the source image and prompt, inverts each one, and averages the resulting ROI probability maps. The inversion is meant to be class-consistent: both prompts yield ROIs of the same class $C_k$.
What would settle it
On a real SAM, compute $J$ and $J_{\mathrm{sim}}$ by finite differences around several source prompts and check their numerical rank: if either Jacobian is singular or $J_{\mathrm{sim}}$ is not square and full rank, Eq. (1) has no unique solution. Alternatively, rerun the MR-Prostate experiment with the auxiliary-prompt loop disabled (setting $\sigma = \infty$); the paper's own Table 3 reports Dice falling from 77.76 to 70.12, so if this gap persists across datasets, the auxiliary fitting, not the inverse-prompt formula, is the mechanism and the advertised 'prediction' is actually a fitted prompt.
Extended reading notes
Core claim
The paper's central claim is that the 'corresponding prompt problem' is solvable in closed form: given a prompt $Z_x^k$ in image $I_x$, the corresponding prompt $Z_y^k$ in $I_y$ follows from Eq. (1), $Z_y^k = (J_{\mathrm{sim}})^{-1} J^{-1} S'_y{}^k + \rho$, where $S'_y$ is the similarity map between the prototype of the ROI segmented from $I_x$ and features of $I_y$, and $J$ and $J_{\mathrm{sim}}$ are first-order derivatives of the segmentation and similarity functions. The same-class region in the target image is then obtained by one SAM call conditioned on this inverted prompt, with optional auxiliary boundary prompts added when the primary prompt misaligns; multiple prompts and spatial augmentations are marginalized to stabilise the result. On five datasets (prostate MR, abdomen MR, lung CT, histopathology, aerial images), the method reports higher Dice and lower target registration error than intensity-based iterative algorithms and unsupervised DDF-predicting networks, and results competitive with weakly-supervised registration trained on full segmentations.
Load-bearing premise
The method assumes the first-order Taylor expansion in Eq. (1) is invertible, so that a full spatial similarity map can be uniquely turned into a single point prompt; for a nonlinear mask decoder with far more similarity-map entries than the 3 prompt coordinates, that inversion is not guaranteed to exist.
Editorial extensions
If this is right
- Registration no longer needs a network trained for each modality or anatomy: the same frozen SAM handles prostate MR, abdomen MR, lung CT, histology, and aerial imagery from arbitrary point prompts.
- Because the prompt is a controllable input, a clinician can steer registration toward a specific ROI class simply by placing a prompt inside it, and prompt location inside a ROI measurably improves Dice and TRE.
- Increasing the number of randomly sampled source prompts improves alignment with diminishing returns, so dense ROI correspondences can be traded off against compute.
- PromptReg's performance with four random prompts rivals weakly-supervised registration trained on full segmentations, suggesting that a large part of weakly-supervised registration's signal is captured by prompt-conditioned segmentation alone.
Reading between the lines
- If Eq. (1)'s Jacobian inversion is rank-deficient in practice, the auxiliary prompt loop in Eq. (2) may be doing most of the correspondence work; a clean ablation that replaces the inverted primary prompt with a random interior point before running the auxiliary loop would reveal which mechanism actually carries the Dice gains.
- The 'corresponding prompt' framing could transfer beyond SAM to other promptable models (e.g., text- or box-prompted segmenters), turning prompt inversion into a generic bridge between segmentation foundation models and registration.
- The probabilistic marginalization suggests a testable extension: instead of averaging inverse-transformed ROIs, one could fuse the distribution of prompt-conditioned masks, which may improve uncertainty estimates for clinical registration.
- The method's success on aerial images hints at non-medical applications, since SAM itself is strongest on natural images, where the inversion may be even more reliable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PromptReg, a training-free registration method built on SAM. It frames registration as finding a prompt Z_y in the target image such that the SAM segmentation conditioned on Z_y corresponds to the ROI segmented from a given prompt in the source image. The method derives a primary prompt through Eq. (1), an inverse-prompt formula based on first-order Taylor expansions, and adds auxiliary prompts via an iterative Hausdorff-distance-based boundary-correction loop (Eq. (2)). Prompt marginalization over multiple transformations and prompts is used for robustness. Experiments on three 3D medical datasets and two 2D datasets report Dice and TRE, claiming improvements over iterative and learning-based baselines and competitiveness with weakly-supervised LabelReg.
Significance. If the primary inverse-prompt mechanism were sound, the paper would open a genuinely new direction: using a frozen foundation model for training-free ROI-based registration, with a broad evaluation across five datasets and several SAM variants. The empirical study is wide-ranging, and the ablations on prompt location, prompt count, and auxiliary threshold are useful. However, the central derivation in Eq. (1) is invalid as written, the auxiliary correction loop appears to be the main driver of the reported gains, and the evaluation metric is closely aligned with the auxiliary objective. These issues affect the paper's core claim rather than its presentation.
major comments (4)
- [Section 3.1, Eq. (1)] The inverse-prompt formula is dimensionally and mathematically invalid as stated. For a single point prompt, J is in R^{N'×3} and J_sim is in R^{H'W'D'×N'}; neither matrix is square, so the ordinary inverses J^{-1} and (J_sim)^{-1} do not exist. The argument linearizes the SAM decoder and the cosine-similarity map as affine functions, but no justification is given for such a global linearization, and no pseudo-inverse construction, rank condition, or injectivity argument is provided. Without these, the claimed recovery of a unique 3D prompt from a high-dimensional similarity volume is not established, and the 'primary prompt' is not computed by the stated formula.
- [Section 3.1, Eq. (2)] The auxiliary prompt update is geometrically ill-defined. It computes Hausdorff distances d_{x→y}^H and d_{y→x}^H between the contours of R_x^k and R_y^k, and uses x* (a point in the source image) as a location in the target image for the positive prompt. Before any transformation is estimated, R_x and R_y live in different coordinate spaces, so these distances and the expression x* + ε∇f(r)|_{r=x*} are not well-defined. The paper provides no current estimate of a transformation that would make these operations meaningful.
- [Section 3.1 and Table 3] The ablation in Table 3 shows that with σ=+∞ (primary prompt only), Dice drops from 77.76 to 70.12 and TRE worsens from 2.06 to 3.12 on MR-Prostate. Thus the iterative boundary-fitting loop of Eq. (2), not the inverse-prompt formula of Eq. (1), is responsible for a substantial part of the reported performance. The central claim of 'training-free registration via inverse prompt engineering' is therefore not supported unless the contributions of Eq. (1) and Eq. (2) are separated and the primary-only result is reported and analyzed.
- [Section 2, Section 3.1, and Section 4] The corresponding-prompt problem is defined by the requirement that the two prompted segmentations be corresponding ROIs, and Eq. (2) optimizes the alignment of the predicted segmentations. The reported evaluation metrics Dice and TRE are computed on the same anatomical ROI overlap that the auxiliary correction is designed to improve. While the auxiliary loop operates on SAM predictions rather than ground-truth labels, the manuscript does not provide an independent measure of registration accuracy that is not aligned with the optimized objective, so the claim of competitive performance with weakly-supervised methods is not fully established.
minor comments (4)
- [Section 3.1] In the displayed equation `G_k = f(Z_x^k) = f_proto(D(F_x, E_pr(Z_y^k)), F_x)`, the prompt argument should be `Z_x^k`; the variables `Z_x^k` and `Z_y^k` are otherwise confused throughout the paragraph.
- [Section 3.1, Eq. (1)] The quantity `ρ` is described as a scalar constant but is added to a three-dimensional prompt; its dimensionality and units should be specified.
- [Section 3.2] The word 'benifit' should be 'benefit'.
- [Section 4] Table 5 is referenced but its content is not included in the manuscript, and the code demo link 'PromptReg' appears to be a placeholder rather than a working URL.
Circularity Check
No significant circularity: the 'inverse prompt' derivation is under-specified but the reported evaluation is externally grounded.
full rationale
The paper's load-bearing derivation (Eq. 1) is not equivalent to its inputs. It attempts to invert a prompt through a Taylor linearization of the SAM decoder and a cosine-similarity map; while the Jacobians are rectangular and no pseudo-inverse is defined, this is a mathematical rigor gap rather than a circular reduction. The auxiliary prompt correction (Eq. 2) is an iterative refinement that drives the SAM segmentations into Hausdorff agreement, and the 'corresponding prompt problem' is a definition, not a derived prediction. Crucially, the reported Dice and TRE are computed on expert-labeled anatomical structures, not on the SAM segmentations used in the optimization, so the results are not forced by the fitting loop. Self-citations to SAMReg [17] motivate the problem and serve as a baseline, but the derivation of PromptReg does not rest on that citation. No step reduces by construction to its own output.
Assumptions & free parameters
free parameters (4)
- sigma =
20.0 pixels
- epsilon =
2.0
- rho =
not specified (manual constant)
- number of prompts K =
4
assumptions (4)
- domain assumption SAM and its variants produce accurate prompt-conditioned segmentations on medical and aerial images without fine-tuning.
- ad hoc to paper First-order Taylor expansion of the segmentation function with respect to the prompt is valid and invertible.
- domain assumption Cosine similarity between prototype and image features is a faithful class correspondence metric.
- ad hoc to paper Asymmetry in bidirectional Hausdorff distances indicates segmentation misalignment correctable by adding prompts.
Cite this review
Pith. "Pith review of Register Anything: Estimating "Corresponding Prompts" for Segment Anything Model." pith.science (2026). https://pith.science/paper/TB2WB4QU
@misc{pith2026250801697,
author = {Pith},
title = {Pith review of: Register Anything: Estimating "Corresponding Prompts" for Segment Anything Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/TB2WB4QU}},
note = {Machine review of arXiv:2508.01697}
}
read the original abstract
Establishing pixel/voxel-level or region-level correspondences is the core challenge in image registration. The latter, also known as region-based correspondence representation, leverages paired regions of interest (ROIs) to enable regional matching while preserving fine-grained capability at pixel/voxel level. Traditionally, this representation is implemented via two steps: segmenting ROIs in each image then matching them between the two images. In this paper, we simplify this into one step by directly "searching for corresponding prompts", using extensively pre-trained segmentation models (e.g., SAM) for a training-free registration approach, PromptReg. Firstly, we introduce the "corresponding prompt problem", which aims to identify a corresponding Prompt Y in Image Y for any given visual Prompt X in Image X, such that the two respectively prompt-conditioned segmentations are a pair of corresponding ROIs from the two images. Secondly, we present an "inverse prompt" solution that generates primary and optionally auxiliary prompts, inverting Prompt X into the prompt space of Image Y. Thirdly, we propose a novel registration algorithm that identifies multiple paired corresponding ROIs by marginalizing the inverted Prompt X across both prompt and spatial dimensions. Comprehensive experiments are conducted on five applications of registering 3D prostate MR, 3D abdomen MR, 3D lung CT, 2D histopathology and, as a non-medical example, 2D aerial images. Based on metrics including Dice and target registration errors on anatomical structures, the proposed registration outperforms both intensity-based iterative algorithms and learning-based DDF-predicting networks, even yielding competitive performance with weakly-supervised approaches that require fully-segmented training data.
Figures
Reference graph
Works this paper leans on
-
[1]
The Lancet389(10071), 815–822 (2017)
Ahmed, H.U., Bosaily, A.E.S., Brown, L.C., Gabe, R., Kaplan, R., Parmar, M.K., Collaco-Moraes, Y., Ward, K., Hindley, R.G., Freeman, A., et al.: Diagnostic accu- racy of multi-parametric mri and trus biopsy in prostate cancer (promis): a paired validating confirmatory study. The Lancet389(10071), 815–822 (2017)
2017
-
[2]
IEEE transactions on medical imaging38(8), 1788–1800 (2019)
Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Voxelmorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging38(8), 1788–1800 (2019)
2019
-
[3]
arXiv preprint arXiv:2211.02701 (2022)
Cardoso, M.J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., Murrey, B., Myronenko, A., Zhao, C., Yang, D., et al.: Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701 (2022)
arXiv 2022
-
[4]
Medical image analysis82, 102615 (2022)
Chen, J., Frey, E.C., He, Y., Segars, W.P., Li, Y., Du, Y.: Transmorph: Transformer for unsupervised medical image registration. Medical image analysis82, 102615 (2022)
2022
-
[5]
Chen, S., Xie, M.: Augmenting advertiser decision support with generative ai and interactive analytics. Authorea Preprints (2025)
work page 2025
-
[6]
arXiv preprint arXiv:2308.16184 (2023)
Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., et al.: Sam-med2d. arXiv preprint arXiv:2308.16184 (2023)
arXiv 2023
-
[7]
Medical image analysis52, 128–143 (2019)
De Vos, B.D., Berendsen, F.F., Viergever, M.A., Sokooti, H., Staring, M., Išgum, I.: A deep learning framework for unsupervised affine and deformable image regis- tration. Medical image analysis52, 128–143 (2019)
2019
-
[8]
arXiv preprint arXiv:1606.03798 (2016)
DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation. arXiv preprint arXiv:1606.03798 (2016)
arXiv 2016
Show all 31 references
-
[9]
In: BMVC
Dong, N., Xing, E.P.: Few-shot semantic segmentation with prototype learning. In: BMVC. vol. 3, p. 4 (2018)
2018
-
[10]
In: Medical Imaging 2018: Image Processing
Eppenhof, K.A., Lafarge, M.W., Moeskops, P., Veta, M., Pluim, J.P.: Deformable image registration using convolutional neural networks. In: Medical Imaging 2018: Image Processing. vol. 10574, pp. 192–197. SPIE (2018) 10 Shiqi Huang et al
2018
-
[11]
In: MIDL (2022)
Evan, M.Y., Wang, A.Q., Dalca, A.V., Sabuncu, M.R.: Keymorph: Robust multi- modal affine registration via unsupervised keypoint detection. In: MIDL (2022)
2022
-
[12]
NeurIPS (2024)
He, C., Li, K., Zhang, Y., Xu, G., Tang, L., Zhang, Y., Guo, Z., Li, X.: Weakly- supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping. NeurIPS (2024)
2024
-
[13]
https://doi.org/10.5281/zenodo.3835682 (2020)
Hering, A., Murphy, K., van Ginneken, Bram: Learn2reg challenge: Ct lung regis- tration. https://doi.org/10.5281/zenodo.3835682 (2020)
2020 doi
-
[14]
In: MICCAI (2019)
Hu, Y., Gibson, E., Barratt, D.C., Emberton, M., Noble, J.A., Vercauteren, T.: Conditional segmentation in lieu of image registration. In: MICCAI (2019)
2019
-
[15]
Medical image analysis 49, 1–13 (2018)
Hu, Y., Modat, M., Gibson, E., Li, W., Ghavami, N., Bonmati, E., Wang, G., Bandula, S., Moore, C.M., Emberton, M., et al.: Weakly-supervised convolutional neural networks for multimodal image registration. Medical image analysis 49, 1–13 (2018)
2018
-
[16]
In: CVPR (2023)
Huang, S., Xu, T., Shen, N., Mu, F., Li, J.: Rethinking few-shot medical segmen- tation: a vector quantization view. In: CVPR (2023)
2023
-
[17]
In: MICCAI (2024)
Huang, S., Xu, T., Shen, Z., Saeed, S.U., Yan, W., Barratt, D., Hu, Y.: One reg- istration is worth two segmentations. In: MICCAI (2024)
2024
-
[18]
Medical Image Analysis69, 101950 (2021)
Kavur, A.E., Gezer, N.S., Barış, M., Aslan, S., Conze, P.H., Groza, V., Pham, D.D., Chatterjee, S., Ernst, P., Özkan, S., et al.: Chaos challenge-combined (ct- mr) healthy abdominal organ segmentation. Medical Image Analysis69, 101950 (2021)
2021
-
[19]
NeurIPS (2024)
Ke, L., Ye, M., Danelljan, M., Tai, Y.W., Tang, C.K., Yu, F., et al.: Segment anything in high quality. NeurIPS (2024)
2024
-
[20]
Kevin, E., Bin, L., Adib, K.: Multimodal biomedical dataset for evaluating regis- tration methods (full-size tma cores).https://doi.org/10.5281/zenodo.4550300 (2021)
2021 doi
-
[21]
arXiv preprint arXiv:2304.02643 (2023)
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
2023 arXiv
-
[22]
Nature Communications15(1), 654 (2024)
Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)
2024
-
[23]
https://doi.org/10.5281/ zenodo.5914759 (2022)
Michele, V., Vittorio, F.: Zurich summer dataset. https://doi.org/10.5281/ zenodo.5914759 (2022)
2022
-
[24]
Journal of medical imaging 1(2), 024003–024003 (2014)
Modat,M.,Cash,D.M.,Daga,P.,Winston,G.P.,Duncan,J.S.,Ourselin,S.:Global image registration using a symmetric block-matching approach. Journal of medical imaging 1(2), 024003–024003 (2014)
2014
-
[25]
Journal of the American Statistical Association92(437), 179– 191 (1997)
Raftery, A.E., Madigan, D., Hoeting, J.A.: Bayesian model averaging for linear regression models. Journal of the American Statistical Association92(437), 179– 191 (1997)
1997
-
[26]
In: MICCAI (2017)
Rohé, M.M., Datar, M., Heimann, T., Sermesant, M., Pennec, X.: Svf-net: learning deformable image registration using shape matching. In: MICCAI (2017)
2017
-
[27]
IEEE Transac- tions on medical imaging20(6), 526–534 (2001)
Rohr, K., Stiehl, H.S., Sprengel, R., Buzug, T.M., Weese, J., Kuhn, M.: Landmark- based elastic registration using approximating thin-plate splines. IEEE Transac- tions on medical imaging20(6), 526–534 (2001)
2001
-
[28]
IEEE Transactions on Medical Imaging 18(8), 712–721 (1999)
Rueckert, D., Sonoda, L., Hayes, C., Hill, D., Leach, M., Hawkes, D.: Nonrigid registration using free-form deformations: application to breast mr images. IEEE Transactions on Medical Imaging 18(8), 712–721 (1999). https://doi.org/10. 1109/42.796284
1999
-
[29]
arXiv preprint arXiv:2310.15161 (2023) Title Suppressed Due to Excessive Length 11
Wang, H., Guo, S., Ye, J., Deng, Z., Cheng, J., Li, T., Chen, J., Su, Y., Huang, Z., Shen, Y., et al.: Sam-med3d. arXiv preprint arXiv:2310.15161 (2023) Title Suppressed Due to Excessive Length 11
2023 arXiv
-
[30]
arXiv preprint arXiv:2304.12620 (2023)
Wu, J., Fu, R., Fang, H., Liu, Y., Wang, Z., Xu, Y., Jin, Y., Arbel, T.: Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620 (2023)
2023 arXiv
-
[31]
In: CVPR (2018)
Yang, H.M., Zhang, X.Y., Yin, F., Liu, C.L.: Robust classification with convolu- tional prototype learning. In: CVPR (2018)
2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.