Pith. sign in

REVIEW 5 major objections 5 minor 25 references

Are Vision Foundation Models Ready for Out-of-the-Box Medical Image Registration?

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A frozen vision foundation model registers breast MRI out of the box, but only coarse anatomy aligns well.

desk verdict Useful benchmark; SAM wins for gross breast alignment, but encoder comparison is weakened by task-tuned PCA and missing significance tests. read the letter →

arxiv 2507.11569 v2 pith:QOGISDXL submitted 2025-07-15 eess.IV cs.AIcs.CV

classification eess.IVcs.AIcs.CV
keywords medicalimageregistrationvisionfoundationmodelszero-shotbreastMRIfibroglandulartissueSegmentAnythingModelfeature-basedcross-modality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether vision foundation models can be dropped into deformable medical image registration with no task-specific training. Its answer is yes for coarse anatomy: a training-free pipeline that runs ConvexAdam on features from the frozen Segment Anything Model encoder aligns breast MRI better than classical optimization methods, and the gap grows when sequences, modalities, or disease status differ between the two volumes. Its answer is no for fine anatomy: every tested encoder leaves fibroglandular tissue, the clinically critical dense tissue, substantially misaligned. The paper also reports, with hedging, that pretraining on medical or breast-specific images does not improve zero-shot registration and can hurt it. If these findings hold, foundation-model features are immediately useful for global breast and organ alignment, while fine-structure accuracy remains an open problem.

What carries the argument

The load-bearing mechanism is the training-free feature registration pipeline. Each axial slice of a volume is passed through a frozen 2D foundation-model encoder, skipped slices are linearly interpolated, patch-token features are projected with PCA to 10–16 channels per encoder, and ConvexAdam—a two-stage scheme of coupled convex discrete search followed by Adam refinement—optimizes a smoothness-regularized similarity energy on the resulting 4D feature volumes. Within this pipeline, SAM's encoder is the pivotal object: because it was trained with a masked autoencoder and then a promptable segmentation objective, its patch tokens are spatially localized and structure-aware, and the paper argues this is why its features outperform the contrastively trained DINO-v2 and the coarser medical-domain encoders.

What would settle it

Run the released code on held-out breast MRI pairs with input resolution and PCA dimension held identical across encoders; if SAM's advantage over SyN and NiftyReg disappears or is matched by another encoder once per-encoder tuning is removed, the claim that SAM's pretraining is the source of the gain is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the pretraining objective and data scale of a vision encoder, not its medical-domain exposure, determine whether its features support zero-shot deformable registration. Across four breast MRI tasks—same-sequence longitudinal, cross-sequence longitudinal, lesion-present to lesion-absent, and PET-CT to MRI—the SAM encoder achieves the best breast Dice scores (87.0%, 85.3%, and 86.9% on the three quantitative tasks), beating SyN, NiftyReg, and ConvexAdam on raw images as well as DINO-v2, MedSAM, SSLSAM, and MedCLIP-SAM. The advantage is largest for large, smooth structures and under large domain shifts, where intensity-based similarity fails. In contrast, fibroglandular tissue is never aligned well, and the paper shows that additional medical or breast-specific pretraining reduces performance on unseen sequences, which it attributes to smaller and narrower pretraining sets. The paper also notes that the zero-shot results still trail trained methods, giving a precise boundary to what out-of-the-box features currently achieve.

Load-bearing premise

The load-bearing premise is that the fixed pipeline—each encoder run at its native input resolution, a per-encoder PCA dimension tuned on nine pairs from one task, and ConvexAdam with fixed hyperparameters—is a fair common testbed, so the observed Dice differences reflect pretraining strategy rather than configuration.

Editorial extensions

If this is right

  • For longitudinal and cross-modality breast registration, a frozen SAM encoder plus ConvexAdam is a drop-in alternative to SyN and NiftyReg when no training data or annotated masks are available.
  • The advantage of foundation-model features grows as the domain gap grows, so the approach is most valuable exactly where intensity-based registration methods are least reliable.
  • Because fibroglandular tissue stays misaligned, clinical workflows that track dense tissue cannot yet rely on this zero-shot pipeline for the structures that matter most.
  • Encoder selection should be driven by the pretraining objective rather than by whether the pretraining data were medical; segmentation-style pretraining beat vision-language and small medical-domain pretraining.
  • Feature dimensionality can be kept small: the paper's ablation shows performance plateaus around 10–16 principal components while memory and computation keep increasing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the per-encoder PCA dimension was tuned on nine pairs from Task 3, a stricter test would fix all pipeline settings independently of the task before comparing encoders.
  • Editorial inference: the PET-to-MRI result suggests the same features may align other large cross-modality pairs such as ultrasound-to-MRI, but the prone-to-supine posture change in that task confounds pure modality transfer.
  • Editorial inference: the authors explicitly decline to draw definitive conclusions about medical pretraining, so the defensible claim is narrower than the title suggests—among these five encoders, domain-specific pretraining did not help.
  • Editorial inference: a natural extension, left implicit, is to use SAM features for global alignment and hand the result to a trained or intensity-based fine-tuning stage to recover fibroglandular tissue.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a benchmark of five vision foundation model (VFM) encoders (DINO-v2, SAM, MedSAM, SSLSAM, MedCLIP-SAM) for training-free, zero-shot deformable registration of breast MRI. The method extracts slice-wise 2D features from a frozen encoder, reduces their dimensionality with PCA, and registers the resulting feature volumes with ConvexAdam. The evaluation spans four tasks: longitudinal same-sequence, longitudinal cross-sequence, lesion tracking, and PET-CT to MRI registration. The main reported findings are that SAM achieves the best alignment of the whole breast, that VFMs outperform traditional optimization-based methods on large structures but not on fibroglandular tissue (FGT), and that medical-domain pre-training (MedSAM, SSLSAM) does not improve and may even degrade performance. The authors release their code publicly.

Significance. If substantiated, this benchmark would be a useful reference for the community: it tests foundation models beyond brain/abdomen registration, compares pretraining strategies in a controlled pipeline, and identifies a concrete failure mode (FGT alignment). The public code and the use of multiple clinical tasks are strengths. However, the central claims—especially the 'clear advantage' of SAM and the negative result on medical pre-training—are currently under-supported by the experimental design and statistical analysis. The paper is of moderate significance as a preliminary empirical study; with strengthened analysis it could become a solid benchmark contribution.

major comments (5)
  1. [§3.2] The pipeline is not a fixed common testbed. The PCA dimension d is optimized on 9 sampled pairs from Task 3 and then applied to all tasks, and Task 3 is exactly where SAM shows its largest margin over DINO-v2 (86.9 vs 80.5). Because the Fig. 2 ablation shows that performance can vary with d, this selection may bias the results in favor of SAM. Please either re-tune d per task (or per encoder per task) or report sensitivity of the main comparisons across a range of d. Additionally, encoder-specific input resolutions (1024×1024 for SAM/MedSAM/SSLSAM, 1792×1792 for DINO-v2, 224×224 for MedCLIP-SAM) confound encoder pretraining with effective feature resolution; a controlled experiment matching feature-map resolution or input resolution is needed to attribute the observed differences to encoder quality.
  2. [Table 1, Results] No statistical significance testing is reported for any of the 50-pair tasks. With n=50, paired tests (e.g., Wilcoxon signed-rank or permutation tests) should be applied to the key comparisons: SAM vs DINO-v2, SAM vs the best optimization-based method, and medical vs natural pretraining. The reported standard deviations are large; for instance, Task 1 FGT DSCs are 53.8±17.6 (SAM), 55.0±22.7 (SSLSAM), and 53.8±26.5 (NiftyReg), so the claim that 'SAM and SSLSAM perform better' is not self-evident. Without significance tests, the central 'clear advantage' claim for SAM is not statistically supported.
  3. [Table 1, Ablation study] The ablation comparing SAM features with raw images uses only 3 randomly selected Task 2 pairs. The conclusion in the Discussion that 'SAM's extracted features outperform raw images' is underpowered at n=3. Please run the ablation on a larger subset (e.g., 20–50 pairs) or provide confidence intervals and a paired test; otherwise this comparison should be labeled as preliminary.
  4. [Task 4, Abstract] The claim that VFMs 'outperform traditional optimization-based methods, especially in cross-modality tasks' is based entirely on qualitative visualization for the PET-CT-to-MRI task (Fig. 3(d)). Provide quantitative metrics for Task 4 (e.g., organ or breast DSC if masks are available, or at least a volume-overlap or intensity-based similarity measure) or explicitly qualify cross-modality conclusions as preliminary. As written, the abstract overstates the evidence for the cross-modality advantage.
  5. [Abstract vs. §4 Discussion] The abstract states that additional pre-training or fine-tuning on medical or breast-specific images 'does not improve registration performance and may even decrease it,' but the Discussion says 'We do not draw definitive conclusions at this stage regarding the effectiveness of medical pre-trained models.' Moreover, descriptively, SSLSAM outperforms SAM in Task 1 for FGT (55.0 vs 53.8) and organ (72.3 vs 69.1). Please reconcile these statements, either by presenting significance tests that support the abstract claim or by weakening the abstract to match the more cautious Discussion.
minor comments (5)
  1. [§3.1] There is a typo in the 'Tasks design' paragraph: 'described in a later paragraphTasks' should be 'described in a later paragraph, Tasks.'
  2. [Throughout] The acronym for the self-supervised SAM variant is inconsistently written as both 'SSLSAM' and 'SSL-SAM' (e.g., §2.2 and Table 1). Please use one form consistently.
  3. [Eq. (3), §2.1] The text says 'we set λ=2 and MSE for Sim' but Eq. (3) writes Sim as a generic similarity term; please specify that Sim is the sum of squared differences (or define MSE in the equation).
  4. [§4 Discussion] The comparison with GuidedMorph is mentioned only qualitatively ('lags significantly behind') without any quantitative result. Consider adding a row to Table 1 or a reference to the specific numbers from the GuidedMorph paper.
  5. [Table 1] The 'Lesion' column header reads 'Vε ↓' but the caption says 'Error Size(%).' Clarify that the values are percentages and define Vε in the caption, as it is only defined inline in §3.1.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the study is an empirical benchmark; the only tuned quantity is the PCA dimension, a hyperparameter, and no claim reduces by construction to its inputs.

full rationale

This paper makes no claimed derivation; its conclusions are empirical measurements of DSC and lesion size error on evaluated pairs. The central comparison (SAM versus other encoders versus SyN, NiftyReg, and ConvexAdam) is assessed directly, and no equation defines the outcome in terms of the input. The PCA dimension d is optimized on nine sampled pairs from Task 3, but d is a hyperparameter controlling feature dimensionality; the reported DSC values are subsequently measured, so the ranking is not forced by the choice of d. The claim that medical pre-training does not improve performance compares pre-trained checkpoints from external work (MedSAM, MedCLIP-SAM) and one prior paper by the same group (SSLSAM); the SSLSAM result is factual information about its pretraining data and does not carry the load of the main conclusion. The ablation replacing ConvexAdam with SyN or NiftyReg is a controlled comparison rather than a circular step. Citations to the authors' own segmentation models and GuidedMorph are used for mask generation and as an external training-based benchmark, respectively, and do not import the conclusion. No self-definitional, fitted-as-prediction, or uniqueness-imported pattern is present. Therefore the paper is self-contained as an empirical benchmark and should receive a low circularity score.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the adequacy of the feature-registration testbed, the accuracy of manually corrected masks, and the transferability of hyperparameters. None of these are proven by the paper, but they are standard assumptions for a benchmark study.

free parameters (1)
  • PCA dimension d per encoder = SAM: 16, MedSAM: 10, SSLSAM: 16, DINO-v2: 10, MedCLIP-SAM: 16
    Chosen by ablation on 9 sampled pairs from Task 3 to balance DSC plateau and compute cost, then applied to all tasks. This is a tuned pipeline hyperparameter rather than a theory-derived constant.
assumptions (4)
  • domain assumption Slice-by-slice 2D encoding of volumes, with linear interpolation of skipped slices, adequately represents 3D anatomy for deformable breast MRI registration.
    Invoked in Section 2.1 Feature extraction; most encoders are 2D, so this modeling choice is central to the pipeline.
  • domain assumption The ConvexAdam energy with MSE and lambda=2, taken from DINO-Reg, is a suitable common optimizer for all encoder feature volumes.
    Section 2.1 and implementation Section 3.2; if these hyperparameters favor some encoders, cross-encoder comparisons would be biased.
  • domain assumption Manual correction of segmentations by the authors yields gold-standard breast, FGT, and organ masks for DSC.
    Section 3.1 Mask collection; no inter-observer variability is reported, so measurement error in masks could affect all DSC comparisons.
  • domain assumption A single institutional dataset with 50 pairs per task is representative of the general breast MRI registration problem.
    Section 3.1 Dataset preparation; conclusions are generalized beyond this cohort despite limited external data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Vision Foundation Models Ready for Out-of-the-Box Medical Image Registration?." pith.science (2026). https://pith.science/paper/QOGISDXL

@misc{pith2026250711569,
  author       = {Pith},
  title        = {Pith review of: Are Vision Foundation Models Ready for Out-of-the-Box Medical Image Registration?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOGISDXL}},
  note         = {Machine review of arXiv:2507.11569}
}
read the original abstract

Foundation models, pre-trained on large image datasets and capable of capturing rich feature representations, have recently shown potential for zero-shot image registration. However, their performance has mostly been tested in the context of rigid or less complex structures, such as the brain or abdominal organs, and it remains unclear whether these models can handle more challenging, deformable anatomy. Breast MRI registration is particularly difficult due to significant anatomical variation between patients, deformation caused by patient positioning, and the presence of thin and complex internal structure of fibroglandular tissue, where accurate alignment is crucial. Whether foundation model-based registration algorithms can address this level of complexity remains an open question. In this study, we provide a comprehensive evaluation of foundation model-based registration algorithms for breast MRI. We assess five pre-trained encoders, including DINO-v2, SAM, MedSAM, SSLSAM, and MedCLIP, across four key breast registration tasks that capture variations in different years and dates, sequences, modalities, and patient disease status (lesion versus no lesion). Our results show that foundation model-based algorithms such as SAM outperform traditional registration baselines for overall breast alignment, especially under large domain shifts, but struggle with capturing fine details of fibroglandular tissue. Interestingly, additional pre-training or fine-tuning on medical or breast-specific images in MedSAM and SSLSAM, does not improve registration performance and may even decrease it in some cases. Further work is needed to understand how domain-specific training influences registration and to explore targeted strategies that improve both global alignment and fine structure accuracy. We also publicly release our code at \href{https://github.com/mazurowski-lab/Foundation-based-reg}{Github}.

Figures

Figures reproduced from arXiv: 2507.11569 by the authors.

Figure 1
Figure 1. Pipeline for VFM-based Breast Image Registration. The Found Encoder [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Encoder characteristics for each VFM (training strategy and dataset [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. (a). Visualization of example results for different VFM-based algorithms. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages

  1. [1]

    IEEE Access10, 122901–122911 (2022)

    Alfano, F., Cordero-Grande, L., Ortuño, J.E., García, K.F., García-Sevilla, M., Zamora, O.B., Conde, M.H., Lizarraga, S., Santos, A., Pascau, J., et al.: Breast tumor localization by prone to supine landmark driven registration for surgical planning. IEEE Access10, 122901–122911 (2022)

  2. [2]

    Medical image analysis12(1), 26–41 (2008)

    Avants, B.B., Epstein, C.L., Grossman, M., Gee, J.C.: Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis12(1), 26–41 (2008)

  3. [3]

    IEEE transactions on medical imaging38(8), 1788–1800 (2019)

    Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Voxelmorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging38(8), 1788–1800 (2019)

  4. [4]

    Data set (2015)

    Bloch, B.N., Jain, A., Jaffe, C.C.: Breast-diagnosis. Data set (2015). https:// doi.org/10.7937/K9/TCIA.2015.SDNRQXXR, http://doi.org/10.7937/K9/TCIA. 2015.SDNRQXXR, the Cancer Imaging Archive

  5. [5]

    Medical image analysis82, 102615 (2022)

    Chen, J., Frey, E.C., He, Y., Segars, W.P., Li, Y., Du, Y.: Transmorph: Transformer for unsupervised medical image registration. Medical image analysis82, 102615 (2022)

  6. [6]

    arXiv preprint arXiv:2505.13414 (2025)

    Chen, Y., Gu, H., Dong, H., Li, Q., Chen, Y., Konz, N., Li, L., Mazurowski, M.A.: Guidedmorph: Two-stage deformable registration for breast mri. arXiv preprint arXiv:2505.13414 (2025)

  7. [7]

    arXiv preprint arXiv:2504.15192 (2025)

    Chen, Y., Li, L., Gu, H., Dong, H., Nguyen, D.L., Kirk, A.D., Mazurowski, M.A., Hwang, E.S.: Breast density in mri: an ai-based quantification and relationship to assessment in mammography. arXiv preprint arXiv:2504.15192 (2025)

  8. [8]

    arXiv preprint arXiv:2506.12186 (2025)

    Dong, H., Chen, Y., Gu, H., Konz, N., Chen, Y., Li, Q., Mazurowski, M.A.: Mri-core: A foundation model for magnetic resonance imaging. arXiv preprint arXiv:2506.12186 (2025)

Show all 25 references
  1. [9]

    arXiv preprint arXiv:2408.00756 (2024)

    Dong, H., Gu, H., Chen, Y., Yang, J., Chen, Y., Mazurowski, M.A.: Segment anything model 2: an application to 2d and 3d medical images. arXiv preprint arXiv:2408.00756 (2024)

  2. [10]

    arXiv preprint arXiv:2404.09957 (2024)

    Gu,H.,Dong,H.,Yang,J.,Mazurowski,M.A.:Howtobuildthebestmedicalimage segmentation algorithm using foundation models: a comprehensive empirical study with segment anything model. arXiv preprint arXiv:2404.09957 (2024)

  3. [11]

    Medical image analysis71, 102036 (2021)

    Kim, B., Kim, D.H., Park, S.H., Kim, J., Lee, J.G., Ye, J.C.: Cyclemorph: cycle consistent unsupervised deformable image registration. Medical image analysis71, 102036 (2021)

  4. [12]

    In: Bengio, Y., LeCun, Y

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015), http://arxiv.org/abs/1412.6980

  5. [13]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)

  6. [14]

    IEEE transactions on medical imaging 29(1), 196–205 (2009)

    Klein, S., Staring, M., Murphy, K., Viergever, M.A., Pluim, J.P.: Elastix: a tool- box for intensity-based medical image registration. IEEE transactions on medical imaging 29(1), 196–205 (2009)

  7. [15]

    Koleilat, T., Asgariandehkordi, H., Rivaz, H., Xiao, Y.: Medclip-samv2: Towards universal text-driven medical image segmentation (2025), https://arxiv.org/ abs/2409.19483 12 Authors Suppressed Due to Excessive Length

  8. [16]

    Scientific reports14(1), 5383 (2024)

    Lew, C.O., Harouni, M., Kirksey, E.R., Kang, E.J., Dong, H., Gu, H., Grimm, L.J., Walsh, R., Lowell, D.A., Mazurowski, M.A.: A publicly available deep learning model and dataset for segmentation of breast, fibroglandular tissue, and vessels in breast mri. Scientific reports14(...

  9. [17]

    Nature Communications15(1), 654 (2024)

    Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)

  10. [18]

    Medical Image Analysis 89, 102918 (2023)

    Mazurowski, M.A., Dong, H., Gu, H., Yang, J., Konz, N., Zhang, Y.: Segment anything model for medical image analysis: an experimental study. Medical Image Analysis 89, 102918 (2023)

  11. [19]

    Journal of digital imaging31, 718–726 (2018)

    Mehrabian, H., Richmond, L., Lu, Y., Martel, A.L.: Deformable registration for longitudinal breast mri screening. Journal of digital imaging31, 718–726 (2018)

  12. [20]

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W.,Howes,R.,Huang,P.Y.,Li,S.W.,Misra,I.,Rabbat,M.,Sharma,V.,Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P....

  13. [21]

    Journal of Medical Imaging9(6), 065001–065001 (2022)

    Ringel, M.J., Richey, W.L., Heiselman, J.S., Luo, M., Meszoely, I.M., Miga, M.I.: Supine magnetic resonance image registration for breast surgery: insights on ma- terial mechanics. Journal of Medical Imaging9(6), 065001–065001 (2022)

  14. [22]

    Diagnostics13(11), 1947 (2023)

    Shi, P., Qiu, J., Abaxi, S.M.D., Wei, H., Lo, F.P.W., Yuan, W.: Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation. Diagnostics13(11), 1947 (2023)

  15. [23]

    IEEE Transactions on Medical Imaging (2024)

    Siebert, H., Großbröhmer, C., Hansen, L., Heinrich, M.P.: Convexadam: Self- configuring dual-optimisation-based 3d multitask medical image registration. IEEE Transactions on Medical Imaging (2024)

  16. [24]

    In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention

    Song, X., Xu, X., Yan, P.: Dino-reg: General purpose image encoder for training- free multi-modal deformable medical image registration. In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention. pp. 608–617. Springer (2024)

  17. [25]

    Nature Biomedical Engineering9(4), 521–538 (2025)

    Sun, Y., Wang, L., Li, G., Lin, W., Wang, L.: A foundation model for enhanc- ing magnetic resonance images and downstream segmentation, registration and diagnostic tasks. Nature Biomedical Engineering9(4), 521–538 (2025)

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.