pith. sign in

arxiv: 2606.22963 · v2 · pith:AAFY4YV6new · submitted 2026-06-22 · 💻 cs.CV

Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation

Yubo Zhou , Jianghao Wu , Ping Ye , Shaoting Zhang , Guotai Wang This is my paper

Pith reviewed 2026-07-02 21:39 UTC · model grok-4.3

classification 💻 cs.CV
keywords test-time adaptationSAM3medical image segmentationconcept alignment contrastprompt memorypseudo-labelingdomain adaptation
0
0 comments X

The pith

CM-TTA adapts SAM3 for medical segmentation by selecting reliable pseudo-labels via concept alignment contrast and balancing adaptation with long-short prompt memory.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a test-time adaptation framework for the Segment Anything Model 3 in medical imaging. Existing methods rely on image-level uncertainty, which fails to ensure region-level accuracy in segmentation. The new approach uses a Concept Alignment Contrast metric to measure how well visual predictions match textual concepts and selects the best augmented view for supervision. It also employs a Long-Short Prompt Memory to handle both quick local changes and stable global prompts, updated with dense supervision from enhanced pseudo-labels. Experiments show this leads to better results on prostate and skin lesion tasks compared to prior TTA methods.

Core claim

The central discovery is that leveraging textual-visual semantic consistency through the Concept Alignment Contrast metric allows for robust selection of high-quality augmented views as supervision, while the Long-Short Prompt Memory module enables both agile and stable prompt updates, and the Densely Supervised Prompt Update strategy optimizes the model using these pseudo-labels, resulting in superior test-time adaptation performance for SAM3 on medical images.

What carries the argument

The Concept Alignment Contrast (CAC) metric, which evaluates prediction quality based on textual-visual semantic consistency, and the Long-Short Prompt Memory (LSPM) that fuses recent and stable prompts.

If this is right

  • Enables reliable pseudo-supervision without annotations by using semantic consistency.
  • Achieves balance between rapid adaptation to new domains and long-term stability in continual settings.
  • Leads to improved segmentation accuracy on medical datasets like prostate and skin lesions.
  • Provides a way to update prompt embeddings densely supervised by enhanced pseudo-labels.
  • Outperforms existing vision-language TTA methods that rely on uncertainty minimization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This could extend to other vision-language models beyond SAM3 for domain adaptation in specialized imaging.
  • The method might reduce the need for large annotated medical datasets by allowing on-the-fly adaptation.
  • Potential for application in real-time clinical settings where domain shifts occur frequently.
  • If CAC proves reliable, it could serve as a general proxy for segmentation quality in unsupervised scenarios.

Load-bearing premise

That the textual-visual semantic consistency measured by the Concept Alignment Contrast metric serves as a reliable proxy for region-level semantic correctness of segmentation predictions when ground-truth annotations are unavailable.

What would settle it

A comparison showing no correlation between CAC scores and actual segmentation accuracy (e.g., Dice coefficient) on held-out medical images with ground truth would falsify the reliability of the metric for supervision selection.

Figures

Figures reproduced from arXiv: 2606.22963 by Guotai Wang, Jianghao Wu, Ping Ye, Shaoting Zhang, Yubo Zhou.

Figure 1
Figure 1. Figure 1: Overview of our CM-TTA for test-time adaptation of SAM3. and skin lesion segmentation tasks by an average Dice gain of 3.17 and 6.07 percentage points over the baseline. 2 Method Previous work has shown that directly fine-tuning all parameters of founda￾tion models is computationally expensive and can reduce the generalizability, while prompt tuning avoids distorting pre-trained representations and enables… view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative comparison of different TTA methods on two datasets. baseline, meaning inference with SAM3 without adaptation. Method-specific hy￾perparameters of the compared methods were tuned for optimal results [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) Comparison of the quality of the best augmented view selected by different metrics on Promise12. (b) Effect of short memory bank length L on average Dice. (c) Effect of the weight of Lc on average Dice. 0.57s for the inference of one test sample on Promise12, which is close to the inference time using unadapted SAM3 directly (0.31s). The qualitative results shown in [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
read the original abstract

Concept segmentation models like Segment Anything Model 3 (SAM3) show strong generalization on natural images, yet their performance degrades in medical imaging due to the domain gap caused by different imaging principles and styles. Test-Time Adaptation (TTA) is essential for improving the testing performance by updating the model on the fly without annotations. However, existing vision-language TTA methods are mainly driven by image-level uncertainty minimization, which does not necessarily reflect region-level semantic correctness in medical segmentation. Moreover, they often lack mechanisms to maintain stability in continual one-pass adaptation, leading to limited performance when reliable dense supervision is missing for segmentation. To address these issues, we propose Concept Alignment Contrast and LongShort Prompt Memory for Test-Time Adaptation (CM-TTA) of SAM3 for medical images. First, for a test sample with multiple augmentations, we introduce a novel Concept Alignment Contrast (CAC) metric, which leverages textual-visual semantic consistency to robustly evaluate prediction quality to select the best augmented view as the supervision. Second, to balance rapid and stable adaptation, we design a Long-Short Prompt Memory (LSPM) module. The short memory dynamically fuses recent prompts based on CAC scores for agile local adaptation, while the long memory maintains a stable global prompt to generate enhanced pseudo-labels. Finally, a Densely Supervised Prompt Update (DSPU) strategy is proposed to optimize the prompt embeddings with enhanced pseudo labels as dense supervision. Extensive experiments on prostate and skin lesion segmentation demonstrate that our CM-TTA framework significantly outperforms existing methods for TTA of SAM3. The code is available at https://github.com/SherlockZYB/CM-TTA.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes CM-TTA, a test-time adaptation framework for SAM3 on medical images. It introduces a Concept Alignment Contrast (CAC) metric that selects the best augmentation view via textual-visual semantic consistency, a Long-Short Prompt Memory (LSPM) module that fuses recent prompts (short memory) with a stable global prompt (long memory), and a Densely Supervised Prompt Update (DSPU) strategy that uses the resulting enhanced pseudo-labels for prompt optimization. Experiments on prostate and skin-lesion segmentation are claimed to show significant gains over prior TTA methods for SAM3.

Significance. If the central proxy holds, the work supplies a concrete mechanism for generating region-level pseudo-supervision in one-pass TTA without relying solely on image-level uncertainty, which could improve deployment of promptable foundation models in annotation-scarce medical settings. Public code release is a clear reproducibility asset.

major comments (2)
  1. [Method description of CAC and pseudo-label generation] The framework's selection of augmented views and the quality of DSPU pseudo-labels rest entirely on the unverified assumption that higher CAC scores correlate with higher region-level semantic accuracy. No correlation analysis (e.g., CAC versus Dice on a labeled hold-out set) or ablation that isolates this proxy is described, which directly affects the reliability of the reported outperformance.
  2. [Abstract] Abstract: the claim that CM-TTA 'significantly outperforms existing methods' is presented without any numerical results, baseline names, statistical tests, or dataset sizes, preventing assessment of effect size or experimental soundness.
minor comments (2)
  1. [Title and Abstract] Notation inconsistency: the title uses 'Long-Short Prompt Memory' while the abstract uses 'LongShort Prompt Memory'; standardize throughout.
  2. [Method] The paper would benefit from an explicit statement of the number of augmentations per test sample and the precise textual prompts used for CAC computation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help improve the clarity and rigor of our work. We address each major comment below and will incorporate revisions as noted.

read point-by-point responses

  1. Referee: [Method description of CAC and pseudo-label generation] The framework's selection of augmented views and the quality of DSPU pseudo-labels rest entirely on the unverified assumption that higher CAC scores correlate with higher region-level semantic accuracy. No correlation analysis (e.g., CAC versus Dice on a labeled hold-out set) or ablation that isolates this proxy is described, which directly affects the reliability of the reported outperformance.

    Authors: We agree that a direct correlation analysis between CAC scores and region-level accuracy (e.g., Dice on a labeled hold-out) would provide stronger validation of the proxy. Our current experiments demonstrate consistent gains on prostate and skin-lesion tasks, but we will add both a correlation study and an ablation isolating CAC in the revised manuscript to address this concern. revision: yes

  2. Referee: [Abstract] Abstract: the claim that CM-TTA 'significantly outperforms existing methods' is presented without any numerical results, baseline names, statistical tests, or dataset sizes, preventing assessment of effect size or experimental soundness.

    Authors: We acknowledge the abstract should include quantitative support. We will revise it to report key Dice/IoU gains, baseline names (e.g., prior TTA methods for SAM3), dataset sizes, and statistical test results to allow proper evaluation of effect size. revision: yes

Circularity Check

0 steps flagged

No circularity in derivation chain

full rationale

The paper proposes new components (CAC metric for selecting augmented views via textual-visual consistency, LSPM for balancing short/long prompt memory, and DSPU for dense prompt updates) that are defined independently and applied to external augmentations and generated pseudo-labels. No step reduces a claimed prediction or result to a fitted input by construction, nor relies on self-citation chains or imported uniqueness theorems for its core logic. Experimental outperformance claims rest on external benchmarks rather than tautological redefinitions, rendering the method self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 3 invented entities

The central claim depends on the effectiveness of three newly introduced components (CAC, LSPM, DSPU) that are postulated to solve identified limitations rather than derived from external benchmarks or prior results.

invented entities (3)
  • Concept Alignment Contrast (CAC) metric no independent evidence
    purpose: To select the best augmented view as supervision using textual-visual semantic consistency
    Newly proposed component central to the method
  • Long-Short Prompt Memory (LSPM) module no independent evidence
    purpose: To balance agile local adaptation with stable global prompt maintenance
    Newly proposed component central to the method
  • Densely Supervised Prompt Update (DSPU) strategy no independent evidence
    purpose: To optimize prompt embeddings using enhanced pseudo-labels as dense supervision
    Newly proposed component central to the method

pith-pipeline@v0.9.1-grok · 5846 in / 1234 out tokens · 31505 ms · 2026-07-02T21:39:04.593481+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · 2 internal anchors

  1. [1]

    SAM 3: Segment Anything with Concepts

    Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Al- wala, K.V., Khedr, H., Huang, A., et al.: SAM 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    In: CVPR

    Chen, Z., Pan, Y., Ye, Y., Lu, M., Xia, Y.: Each test image deserves a specific prompt: Continual test-time adaptation for 2d medical image segmentation. In: CVPR. pp. 11184–11193 (2024)

    work page 2024

  3. [3]

    Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)

    Codella, N., Rotemberg, V., Tschandl, P., Celebi, M.E., Dusza, S., Gutman, D., Helba,B.,Kalloo,A.,Liopyris,K.,Marchetti,M.,etal.:Skinlesionanalysistoward melanoma detection 2018: A challenge hosted by the International Skin Imaging Collaboration (ISIC). arXiv preprint arXiv:1902.03368 (2019)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  4. [4]

    NeurIPS37, 129062–129093 (2024)

    Farina, M., Franchi, G., Iacca, G., Mancini, M., Ricci, E.: Frustratingly easy test- time adaptation of vision-language models. NeurIPS37, 129062–129093 (2024)

    work page 2024

  5. [5]

    Medical image analysis 72, 102136 (2021)

    He, Y., Carass, A., Zuo, L., Dewey, B.E., Prince, J.L.: Autoencoder based self- supervised test-time adaptation for medical image analysis. Medical image analysis 72, 102136 (2021)

    work page 2021

  6. [6]

    In: WACV

    Imam, R., Gani, H., Huzaifa, M., Nandakumar, K.: Test-time low rank adaptation via confidence maximization for zero-shot generalization of vision-language models. In: WACV. pp. 5449–5459. IEEE (2025)

    work page 2025

  7. [7]

    arXiv preprint arXiv:2601.10880 (2026)

    Jiang, C., Ding, T., Song, C., Tu, J., Yan, Z., Shao, Y., Wang, Z., Shang, Y., Han, T., Tian, Y.: Medical SAM3: A foundation model for universal prompt-driven medical image segmentation. arXiv preprint arXiv:2601.10880 (2026)

    work page arXiv 2026

  8. [8]

    arXiv preprint arXiv:2202.10054 (2022)

    Kumar, A., Raghunathan, A., Jones, R., Ma, T., Liang, P.: Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054 (2022)

    work page arXiv 2022

  9. [9]

    In: CVPR

    Lee, T., Chottananurak, S., Gong, T., Lee, S.J.: Aetta: Label-free accuracy esti- mation for test-time adaptation. In: CVPR. pp. 28643–28652 (2024)

    work page 2024

  10. [10]

    In: MICCAI

    Li, X., Fang, H., Wang, C., Liu, M., Duan, L., Xu, Y.: Cache-driven spatial test- time adaptation for cross-modality medical image segmentation. In: MICCAI. pp. 146–156. Springer (2024)

    work page 2024

  11. [11]

    International Journal of Computer Vision133(1), 31–64 (2025)

    Liang, J., He, R., Tan, T.: A comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision133(1), 31–64 (2025)

    work page 2025

  12. [12]

    Medical image analysis 18(2), 359–373 (2014) 10 Y

    Litjens, G., Toth, R., Van De Ven, W., Hoeks, C., Kerkstra, S., Van Ginneken, B., Vincent, G., Guillard, G., Birbeck, N., Zhang, J., et al.: Evaluation of prostate seg- mentation algorithms for MRI: the PROMISE12 challenge. Medical image analysis 18(2), 359–373 (2014) 10 Y. Zhou et al

    work page 2014

  13. [13]

    arXiv preprint arXiv:2511.19046 (2025)

    Liu, A., Xue, R., Cao, X.R., Shen, Y., Lu, Y., Li, X., Chen, Q., Chen, J.: Med- SAM3: Delving into segment anything with medical concepts. arXiv preprint arXiv:2511.19046 (2025)

    work page arXiv 2025

  14. [14]

    In: AAAI

    Liu, Q., Chen, C., Dou, Q., Heng, P.A.: Single-domain generalization in medical image segmentation via test-time adaptation from shape dictionary. In: AAAI. vol. 36, pp. 1756–1764 (2022)

    work page 2022

  15. [15]

    Medical image analysis83, 102641 (2023)

    Liu, X., Xing, F., El Fakhri, G., Woo, J.: Memory consistent unsupervised off-the- shelf model adaptation for source-relaxed medical image segmentation. Medical image analysis83, 102641 (2023)

    work page 2023

  16. [16]

    In: NeurIPS

    Lu, Y., Xu, J., Peng, Z., Li, R., Zhang, R., et al.: Historical test-time prompt tuning for vision foundation models. In: NeurIPS. vol. 37, pp. 12872–12896 (2024)

    work page 2024

  17. [17]

    IEEE Transactions on Medical Imaging41(6), 1560– 1574 (2022)

    Mishra, S., Zhang, Y., Chen, D.Z., Hu, X.S.: Data-driven deep supervision for medical image segmentation. IEEE Transactions on Medical Imaging41(6), 1560– 1574 (2022)

    work page 2022

  18. [18]

    Journal of Medical Physics35(1), 3–14 (2010)

    Sharma, N., Aggarwal, L.M.: Automated medical image segmentation techniques. Journal of Medical Physics35(1), 3–14 (2010)

    work page 2010

  19. [19]

    In: NeurIPS (2025)

    Sheng, L., Liang, J., He, R., Wang, Z., Tan, T.: The illusion of progress? A critical look at test-time adaptation for vision-language models. In: NeurIPS (2025)

    work page 2025

  20. [20]

    In: NeurIPS

    Shu, M., Nie, W., Huang, D.A., Yu, Z., Goldstein, T., Anandkumar, A., Yuan, L.: Test-time prompt tuning for zero-shot generalization in vision-language models. In: NeurIPS. vol. 35, pp. 14274–14289 (2022)

    work page 2022

  21. [21]

    NeurIPS30 (2017)

    Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NeurIPS30 (2017)

    work page 2017

  22. [22]

    In: ICLR (2021)

    Wang, D., Shelhamer, E., Liu, S., Olshausen, B., Darrell, T.: Tent: Fully test-time adaptation by entropy minimization. In: ICLR (2021)

    work page 2021

  23. [23]

    In: CVPR

    Wang, Q., Fink, O., Van Gool, L., Dai, D.: Continual test-time domain adaptation. In: CVPR. pp. 7201–7211 (2022)

    work page 2022

  24. [24]

    IET image processing16(5), 1243– 1267 (2022)

    Wang, R., Lei, T., Cui, R., Zhang, B., Meng, H., Nandi, A.K.: Medical image segmentation using deep learning: A survey. IET image processing16(5), 1243– 1267 (2022)

    work page 2022

  25. [25]

    In: CVPR

    Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Lopes, R.G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al.: Robust fine-tuning of zero- shot models. In: CVPR. pp. 7959–7971 (2022)

    work page 2022

  26. [26]

    IEEE Transactions on Medical Imaging43(9), 3098–3109 (2024)

    Wu, J., Guo, D., Wang, G., Yue, Q., Yu, H., Li, K., Zhang, S.: Fpl+: Filtered pseudo label-based unsupervised cross-modality adaptation for 3d medical image segmentation. IEEE Transactions on Medical Imaging43(9), 3098–3109 (2024)

    work page 2024

  27. [27]

    In: ICLR (2025)

    Xiao, Z., Yan, S., Hong, J., Cai, J., Jiang, X., Hu, Y., Shen, J., Wang, Q., Snoek, C.G.M.: Dynaprompt: Dynamic test-time prompt tuning. In: ICLR (2025)

    work page 2025

  28. [28]

    Medical Image Analysis88, 102873 (2023)

    Xu, X., Chen, Y., Wu, J., Lu, J., Ye, Y., Huang, Y., Dou, X., Li, K., Wang, G., Zhang, S., Gong, W.: A novel one-to-multiple unsupervised domain adaptation framework for abdominal organ segmentation. Medical Image Analysis88, 102873 (2023)

    work page 2023

  29. [29]

    IEEE Transactions on Medical Imaging41(12), 3575–3586 (2022)

    Yang, H., Chen, C., Jiang, M., Liu, Q., Cao, J., Heng, P.A., Dou, Q.: DLTTA: Dynamic learning rate for test-time adaptation on cross-domain medical images. IEEE Transactions on Medical Imaging41(12), 3575–3586 (2022)

    work page 2022