REVIEW 4 major objections 5 minor 27 references
Adapting Foundation Models for Few-Shot Medical Image Segmentation: Actively and Sequentially
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper frames few-shot domain adaptation of medical segmentation models as a multi-armed bandit problem, and the resulting sequential UCB policy lifts average Dice by 27.75% on MRI and 7.52% on CT.
desk verdict Neat MAB-based auxiliary selection idea, but the abstract's headline Dice gains don't match Table 1; fixable but not citable as is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Key machinery is the banditized dataset scheduler. Let $D_a$ be one auxiliary dataset and $D_T$ the target. At turn $t$ the reward for pulling arm $a$ is $$R_{a,t}=\$\alpha$\bigl(-L(f_{\theta_t},D_a)\bigr)+(1-\$\alpha$)\frac{\nabla_a\cdot\nabla_T}{\|\nabla_a\|_2\|\nabla_T\|_2},$$ where $\nabla_a$ and $\nabla_T$ are the auxiliary and target gradients. The first term rewards convergence on the auxiliary task; the second rewards agreement in gradient direction with the target. The time-varying $\alpha$ shifts emphasis from convergence toward target alignment as training proceeds. The decision policy is upper confidence bound: each arm's smoothed mean reward $\hat R_a$ is updated with an exponential smoothing factor, and the agent selects the arm maximizing $\hat R_a+\sqrt{2\ln t/n_a}$ for arms pulled $n_a$ times, with unplayed arms given infinite upper confidence bound. This machinery does the work of converting a heuristic data-selection problem into a formally framed exploration-exploitation trade-off that uses only quantities already computed during training.
What would settle it
Before any bandit training, compute each auxiliary dataset's reward on a fresh target batch at turn 0, then run a separate full fine-tuning for each auxiliary dataset and measure target Dice. If the rank correlation between the turn-0 reward and the final Dice is near zero, the reward is not predicting transfer and the UCB selection would be no better than random.
Extended reading notes
Core claim
The paper's central claim is that auxiliary-data selection for few-shot segmentation can be solved online, during a single fine-tuning pass, with essentially no extra computation. The agent starts from a pre-trained foundation model, keeps a pool of auxiliary datasets, and at each turn pulls one arm (one auxiliary dataset) to jointly train with the few target samples. The reward for a pulled arm combines two model-intrinsic signals: the negative loss on the auxiliary data, which encourages the model to keep converging, and the cosine similarity between auxiliary and target gradients, which encourages transfer toward the target task. The bandit policy smooths reward estimates, tries every arm at least once, and then exploits high-reward arms while retaining an exploration bonus that shrinks as an arm is pulled more often. Reported results show this policy outperforming the compared alternatives on PVL and WMH (MRI) and vessel and liver (CT) targets across UNet, SwinUNet, and MambaUNet backbones, with lower training cost than mixing all auxiliary datasets at once.
Load-bearing premise
The load-bearing premise is that the handful of loss values and gradient directions observed during training reliably say which auxiliary dataset will actually help the target task; if those signals are noisy or misleading, the bandit will confidently pick the wrong auxiliary data.
Editorial extensions
If this is right
- Fine-tuning a medical foundation model with a few target slices no longer requires choosing the best auxiliary dataset in advance; a UCB schedule can make that choice online during the same training pass.
- Because the reward is read off from losses and gradients already computed in backpropagation, dynamic selection adds minimal memory and keeps training time close to direct fine-tuning, unlike all-auxiliary multi-task training.
- Auxiliary datasets that are not experientially similar to the target can still be useful early in training, since the policy can traverse a sequence of increasingly aligned datasets rather than needing one chosen at the start.
- The method transfers across convolutional, transformer, and Mamba-based UNet backbones and across MRI and CT modalities, suggesting the selection mechanism is not tied to a particular architecture.
- Adaptation can be source-free: with no access to the original source data, a pre-trained model plus public auxiliary datasets can be tailored to a low-resource target task.
Reading between the lines
- The reward is estimated from mini-batches on a handful of target slices, so the gradient-alignment term is likely high-variance; averaging the reward over several target batches before deciding would be a natural robustness check.
- Because the UCB exploration bonus assumes bounded, near-stationary rewards and the model changes every turn, the policy's exploration is better understood as a heuristic than as a provably optimal bandit schedule; normalizing rewards or using a non-stationary bandit analysis could tighten the connection.
- The same online-selection reward could apply outside segmentation, such as choosing auxiliary datasets for few-shot classification or choosing fine-tuning examples per task, since it only needs losses and gradients.
- The observed progressive alignment suggests a curriculum emerges; a direct comparison against a hand-crafted curriculum sorted by image-level similarity to the target would show whether the bandit dynamics add anything beyond a good ordering.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes ASAP, a framework for few-shot domain adaptation (FSDA) of medical image segmentation foundation models. FSDA is formulated as a multi-armed bandit over a pool of auxiliary datasets; at each turn the agent selects an auxiliary dataset, jointly trains it with the small labeled target set, and updates a reward composed of the negative auxiliary loss and the gradient cosine similarity between auxiliary and target. The policy is a UCB-style rule with an exponentially smoothed reward estimate. Experiments compare with direct fine-tuning, GMS, MTL, and DAL on two MRI targets (PVL, WMH) and two CT targets (vessel, liver) using three backbones (UNet, SwinUNet, MambaUNet), reporting Dice and mIoU, and the abstract claims average gains of 27.75% on MRI and 7.52% on CT.
Significance. If the central numbers and theoretical framing were corrected, the contribution would be a practical and lightweight auxiliary-selection mechanism: it is source-free, requires no external model, is implemented on three backbones, and achieves the best Dice score in all 12 rows of Table 1. The code is promised publicly. However, the current abstract overstates the gains, the UCB guarantee does not apply as stated, and the absence of uncertainty quantification makes the magnitude of the improvements unclear. These are load-bearing issues that affect the paper's main claims.
major comments (4)
- [Abstract; Table 1] The abstract's headline 'average gain of 27.75% on MRI and 7.52% on CT datasets in Dice score' is not supported by Table 1 unless the baseline and averaging are specified. Recomputing relative Dice improvements over the best FSDA baseline per row gives approximately 6.3% average on MRI and 3.4% on CT; recomputing over direct fine-tuning gives approximately 14.5% on MRI and 7.7% on CT. The CT value is close to the direct-fine-tuning average, but no definition reproduces the MRI value of 27.75%. Because this number is the central quantitative evidence for 'significantly outperforming' the baselines, the authors should state the exact baseline, averaging scheme, and recompute all reported gains, then adjust the abstract and conclusion accordingly.
- [§2.3, Eq. (5)] Equation (5) is justified by citing Hoeffding's inequality, but the rewards are neither bounded nor stationary. R_PM in Eq. (1) is a negative loss, unbounded below, and the model parameters are updated at every turn, so each arm's reward distribution changes over time; moreover, Eq. (4) maintains an exponential moving average rather than a sample mean. The exploration bonus sqrt(2 ln t / n_a) therefore does not follow from the cited finite-time analysis, and the UCB policy has no theoretical regret guarantee in this setting. The authors should either derive an appropriate bound under weaker assumptions, or explicitly present the UCB rule as a heuristic and support it with ablations on the exploration term.
- [§3.1, Algorithm 1] Algorithm 1 depends on hyperparameters that are never reported: the decay schedule of alpha, the smoothing factor beta, and the number of bandit turns N. These determine the reward mixing and the exploration-exploitation balance, so the experiments cannot be reproduced without them. The paper should give the values or ranges, and ideally a sensitivity analysis.
- [§3.2, Table 1] Section 3.2 reports means without error bars, confidence intervals, or significance tests in an extreme 3-shot, 5-fold setting. Several reported advantages are very small relative to expected sampling noise, e.g., the WMH/SwinUNet gap over DAL is 0.46 Dice points (67.62 vs. 67.16). The authors should report per-fold mean and standard deviation and perform paired significance tests (or clearly state the comparisons are descriptive). This is important because the central claim is that ASAP 'significantly outperforms' strong baselines.
minor comments (5)
- [§3.2] The word 'simutaneously' should be 'simultaneously'.
- [§5] The ethical statement says 'animal subject data', but the datasets used are human brain and CT imaging datasets; this should be corrected.
- [Abstract; §6] The abstract gives the repository URL, but the full text says only 'Code is available at ASAP'; include the URL in the full text.
- [Algorithm 1] Line 3 initializes all n_a=1 and uses a 0.5/0.5 weighting, while Eq. (5) has an infinite-UCB case for n_a=0 and Eq. (3) uses a time-varying alpha. Clarify the initialization convention and why a different mixing weight is used.
- [Algorithm 1] In Algorithm 1, gradients for the reward are computed before the parameter update (lines 9-13); clarify whether R_{a,t} is meant to measure the effect of the update or the state before it.
Circularity Check
No significant circularity: the ASAP reward is a hand-designed policy objective evaluated against external baselines; no prediction reduces to a fitted input or to a self-citation chain.
full rationale
The paper's core procedure is a multi-armed bandit over auxiliary datasets. The reward (Eq. 3) is a designed combination of the negative auxiliary loss (Eq. 1) and the gradient cosine similarity with the target (Eq. 2). This is an objective that generates a selection policy, not a quantity fitted to the reported segmentation scores; the final evaluation is on held-out target test data against external baselines (Direct FT, GMS, MTL, DAL). No equation in Sec. 2 defines the target outcome in terms of the reward, and no reported Dice gain is a fitted parameter renamed as a prediction. The self-citations ([10] supplies the PVL dataset; [15] supplies the Flemme pretraining platform) are data and infrastructure citations and are not load-bearing in the derivation of the selection rule. The Hoeffding-based UCB formula (Eq. 5) is applied to rewards that are unbounded and nonstationary, which is a theoretical-validity concern rather than a circularity: the exploration bonus is not doing the claimed statistical work, but the method's output is not equivalent to its input by construction. Therefore no circular step meeting the quoted-evidence bar is present; the score reflects only minor non-load-bearing self-citations.
Assumptions & free parameters
free parameters (5)
- alpha (reward mixing weight) =
not specified
- beta (UCB smoothing factor) =
not specified
- N (number of bandit turns) =
not specified
- target sample count (3-shot) =
3
- auxiliary dataset pool composition =
30 datasets from FeTS/iSeg (MRI) and TotalSegmentator (CT)
assumptions (4)
- domain assumption UCB1 regret bounds require bounded, stationary rewards; the paper's reward is unbounded and nonstationary.
- domain assumption Gradient alignment is a reliable proxy for transferability.
- ad hoc to paper The reward schedule alpha decays with turns, and beta is a fixed smoothing factor; both are assumed to yield stable learning.
- domain assumption It is valid to apply the MAB framework to the FSDA setting with a changing model.
Cite this review
Pith. "Pith review of Adapting Foundation Models for Few-Shot Medical Image Segmentation: Actively and Sequentially." pith.science (2026). https://pith.science/paper/27MM6S5D
@misc{pith2026250201000,
author = {Pith},
title = {Pith review of: Adapting Foundation Models for Few-Shot Medical Image Segmentation: Actively and Sequentially},
year = {2026},
howpublished = {\url{https://pith.science/paper/27MM6S5D}},
note = {Machine review of arXiv:2502.01000}
}
read the original abstract
Recent advances in foundation models have brought promising results in computer vision, including medical image segmentation. Fine-tuning foundation models on specific low-resource medical tasks has become a standard practice. However, ensuring reliable and robust model adaptation when the target task has a large domain gap and few annotated samples remains a challenge. Previous few-shot domain adaptation (FSDA) methods seek to bridge the distribution gap between source and target domains by utilizing auxiliary data. The selection and scheduling of auxiliaries are often based on heuristics, which can easily cause negative transfer. In this work, we propose an Active and Sequential domain AdaPtation (ASAP) framework for dynamic auxiliary dataset selection in FSDA. We formulate FSDA as a multi-armed bandit problem and derive an efficient reward function to prioritize training on auxiliary datasets that align closely with the target task, through a single-round fine-tuning. Empirical validation on diverse medical segmentation datasets demonstrates that our method achieves favorable segmentation performance, significantly outperforming the state-of-the-art FSDA methods, achieving an average gain of 27.75% on MRI and 7.52% on CT datasets in Dice score. Code is available at the git repository: https://github.com/techicoco/ASAP.
Figures
Reference graph
Works this paper leans on
-
[11]
Unsupervised domain adaptation with variational approximation for cardiac segmentation,
Fuping Wu and Xiahai Zhuang, “Unsupervised domain adaptation with variational approximation for cardiac segmentation,” IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3555–3567, 2021
work page 2021
-
[1]
INTRODUCTION Recent works like SwinUNet [1], MambaUNet [2] and MONAI [3] develop medical-tailored foundation models on large-scale medical image datasets. Intense interest has emerged in adapting these foundation models for specific medical image analysis tasks. However, the generalization capability of foundation models is limited by the large vari- abil...
arXiv 2025
-
[2]
METHODOLOGY In this section, we will elaborate on the proposed active and sequential domain adaptation (ASAP) framework, shown in Fig. 1. First, we clarify the setting of few-shot domain adap- tation with auxiliary datasets. Then we formulate it as a multi- armed bandit (MAB) problem and describe how we solve it. 2.1. Problem definition For domain adaptat...
-
[3]
EXPERIMENTS AND RESULTS To showcase the flexibility of our ASAP framework, we con- duct extensive experiments on MRI and CT datasets covering various modalities and anatomical regions. 3.1. Datasets and Implementation Details For MRI experiments, we construct the auxiliary datasets pool based on FeTS 2022 [17] (brain tumor segmentation) and iSeg2019 [18] ...
work page 2022
-
[4]
CONCLUSION We propose a novel active and sequential domain adaptation (ASAP) framework to adapt foundation models for the few- shot medical image segmentation tasks. With our desiderata in mind, the proposed ASAP achieve: 1) no requirement for access to the source domain or a substantial amount of target data, 2) incorporation of auxiliary data with dynam...
-
[5]
Ethical approval was not required as confirmed by the license attached with the open access data
COMPLIANCE WITH ETHICAL STANDARDS This research study was conducted retrospectively using the animal subject data made available in open access by [17–21]. Ethical approval was not required as confirmed by the license attached with the open access data
-
[6]
Yanyang Gu, Zongyuan Ge, C Paul Bonnington, and Jun Zhou, “Progressive transfer learning and adversarial do- main adaptation for cross-domain skin disease classifi- cation,” IEEE journal of biomedical and health infor- matics, vol. 24, no. 5, pp. 1379–1393, 2019
work page 2019
-
[7]
Swin- unet: Unet-like pure transformer for medical image seg- mentation,
Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xi- aopeng Zhang, Qi Tian, and Manning Wang, “Swin- unet: Unet-like pure transformer for medical image seg- mentation,” in European conference on computer vision. Springer, 2022, pp. 205–218
2022
Show all 27 references
-
[8]
Mamba-unet: Unet-like pure visual mamba for medical image segmentation,
Ziyang Wang, Jian-Qing Zheng, Yichi Zhang, Ge Cui, and Lei Li, “Mamba-unet: Unet-like pure visual mamba for medical image segmentation,” arXiv preprint arXiv:2402.05079, 2024
2024 arXiv
-
[9]
Monai: An open-source framework for deep learning in healthcare,
M Jorge Cardoso, Wenqi Li, Richard Brown, Nic Ma, Eric Kerfoot, Yiheng Wang, Benjamin Murrey, Andriy Myronenko, Can Zhao, Dong Yang, et al., “Monai: An open-source framework for deep learning in healthcare,” arXiv preprint arXiv:2211.02701, 2022
2022 arXiv
-
[10]
A domain-adaptive u- net for electron microscopy image segmentation,
R ´oger Berm´udez-Chac´on, Pablo M´arquez-Neila, Math- ieu Salzmann, and Pascal Fua, “A domain-adaptive u- net for electron microscopy image segmentation,” in 2018 IEEE 15th International Symposium on Biomedi- cal Imaging (ISBI 2018). IEEE, 2018, pp. 400–404
2018
-
[12]
Ban- dit problems and the exploration/exploitation tradeoff,
William G Macready and David H Wolpert, “Ban- dit problems and the exploration/exploitation tradeoff,” IEEE Transactions on evolutionary computation, vol. 2, no. 1, pp. 2–22, 1998
1998
-
[13]
(5) This allows us to balance the exploitation of arms with a high predicted reward and the exploration of areas with high un- certainty
for arm a at turn t being played na times: U CBa,t = ( ∞, if na = 0 ˆRa + q 2 lnt na , otherwise. (5) This allows us to balance the exploitation of arms with a high predicted reward and the exploration of areas with high un- certainty. The proposed algorithm is shown in Algori...
-
[14]
Auxiliary learning by im- plicit differentiation,
Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya, “Auxiliary learning by im- plicit differentiation,” arXiv preprint arXiv:2007.02693, 2021
2007 arXiv
-
[15]
One model is all you need: multi-task learning enables simultaneous histology image segmen- tation and classification,
Simon Graham, Quoc Dang Vu, Mostafa Jahanifar, Shan E Ahmed Raza, Fayyaz Minhas, David Snead, and Nasir Rajpoot, “One model is all you need: multi-task learning enables simultaneous histology image segmen- tation and classification,” Medical Image Analysis, vol. 83, pp. 102685, 2023
2023
-
[16]
Improving few-shot generalization by exploring and exploiting auxiliary data,
Alon Albalak, Colin A Raffel, and William Yang Wang, “Improving few-shot generalization by exploring and exploiting auxiliary data,” Advances in Neural Infor- mation Processing Systems, vol. 36, 2024
2024
-
[17]
Joint pvl detection and manual ability classification using semi-supervised multi-task learning,
Jingyun Yang, Jie Hu, Yicong Li, Heng Liu, and Yang Li, “Joint pvl detection and manual ability classification using semi-supervised multi-task learning,” in Medical Image Computing and Computer Assisted Intervention– MICCAI 2021: 24th International Conference, Stras- bourg, F...
2021
-
[18]
Gradient surgery for multi-task learning,
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn, “Gradient surgery for multi-task learning,” Advances in Neural In- formation Processing Systems, vol. 33, pp. 5824–5836, 2020
2020
-
[19]
Finite-time analysis of the multiarmed bandit prob- lem,
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer, “Finite-time analysis of the multiarmed bandit prob- lem,” Machine learning, vol. 47, pp. 235–256, 2002
2002
-
[20]
U-net: Convolutional networks for biomedical im- age segmentation,
Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-net: Convolutional networks for biomedical im- age segmentation,” in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th in- ternational conference, October 5-9, 2015, proceedings, part III 18. Spr...
2015
-
[21]
Flemme: A flexible and modular learning platform for medical images,
Guoqing Zhang, Jingyun Yang, and Yang Li, “Flemme: A flexible and modular learning platform for medical images,” in 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) . IEEE, 2024, pp. 4018–4023
2024
-
[22]
Nonstationary stochastic multiarmed bandits: Ucb policies and mini- max regret,
Lai Wei and Vaibhav Srivastava, “Nonstationary stochastic multiarmed bandits: Ucb policies and mini- max regret,” arXiv preprint arXiv:2101.08980, 2021
2021 arXiv
-
[23]
The federated tumor segmentation (fets) chal- lenge,
Sarthak Pati, Ujjwal Baid, Maximilian Zenk, Brandon Edwards, Micah Sheller, G Anthony Reina, Patrick Fo- ley, Alexey Gruzdev, Jason Martin, Shadi Albarqouni, et al., “The federated tumor segmentation (fets) chal- lenge,” arXiv preprint arXiv:2105.05874, 2021
2021 arXiv
-
[24]
Multi-site infant brain seg- mentation algorithms: The iseg-2019 challenge,
Yue Sun, Kun Gao, Zhengwang Wu, Guannan Li, Xi- aopeng Zong, Zhihao Lei, Ying Wei, Jun Ma, Xiaoping Yang, Xue Feng, et al., “Multi-site infant brain seg- mentation algorithms: The iseg-2019 challenge,” IEEE Transactions on Medical Imaging , vol. 40, no. 5, pp. 1363–1376, 2021
2019
-
[25]
Standardized assessment of au- tomatic segmentation of white matter hyperintensities and results of the wmh segmentation challenge,
Hugo J Kuijf, J Matthijs Biesbroek, Jeroen De Bresser, Rutger Heinen, Simon Andermatt, Mariana Bento, Matt Berseth, Mikhail Belyaev, M Jorge Cardoso, Adria Casamitjana, et al., “Standardized assessment of au- tomatic segmentation of white matter hyperintensities and results of...
2019
-
[26]
Totalsegmentator: robust segmentation of 104 anatomic structures in ct images,
Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al., “Totalsegmentator: robust segmentation of 104 anatomic structures in ct images,” Radiology: Arti- ficial I...
2023
-
[27]
The medical seg- mentation decathlon,
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ron- neberger, Ronald M Summers, et al., “The medical seg- mentation decathlon,” Nature communications, vol. 13, no. 1, pp. 4128, 2022
2022
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.