REVIEW 4 major objections 4 minor 1 cited by
Transfer Risk Map: Mitigating Pixel-level Negative Transfer in Medical Segmentation
T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read A per-pixel transfer risk map can reduce negative transfer in medical segmentation fine-tuning, with gains of 4.37% on FeTS 2021 and 1.81% on iSeg-2019.
desk verdict Incremental pixel-weighted fine-tuning that mostly works in tables but lacks significance testing and ignores a spatial-alignment issue. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the transfer risk map $w$, a fixed spatial weight matrix computed once from pixel-level LEEP scores on the target dataset. Each entry $w_{j,k} = 10^{t^s_{j,k}}$ turns a min-max normalized LEEP hardness score into an upweighting factor for the loss at coordinate $(j,k)$, and the exponential base 10 makes the emphasis on hard pixels nonlinear. The map is applied to every training image at the same coordinates, and the reweighted per-pixel loss is averaged over foreground pixels to counter class imbalance. This one map is what carries the claimed effect: it concentrates fine-tuning pressure on the regions where transfer is riskiest.
What would settle it
Run the same transfer fine-tuning on a brain MRI dataset whose slices from different subjects are randomly rotated, shifted, or resampled before training, keeping the labels in the same coordinate frame; if the reported Dice gains disappear or reverse, the central claim depends on the unstated alignment assumption.
Extended reading notes
Core claim
The central discovery claimed is that pixel-level transferability, measured by adapting LEEP (Log Expected Empirical Prediction), a transferability score for classification, to dense prediction, predicts where negative transfer will hurt during fine-tuning, and that weighting the loss at exactly those pixels improves target performance. Concretely, the paper builds a single transfer risk map $w$ over image coordinates from min-max normalized pixel-level LEEP scores, scaled by $10^{(\cdot)}$ to emphasize high-risk pixels, then fine-tunes a frozen-encoder U-Net with the loss $\sum w_{j,k}\,\ell_{j,k}$ averaged over foreground pixels only. Across cross-modality and cross-task brain MRI experiments, the weighted fine-tuning beats vanilla fine-tuning, class-weighted fine-tuning, and a prior transferability-map baseline, with average Dice gains of 4.37% on FeTS 2021 and 1.81% on iSeg-2019, and a 2.9% gain with as few as 50 target slices.
Load-bearing premise
The fixed per-coordinate transfer risk map assumes that the 2D slices used for training are spatially aligned across subjects, because the same pixel coordinate is treated as the same anatomical location for every image; the paper describes no registration or alignment step.
Editorial extensions
If this is right
- If the central claim is correct, cross-modality and cross-task transfer in medical segmentation can be improved by reweighting the loss alone, with no access to source data.
- The weighting scheme should remain useful in few-shot regimes, since the paper reports a 2.9% average Dice improvement even when the target has only 50 labeled slices.
- Because the method lives entirely in the loss, it can be combined with any segmentation architecture, including the vision transformer extension the authors mention.
- Averaging the reweighted loss over foreground pixels only prevents a large well-learned background from diluting the loss, which matters for small lesions and fine structures.
Reading between the lines
- The paper leaves implicit that its fixed coordinate map presumes anatomical alignment across subjects; a natural extension would be to compute per-image risk maps or add a registration step before building the map.
- A straightforward testable follow-up is to update the transfer risk map periodically during fine-tuning rather than once at the start, since the paper only compares maps before and after training.
- The same recipe could apply to other dense prediction tasks, such as depth estimation or panoptic segmentation, wherever a pixel-wise transferability score can be defined, though the paper does not make this claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a transferability-guided weighting scheme for fine-tuning medical image segmentation models. A pixel-level transfer risk map is computed using a segmentation-adapted LEEP score evaluated on the target training data, and this map is used to reweight the per-pixel loss during fine-tuning, with normalization by foreground size to mitigate class imbalance. Experiments on two brain MRI datasets (FeTS 2021 and iSeg-2019) report consistent Dice improvements over vanilla fine-tuning, class-weighted fine-tuning, and a transferability-map baseline, along with improvements in a few-shot setting. The central claim is that the method avoids negative transfer across modalities and tasks.
Significance. If substantiated, the proposed method would be a simple and practical addition to the transfer-learning toolbox for medical segmentation: it requires only a pre-trained source model and labeled target data, and it operates at the pixel level where negative transfer is plausibly non-uniform. The use of LEEP on a per-pixel basis is a natural extension of existing transferability metrics, and the reported gains are consistent across many transfer pairs. However, the evidence currently falls short of the paper's own claims: there is no from-scratch baseline, no statistical significance testing, and a load-bearing spatial-alignment assumption is never stated or verified. The method's utility as a mechanism for avoiding negative transfer therefore remains unproven, although the core idea is defensible and testable.
major comments (4)
- [Section III-C, Tables I and III; Abstract/Conclusion] The central claim that the method 'indeed avoids negative transfer' (Abstract, line 6 and Conclusion) is not supported by the reported experiments. Negative transfer is defined relative to training without transfer (i.e., from scratch), yet all comparisons in Tables I, II, and III are between fine-tuning variants. A from-scratch baseline trained on the target data is necessary to establish whether the proposed weighting actually prevents the source model from hurting target performance, as opposed to merely improving a fine-tuning procedure.
- [Section II-C, Equations (4)-(7)] The transfer risk map w is a single W x H mask computed once over all target images at fixed coordinates (j,k) and then applied to every training image in Eq. (7). This assumes that pixel coordinate (j,k) corresponds to the same anatomical structure across subjects. No registration, atlas resampling, or slice-selection procedure is described in Section III-B, and for unaligned brain MRI volumes this assumption is unlikely to hold. Without an alignment step or a demonstration that the mask is anatomically consistent, the weighting may emphasize different structures in different images, and the reported gains cannot be confidently attributed to pixel-level transfer risk.
- [Section III-B and Tables I-III] The paper provides no error bars, significance tests, or a description of the train/test split. It is not stated how the target data are divided (e.g., patient-level split, which 2D slices are used for training vs. testing, validation procedure for early stopping), nor how many random seeds the reported Dice scores are averaged over. Given that the text repeatedly describes improvements as 'significantly enhanced' (e.g., Section III-C), the absence of any statistical support is a serious gap that prevents evaluation of whether the observed gains are reliable.
- [Section II-B, Equations (2)-(4)] The pixel-level LEEP computation is underspecified. The text says the global feature vector is decomposed into pixel-wise features, but it does not state which network layer is used, how feature resolution is matched to the input resolution (given the U-Net's downsampling), or how the empirical conditional distribution P(y_{j,k} | z) is estimated per pixel. Without these details, the transfer risk map in Eq. (4) is not reproducible, and it is unclear whether the map truly reflects per-pixel transferability as opposed to some artifact of feature-map interpolation.
minor comments (4)
- [Section II-C, Eq. (6)] The exponential scaling base 10 is introduced without any ablation or justification; the choice of base strongly affects the dynamic range of the weights, and a sensitivity analysis or rationale should be provided.
- [Section III-D, Table III] The text states 'a 2.4% improvement and a 2.9% gain in Dice score in average' for the 50-sample case, but the table shows a relative improvement of approximately 2.85% for that row; the wording is inconsistent and should be clarified.
- [Section II-C, Eq. (7)] The loss is averaged over foreground pixels only, but the numerator sums over all pixels; this means background pixels contribute to the loss value in the numerator but not to the denominator. The authors should clarify whether this is intentional and discuss the effect on gradients, since the background loss is still backpropagated despite being excluded from the normalization.
- [Section III-C, Tables I and II] The reported 'overall gain' percentages appear to be relative improvements over vanilla fine-tuning (e.g., 4.37% for FeTS 2021), but this is not stated explicitly; the text should define how these average gains are computed.
Circularity Check
The central Dice-gain result is not circular, but the before/after transfer-risk validation in Sec. III-E reduces to a supervised training-progress measure because LEEP is computed and refit on the same target labels.
-
fitted input called prediction
[Section III-E ('Effectiveness of Negative Transfer Mitigation'), Fig. 2; Eqs. (2)-(6)]
"It is evident that our fine-tuning approach has effectively condensed the extensive regions of high transfer risk into well-defined, smaller segments, concurrently reducing the transfer hardness."
The risk map w comes from Eq. (4): t_{j,k}=LEEP(θ, {x^{j,k}_i, y^{j,k}_i}) on the target set D, and LEEP in Eq. (2) builds P̂(y|z) from those same target labels. After fine-tuning on D, θ's likelihood on D rises, so LEEP-based 'transfer hardness' falls at least partly because the metric is refit to the labels used in training, not because an independent negative-transfer effect was removed. The before/after visualization is thus not an independent validation; it is a supervised training-progress measure presented as evidence of negative-transfer mitigation.
full rationale
The main quantitative claims (4.37% Dice gain on FeTS2021, 1.81% on iSeg-2019, 2.9% few-shot gain) compare fine-tuning variants on public benchmarks and do not reduce to the risk-map construction by definition; the risk map is a fixed supervised weight computed before training, not a prediction of the final Dice. The method's dependence on LEEP is externally grounded in [18], so the self-citation [22] (authors' own prior work) is used for comparison and for choosing LEEP but is not the load-bearing derivation. However, Section III-E presents the pre/post fine-tuning transfer-risk maps as validation of negative-transfer mitigation; because LEEP uses the empirical conditional distribution estimated on the same target labels that are used to fine-tune the model, the observed decrease in 'transfer hardness' is largely a consequence of the metric being fitted to the training labels. This is a partial circularity in the supporting argument, though the independent benchmark results keep the central claim from being fully circular. The lack of an explicit train/test split and the unstated assumption that fixed pixel coordinates correspond to the same anatomy across slices are validity risks, not circularity.
Assumptions & free parameters
free parameters (1)
- Exponential scaling base =
10
assumptions (4)
- domain assumption LEEP score computed on target data with source model predictions is a valid per-pixel measure of negative transfer risk.
- ad hoc to paper Pixel coordinates are spatially aligned across target images so a single risk map can be applied to every image.
- domain assumption The source-free setting permits estimating LEEP's empirical conditional distribution from target data only.
- domain assumption Foreground-only normalization of the loss is a valid class-imbalance correction and does not distort the training objective.
invented entities (1)
-
Transfer risk map
Cite this review
Pith. "Pith review of Transfer Risk Map: Mitigating Pixel-level Negative Transfer in Medical Segmentation." pith.science (2026). https://pith.science/paper/GSDNLCOO
@misc{pith2026250202340,
author = {Pith},
title = {Pith review of: Transfer Risk Map: Mitigating Pixel-level Negative Transfer in Medical Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/GSDNLCOO}},
note = {Machine review of arXiv:2502.02340}
}
read the original abstract
How to mitigate negative transfer in transfer learning is a long-standing and challenging issue, especially in the application of medical image segmentation. Existing methods for reducing negative transfer focus on classification or regression tasks, ignoring the non-uniform negative transfer risk in different image regions. In this work, we propose a simple yet effective weighted fine-tuning method that directs the model's attention towards regions with significant transfer risk for medical semantic segmentation. Specifically, we compute a transferability-guided transfer risk map to quantify the transfer hardness for each pixel and the potential risks of negative transfer. During the fine-tuning phase, we introduce a map-weighted loss function, normalized with image foreground size to counter class imbalance. Extensive experiments on brain segmentation datasets show our method significantly improves the target task performance, with gains of 4.37% on FeTS2021 and 1.81% on iSeg2019, avoiding negative transfer across modalities and tasks. Meanwhile, a 2.9% gain under a few-shot scenario validates the robustness of our approach.
Figures
Forward citations
Cited by 1 Pith paper
-
Understanding Knowledge Transferability for Transfer Learning: A Survey
A survey that classifies transferability metrics by knowledge modality (dataset vs. model) and granularity (task vs. instance), with a theoretical primer and applications to eight learning paradigms.
Reference graph
Works this paper leans on
-
[22]
Efficient prediction of model transferability in semantic segmentation tasks,
Y . Tan, Y . Li, Y . Li, and X.-P. Zhang, “Efficient prediction of model transferability in semantic segmentation tasks,” in 2023 IEEE Interna- tional Conference on Image Processing (ICIP) , 2023, pp. 720–724
work page 2023
-
[1]
Medical image segmentation using deep learning: A survey,
R. Wang, T. Lei, R. Cui, B. Zhang, H. Meng, and A. K. Nandi, “Medical image segmentation using deep learning: A survey,”IET image processing, vol. 16, no. 5, pp. 1243–1267, 2022
work page 2022
-
[2]
Deep learning tech- niques for medical image segmentation: achievements and challenges,
M. H. Hesamian, W. Jia, X. He, and P. Kennedy, “Deep learning tech- niques for medical image segmentation: achievements and challenges,” Journal of digital imaging , vol. 32, pp. 582–596, 2019
work page 2019
-
[3]
Transfer learning techniques for medical image analysis: A review,
P. Kora, C. P. Ooi, O. Faust, U. Raghavendra, A. Gudigar, W. Y . Chan, K. Meenakshi, K. Swaraja, P. Plawiak, and U. R. Acharya, “Transfer learning techniques for medical image analysis: A review,” Biocybernetics and Biomedical Engineering , vol. 42, no. 1, pp. 79–107, 2022
work page 2022
-
[4]
D. Karimi, S. K. Warfield, and A. Gholipour, “Transfer learning in med- ical image segmentation: New insights from analysis of the dynamics of model parameters and learned representations,” Artificial intelligence in medicine, vol. 116, p. 102078, 2021
work page 2021
-
[5]
A survey on transfer learning,
S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans- actions on knowledge and data engineering , vol. 22, no. 10, pp. 1345– 1359, 2009
2009
-
[6]
Characterizing and avoiding negative transfer,
Z. Wang, Z. Dai, B. Poczos, and J. Carbonell, “Characterizing and avoiding negative transfer,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019
work page 2019
-
[7]
Domain adaptation for medical image analysis: a survey,
H. Guan and M. Liu, “Domain adaptation for medical image analysis: a survey,” IEEE Transactions on Biomedical Engineering , vol. 69, no. 3, pp. 1173–1185, 2021
2021
Show all 29 references
-
[8]
A survey on negative transfer,
W. Zhang, L. Deng, L. Zhang, and D. Wu, “A survey on negative transfer,” IEEE/CAA Journal of Automatica Sinica , vol. 10, no. 2, pp. 305–329, 2023
2023
-
[9]
Finding the most trans- ferable tasks for brain image segmentation,
Y . Li, Y . Tan, J. Yang, Y . Li, and X.-P. Zhang, “Finding the most trans- ferable tasks for brain image segmentation,” in 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) . IEEE, 2022, pp. 1620–1625
2022
-
[10]
A cross-dataset adaptive domain selection transfer learning framework for motor imagery-based brain-computer interfaces,
J. Jin, G. Bai, R. Xu, K. Qin, H. Sun, X. Wang, and A. Cichocki, “A cross-dataset adaptive domain selection transfer learning framework for motor imagery-based brain-computer interfaces,” Journal of Neural Engineering, vol. 21, no. 3, p. 036057, 2024
2024
-
[11]
Non-negative transfer learning with consistent inter-domain distribution,
Z. Peng, Y . Jia, and J. Hou, “Non-negative transfer learning with consistent inter-domain distribution,” IEEE Signal Processing Letters , vol. 27, pp. 1720–1724, 2020
2020
-
[12]
Dual transfer learning,
M. Long, J. Wang, G. Ding, W. Cheng, X. Zhang, and W. Wang, “Dual transfer learning,” in Proceedings of the 2012 SIAM International Conference on Data Mining . SIAM, 2012, pp. 540–551
2012
-
[13]
A kernel two-sample test,
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Sch ¨olkopf, and A. Smola, “A kernel two-sample test,” The Journal of Machine Learning Research, vol. 13, no. 1, pp. 723–773, 2012
2012
-
[14]
On information and sufficiency,
S. Kullback and R. A. Leibler, “On information and sufficiency,” The annals of mathematical statistics , vol. 22, no. 1, pp. 79–86, 1951
1951
-
[15]
Improving eeg-based emotion classification using conditional transfer learning,
Y .-P. Lin and T.-P. Jung, “Improving eeg-based emotion classification using conditional transfer learning,” Frontiers in human neuroscience , vol. 11, p. 334, 2017
2017
-
[16]
Transferability and hardness of supervised classification tasks,
A. T. Tran, C. V . Nguyen, and T. Hassner, “Transferability and hardness of supervised classification tasks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 1395–1405
2019
-
[17]
An information-theoretic approach to transferability in task transfer learning,
Y . Bao, Y . Li, S.-L. Huang, L. Zhang, L. Zheng, A. Zamir, and L. Guibas, “An information-theoretic approach to transferability in task transfer learning,” in 2019 IEEE international conference on image processing (ICIP). IEEE, 2019, pp. 2309–2313
2019
-
[18]
Leep: A new measure to evaluate transferability of learned representations,
C. Nguyen, T. Hassner, M. Seeger, and C. Archambeau, “Leep: A new measure to evaluate transferability of learned representations,” in International Conference on Machine Learning . PMLR, 2020, pp. 7294–7305
2020
-
[19]
Logme: Practical assessment of pre-trained models for transfer learning,
K. You, Y . Liu, J. Wang, and M. Long, “Logme: Practical assessment of pre-trained models for transfer learning,” in International Conference on Machine Learning . PMLR, 2021, pp. 12 133–12 143
2021
-
[20]
Otce: A transferability metric for cross- domain cross-task representations,
Y . Tan, Y . Li, and S.-L. Huang, “Otce: A transferability metric for cross- domain cross-task representations,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 15 779–15 788
2021
-
[21]
Where and how to transfer: Knowledge aggregation-induced transferability perception for unsupervised domain adaptation,
J. Dong, Y . Cong, G. Sun, Z. Fang, and Z. Ding, “Where and how to transfer: Knowledge aggregation-induced transferability perception for unsupervised domain adaptation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 3, pp. 1664–1681, 2024
2024
-
[23]
Deep semantic segmentation of natural and medical images: a review,
S. Asgari Taghanaki, K. Abhishek, J. P. Cohen, J. Cohen-Adad, and G. Hamarneh, “Deep semantic segmentation of natural and medical images: a review,” Artificial Intelligence Review , vol. 54, pp. 137–178, 2021
2021
-
[24]
The federated tumor segmentation (fets) challenge,
S. Pati, U. Baid, M. Zenk, B. Edwards, M. Sheller, G. A. Reina, P. Foley, A. Gruzdev, J. Martin, S. Albarqouni et al. , “The federated tumor segmentation (fets) challenge,” arXiv preprint arXiv:2105.05874, 2021
2021 arXiv
-
[25]
Openfl: An open-source framework for federated learning,
G. A. Reina, A. Gruzdev, P. Foley, O. Perepelkina, M. Sharma, I. Davidyuk, I. Trushkin, M. Radionov, A. Mokrov, D. Agapov et al. , “Openfl: An open-source framework for federated learning,” arXiv preprint arXiv:2105.06413, 2021
2021 arXiv
-
[26]
Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,
S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, “Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,” Scientific data, vol. 4, no. 1, pp. 1–13, 2017
2017
-
[27]
Multi-site infant brain segmentation algorithms: the iseg-2019 challenge,
Y . Sun, K. Gao, Z. Wu, G. Li, X. Zong, Z. Lei, Y . Wei, J. Ma, X. Yang, X. Feng et al. , “Multi-site infant brain segmentation algorithms: the iseg-2019 challenge,” IEEE Transactions on Medical Imaging , vol. 40, no. 5, pp. 1363–1376, 2021
2019
-
[28]
U-net: Convolutional networks for biomedical image segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 , N. Navab, J. Horneg- ger, W. M. Wells, and A. F. Frangi, Eds. Cham: Springer International ...
2015
-
[29]
Transfer learning with class-weighted and focal loss function for automatic skin cancer classification,
D. N. Le, H. X. Le, L. T. Ngo, and H. T. Ngo, “Transfer learning with class-weighted and focal loss function for automatic skin cancer classification,” arXiv preprint arXiv:2009.05977 , 2020
2009 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.