REVIEW 3 major objections 4 minor 30 references
AdvMIM: Adversarial Masked Image Modeling for Semi-Supervised Medical Image Segmentation
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper proposes that semi-supervised medical segmentation with vision transformers fails mainly because the transformer receives too little supervision, and that an auxiliary masked-image domain supplies the missing signal: the network…
desk verdict A genuinely new masked-domain adversarial training recipe for semi-supervised segmentation, with a headline ACDC gain that rests on an uncontrolled cross-paper baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the masked domain: an image modified by the masking operation of masked image modeling with ratio $\rho=0.7$, where masked patches are replaced by shared learnable mask tokens plus positional embeddings, and the transformer must predict the entire segmentation map rather than reconstruct masked pixels. This domain generates extra training signal from both true labels and pseudo-labels, and it is kept useful by a second mechanism: a domain discriminator $D_s$ that takes prediction masks and classifies whether they came from original labeled inputs or masked unlabeled inputs, trained with the least-squares GAN objective while the segmentation network tries to fool it. Theorem 1 ties the two together, bounding original-domain error by a weighted sum of the two supervised errors, the domain discrepancy $d_{H\Delta H}(P,Q')$, the optimal joint error $\lambda$, and the pseudo-label noise ratio $\gamma$; the adversarial loss is the term that explicitly minimizes the discrepancy.
What would settle it
Run the exact framework with mask ratio 0, so the masked input is identical to the original input and the auxiliary domain carries no new information; if ACDC 3% Dice stays near 85.4 rather than falling to the cross-teaching-only level, then masked-domain supervision is not the cause of the gain. Alternatively, record the discriminator's accuracy at convergence: if it remains near chance after adversarial training while performance still changes, the domain-gap term of Theorem 1 is not doing the predicted work.
Extended reading notes
Core claim
Stated on the paper's own terms, the discovery is that masked image modeling need not reconstruct pixels; asking a transformer to output the whole segmentation mask from a heavily masked image is itself a powerful semi-supervised task, because it forces the network to synthesize anatomy from context and thereby yields supervision at every masked location. The paper further claims that this auxiliary domain is useful only if its distribution is aligned with the original one, and quantifies the alignment with Theorem 1: for any hypothesis and any pseudo-label noise ratio $\gamma$, the original-domain error is at most $\frac{1}{2}\epsilon_{P'}(h) + \frac{1}{2}\epsilon_{Q'}(h) + \frac{1}{4}d_{H\Delta H}(P,Q') + \frac{1}{2}\lambda + \gamma$. The two per-domain loss terms are minimized by the segmentation and masked-domain losses, the noise term by confidence-weighted pseudo-labels, and the domain-gap term by a least-squares adversarial loss on the prediction masks. Empirically the paper reports state-of-the-art numbers on three public datasets, e.g. 85.4 Dice on ACDC with 3% labeled data.
Load-bearing premise
The load-bearing premise is that predicting a full mask from a heavily masked input teaches the network genuine organ-boundary knowledge that transfers to unmasked images, rather than teaching it to produce plausible context-driven guesses; the theoretical bound also assumes the unavoidable joint error $\lambda$ is small for the trained Swin-UNet, which the paper does not measure.
Editorial extensions
If this is right
- Masked image modeling can serve as an auxiliary supervision task for segmentation, not just for pixel reconstruction, making it a viable component in low-label medical settings.
- Explicitly reducing the gap between masked-domain and original-domain predictions, via a discriminator on output masks, is presented as necessary for the auxiliary task to transfer.
- The construction applies symmetrically to the CNN branch, so both halves of a CNN-transformer cross-teaching pair receive the extra supervision and alignment.
- The largest documented gains appear exactly where labels are scarcest, on 3% partitions, suggesting the method's value scales with annotation scarcity.
- The certainty-weighted pseudo-label objective is part of the bound's noise term, linking pseudo-label quality directly to the theoretical guarantee.
Reading between the lines
- If the gain reproduces under matched training protocols, the method implies that masking objectives and adversarial domain alignment can be folded into any cross-teaching semi-supervised scheme without changing the architecture, lowering the practical label threshold for transformer-based segmentation.
- The mask-ratio sensitivity, best at 0.7 and degraded at 0.9, suggests a testable extension: an adaptive mask-ratio schedule that anneals from high to low as training converges could keep supervision high while shrinking the domain gap the discriminator must close.
- The bound suggests that any reduction of pseudo-label noise, such as sharper confidence thresholds or uncertainty filtering, should have an effect comparable to the adversarial term; ablating those two contributors separately would isolate which mechanism drives the reported gains.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AdvMIM, a semi-supervised medical image segmentation method that constructs an auxiliary masked domain using masked image modeling, trains a Swin-UNet transformer to predict full segmentation masks from masked inputs, and adds an adversarial loss to reduce the domain gap between original and masked prediction masks. The method is extended to a CNN branch via cross-teaching. A theoretical bound (Theorem 1) is presented to motivate the adversarial domain adaptation, and experiments on ACDC, Synapse, and ISIC report large Dice/HD improvements over prior state-of-the-art methods, most notably a 10.1-point Dice gain over M-CnT on ACDC with 3% labeled data. The code is publicly available.
Significance. If the empirical gains are reproducible under controlled conditions, the idea of using masked image modeling as an auxiliary training-domain generator for semi-supervised segmentation is a useful contribution to the field, and the public code is a concrete asset. The ablation study in Table 1 shows consistent gains from each component on two datasets, which supports the internal logic of the method. However, the paper's load-bearing claim of 'significant' superiority over existing methods is currently supported only by cross-paper baseline numbers without multiple seeds or statistical tests, so the significance is not yet established at the level the manuscript claims.
major comments (3)
- [Section 3, Table 2] The central empirical claim that AdvMIM 'outperforms existing methods significantly' rests on comparisons against baseline numbers taken from their original publications, without re-running those methods under the authors' training protocol (SGD with lr 0.05, batch size 16, 30k iterations, 224x224 patches). The reported 10.1 Dice-point gain over M-CnT on ACDC 3% is especially striking and could be inflated by differences in preprocessing, augmentation, evaluation details, or run-to-run variance. No standard deviations or multiple-seed results are reported for any method, so the word 'significantly' is not statistically supported. The authors should re-run at least the strongest baselines (M-CnT, CTCT) using their own protocol and report mean and standard deviation over multiple seeds for all methods.
- [Section 2.3, Theorem 1] The proof of Theorem 1 is only a sketch, and the bound's applicability is not established. The term lambda is defined as min_h epsilon_P(h) + epsilon_Q(h), but the distribution Q (as opposed to the pseudo-labeled masked distribution Q') is never defined in the paper. Furthermore, the discrepancy term d_{HΔH}(P, Q') is a divergence over input distributions in the Ben-David et al. framework, whereas the proposed adversarial loss in Eqs. (6)-(7) operates on the prediction masks S(x^l) and S(x^m_u). The manuscript does not explain how minimizing a divergence over prediction-mask spaces reduces the input-space discrepancy d_{HΔH}(P, Q'). A complete proof, or at least a precise statement of which distributions and spaces are involved, is needed before the theoretical motivation can be assessed.
- [Section 3, Tables 3-4] The sensitivity analyses for the mask ratio rho and the adversarial loss weight lambda_adv are conducted only on ACDC with 3% labeled data. The default values (rho = 0.7, lambda_adv = 0.001) are selected on this setting and then transferred to Synapse and ISIC without any evidence that these choices are robust across datasets. The paper should report sensitivity results on at least one other dataset, or provide a principled justification for why the ACDC 3% tuning should transfer to other anatomies and modalities.
minor comments (4)
- [Section 2.2, Eqs. (3)-(4)] Equations (3) and (4) contain mismatched parentheses and brackets: Eq. (3) has an extra closing parenthesis after S(x_m^l), and Eq. (4) is missing a closing bracket for the Dice term. These typographical errors should be corrected.
- [Section 2.2] The description of the masking operation says 'we replace the masked image patches with shared learnable mask tokens with positional embedding,' but it is unclear whether positional embeddings are added to the mask tokens themselves or to the remaining visible patches. Clarify this implementation detail.
- [Section 3, Table 2] The table caption states that second-best results are underlined, but the in-text table rendering does not visibly distinguish underlined entries. Please ensure the formatting is clear in the final version.
- [Section 2.3, Eq. (8)] The overall objective in Eq. (8) should explicitly distinguish the discriminator losses for D_s and D_c; currently L_dmim is written once for both, which is ambiguous. Also, a brief note on the training schedule or stability of the min-max optimization would help reproducibility.
Circularity Check
No significant circularity: AdvMIM's outputs are empirical Dice/HD scores produced from labels and pseudo-labels, and its theoretical bound is adapted from an external published lemma rather than from the authors' own prior work.
full rationale
AdvMIM's load-bearing claims are (i) the empirical Dice/HD gains in Table 2 and (ii) the bound in Theorem 1. Neither is circular. The reported numbers are produced by training on labeled and pseudo-labeled data; test labels are never used to fit the model or to select the reported metrics, so the comparisons are not fitted values renamed as predictions. The masked-domain learning loss in Eqs. (3)-(4) uses the full segmentation masks as supervision for masked inputs; this is a distinct input/task pair, and its benefit is supported only by the Table 1 ablation, not by construction. The adversarial loss in Eqs. (6)-(7) is a standard GAN objective with a domain discriminator; it is not derived from the evaluation metric. Theorem 1's proof sketch invokes 'Lemma 4 in [3]' (Ben-David et al. 2010), an external published result, rather than a self-citation; the pseudo-label noise term gamma and the optimal error term lambda appear as explicit quantities rather than as hidden fitted constants. The only self-reference is [29], cited in the Introduction as one example of deep co-training methods; that citation plays no load-bearing role in the derivation or evaluation. Mask-ratio and lambda_adv choices are reported as sensitivity analyses (Tables 3-4), not as predictions. The abbreviated proof sketch is a rigor concern, but it is not a circularity concern. Therefore the paper shows no significant circularity.
Assumptions & free parameters
free parameters (2)
- masking ratio ρ =
0.7
- adversarial loss weight λ_adv =
0.001
assumptions (4)
- standard math Lemma 4 from Ben-David et al. (2010) correctly bounds the multi-domain error as stated in Theorem 1's proof.
- domain assumption The pseudo-label noise ratio γ is bounded and the certainty weighting effectively minimizes it.
- domain assumption The optimal error λ between original and masked domains is small.
- ad hoc to paper Random masking with ratio 0.7 produces an auxiliary domain that is useful for learning the segmentation task.
Cite this review
Pith. "Pith review of AdvMIM: Adversarial Masked Image Modeling for Semi-Supervised Medical Image Segmentation." pith.science (2026). https://pith.science/paper/ZFJ2XWA4
@misc{pith2026250620563,
author = {Pith},
title = {Pith review of: AdvMIM: Adversarial Masked Image Modeling for Semi-Supervised Medical Image Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZFJ2XWA4}},
note = {Machine review of arXiv:2506.20563}
}
read the original abstract
Vision Transformer has recently gained tremendous popularity in medical image segmentation task due to its superior capability in capturing long-range dependencies. However, transformer requires a large amount of labeled data to be effective, which hinders its applicability in annotation scarce semi-supervised learning scenario where only limited labeled data is available. State-of-the-art semi-supervised learning methods propose combinatorial CNN-Transformer learning to cross teach a transformer with a convolutional neural network, which achieves promising results. However, it remains a challenging task to effectively train the transformer with limited labeled data. In this paper, we propose an adversarial masked image modeling method to fully unleash the potential of transformer for semi-supervised medical image segmentation. The key challenge in semi-supervised learning with transformer lies in the lack of sufficient supervision signal. To this end, we propose to construct an auxiliary masked domain from original domain with masked image modeling and train the transformer to predict the entire segmentation mask with masked inputs to increase supervision signal. We leverage the original labels from labeled data and pseudo-labels from unlabeled data to learn the masked domain. To further benefit the original domain from masked domain, we provide a theoretical analysis of our method from a multi-domain learning perspective and devise a novel adversarial training loss to reduce the domain gap between the original and masked domain, which boosts semi-supervised learning performance. We also extend adversarial masked image modeling to CNN network. Extensive experiments on three public medical image segmentation datasets demonstrate the effectiveness of our method, where our method outperforms existing methods significantly. Our code is publicly available at https://github.com/zlheui/AdvMIM.
Figures
Reference graph
Works this paper leans on
-
[1]
Bai, W., Oktay, O., Sinclair, M., Suzuki, H., Rajchl, M., Tarroni, G., Glocker, B., King, A., Matthews, P.M., Rueckert, D.: Semi-supervised learning for network- based cardiac mr image segmentation. In: MICCAI 2017. Springer (2017)
work page 2017
-
[2]
Basak, H., Bhattacharya, R., Hussain, R., Chatterjee, A.: An exceedingly simple consistency regularization method for semi-supervised medical image segmenta- tion. In: ISBI 2022. pp. 1–4. IEEE (2022)
work page 2022
-
[3]
Machine learning79, 151–175 (2010)
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.W.: A theory of learning from different domains. Machine learning79, 151–175 (2010)
work page 2010
-
[4]
Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.A., Cetin, I., Lekadir, K., Camara, O., Ballester, M.A.G., et al.: Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging37(11), 2514–2525 (2018)
2018
-
[5]
In: European conference on computer vision
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M.: Swin- unet: Unet-like pure transformer for medical image segmentation. In: European conference on computer vision. pp. 205–218. Springer (2022)
2022
-
[6]
arXiv preprint arXiv:2102.04306 (2021)
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.:Transunet:Transformersmakestrongencodersformedicalimagesegmentation. arXiv preprint arXiv:2102.04306 (2021)
arXiv 2021
- [7]
-
[8]
Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al.: Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomed- ical imaging (isbi), hosted by the international skin imaging collaboration (isic). In: ISBI 2018. pp. 168–17...
work page 2018
Show all 30 references
-
[9]
arXiv preprint arXiv:2010.11929 (2020) 10 Lei Zhu et al
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020) 10 Lei Zhu et al
2020 arXiv
-
[10]
In: Advances in neural information processing systems
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
2014
-
[11]
IEEE journal of biomedical and health informatics24(9), 2599–2608 (2020)
Hu, X., Guo, R., Chen, J., Li, H., Waldmannstetter, D., Zhao, Y., Li, B., Shi, K., Menze, B.: Coarse-to-fine adversarial networks and zone-based uncertainty analysis for nk/t-cell lymphoma segmentation in ct/pet images. IEEE journal of biomedical and health informatics24(9), 2...
2020
-
[12]
In: AAAI 2024
Huang, H., Huang, Y., Xie, S., Lin, L., Tong, R., Chen, Y.W., Li, Y., Zheng, Y.: Combinatorial cnn-transformer learning with manifold constraints for semi- supervised medical image segmentation. In: AAAI 2024. vol. 38, pp. 2330–2338 (2024)
2024
-
[13]
In: MICCAI: multi Atlas labeling beyond cranial vault-workshop challenge (2015)
Landman, B., Xu, Z., Igelsias, J.E., Styner, M., Langerak, T., Klein, A.: Segmenta- tion outside the cranial vault challenge. In: MICCAI: multi Atlas labeling beyond cranial vault-workshop challenge (2015)
2015
-
[14]
In: Workshop on challenges in representation learning, ICML
Lee, D.H., et al.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on challenges in representation learning, ICML. vol. 3, p. 896. Atlanta (2013)
2013
-
[15]
In: International conference on medical imaging with deep learning
Luo, X., Hu, M., Song, T., Wang, G., Zhang, S.: Semi-supervised medical image segmentation via cross teaching between cnn and transformer. In: International conference on medical imaging with deep learning. pp. 820–833. PMLR (2022)
2022
-
[16]
In: MICCAI
Luo, X., Liao, W., Chen, J., Song, T., Chen, Y., Zhang, S., Chen, N., Wang, G., Zhang, S.: Efficient semi-supervised gross target volume of nasopharyngeal carcinoma segmentation via uncertainty rectified pyramid consistency. In: MICCAI
-
[17]
In: Proceedings of the IEEE international conference on computer vision
Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares gen- erative adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2794–2802 (2017)
2017
-
[18]
In: CVPR
Ouali, Y., Hudelot, C., Tami, M.: Semi-supervised semantic segmentation with cross-consistency training. In: CVPR. pp. 12674–12684 (2020)
2020
-
[19]
In: Proceedings of the european conference on com- puter vision (eccv)
Qiao, S., Shen, W., Zhang, Z., Wang, B., Yuille, A.: Deep co-training for semi- supervised image recognition. In: Proceedings of the european conference on com- puter vision (eccv). pp. 135–152 (2018)
2018
-
[20]
In: MICCAI 2015
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: MICCAI 2015. pp. 234–241. Springer (2015)
2015
-
[21]
Advances in neural information processing systems30(2017)
Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems30(2017)
2017
-
[22]
arxiv 2020
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Train- ing data-efficient image transformers & distillation through attention. arxiv 2020. arXiv preprint arXiv:2012.128772(3) (2020)
2020 arXiv
-
[23]
Neural Networks145, 90–106 (2022)
Verma, V., Kawaguchi, K., Lamb, A., Kannala, J., Solin, A., Bengio, Y., Lopez- Paz, D.: Interpolation consistency training for semi-supervised learning. Neural Networks145, 90–106 (2022)
2022
-
[24]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2517– 2526 (2019)
2019
-
[25]
In: MICCAI 2022
Wu, Y., Wu, Z., Wu, Q., Ge, Z., Cai, J.: Exploring smoothness and class-separation for semi-supervised medical image segmentation. In: MICCAI 2022. pp. 34–43. Springer (2022) AdvMIM for Semi-Supervised Medical Image Segmentation 11
2022
-
[26]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., Hu, H.: Simmim: A simple framework for masked image modeling. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9653–9663 (2022)
2022
-
[27]
In: MICCAI 2019
Yu, L., Wang, S., Li, X., Fu, C.W., Heng, P.A.: Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation. In: MICCAI 2019. pp. 605–613. Springer (2019)
2019
-
[28]
In: MICCAI 2017
Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D.P., Chen, D.Z.: Deep adversarial networks for biomedical image segmentation utilizing unannotated im- ages. In: MICCAI 2017. pp. 408–416. Springer (2017)
2017
-
[29]
In: MICCAI 2021
Zhu, L., Yang, K., Zhang, M., Chan, L.L., Ng, T.K., Ooi, B.C.: Semi-supervised unpaired multi-modal learning for label-efficient medical image segmentation. In: MICCAI 2021. pp. 394–404. Springer (2021)
2021
-
[2021]
pp. 318–329. Springer (2021)
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.