REVIEW 4 major objections 6 minor 25 references
Self-Supervised Learning for Pre-training Capsule Networks: Overcoming Medical Imaging Dataset Challenges
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Self-supervised pre-training on a small, imbalanced medical dataset can initialize a capsule network as well as or better than ImageNet pre-training.
desk verdict A plausible first exploration of SSL pre-training for CapsNets whose headline claim is undercut by the paper's own table: no error bars, impossible balanced-accuracy values, and ImageNet ahead on MCC and AUROC. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the modified capsule network plus the self-supervised pretext tasks used to pre-train it. The architecture adds five convolutional layers before the primary-capsule layer with smaller kernels and strides, replaces the original margin loss with spread loss, and uses a decoder built from transposed convolutions with down-weighted skip connections. The mechanism that is claimed to do the work is contrastive learning with an in-painting auxiliary task: augmented pairs of the same image are pulled together by the NT-Xent loss while the reconstruction branch fills masked patches, and the combined loss is meant to teach colour and texture features that matter for polyp diagnosis. The paper also compares this against a colourisation pretext task and against ImageNet-pretrained ResNet features applied to the first four convolutional layers.
What would settle it
Re-run the fine-tuning stage from a random initialization using the exact same augmentation pipeline, weighted sampler, loss weights, and training schedule as the SSL-contrastive branch; if that run matches or exceeds 0.40 accuracy on the PICCOLO test set, the central claim is falsified. A second check: train the SSL branch for only one epoch so the weights barely move, then fine-tune; if downstream accuracy is unchanged, the pre-training step is not the cause of the gain.
Extended reading notes
Core claim
The central claim is that contrastive self-supervised learning combined with an in-painting task is a viable weight-initialization strategy for capsule networks on small medical datasets, achieving results comparable to ImageNet pre-training without requiring a large external corpus. On the PICCOLO test set, the SSL-contrastive pre-trained model reaches 0.40 accuracy against 0.38 for the ImageNet-initialized model, with higher balanced accuracy (0.16 vs 0.15) and higher specificity (0.70 vs 0.69), while the ImageNet model retains higher MCC (0.25 vs 0.22) and AUROC (0.71 vs 0.62). The paper reports these as increases of 5.26%, 6.67%, and 1.45% respectively in the conclusion. The colourisation pretext task performs poorly, near random, so the paper's positive result is specific to the contrastive-with-in-painting task, not SSL in general.
Load-bearing premise
The claim assumes that the self-supervised pre-training actually moves the capsule network's weights into a better starting point for fine-tuning; if the tiny weight changes observed during SSL training do not meaningfully affect downstream performance, the reported gains could instead come from the augmented training, the weighted sampling, or chance.
Editorial extensions
If this is right
- SSL pre-training on the target medical dataset can substitute for ImageNet pre-training when initializing capsule networks, removing the need for a large external dataset.
- Contrastive learning combined with in-painting is the productive SSL recipe for this domain; colourisation alone is not.
- Because capsule networks learn slowly, extending SSL training from 200 toward 500 epochs and using layer-specific learning rates are the paper's own suggested next steps for improving the gains.
- SSL pre-training on imbalanced data teaches the network the dataset's class biases, adding 5-10 fine-tuning epochs before the bias is overcome.
- Capsule networks are sensitive to initialization, and a self-supervised initialization that captures domain-specific features is a way to reduce that sensitivity.
Reading between the lines
- The reported gains are relative improvements on a single test split; the paper does not report repeated-seed variance, so it remains open whether the 5.26% accuracy edge is stable across runs.
- The observed weight-update magnitudes of $10^{-6}$ to $10^{-5}$ during SSL pre-training raise the possibility that the downstream gains come from the augmented training and weighted sampling rather than from the SSL initialization itself; this competing explanation is not isolated in the paper.
- If the mechanism is domain-relevant feature learning, the same contrastive-plus-in-painting recipe could plausibly transfer to other small medical imaging tasks with unlabelled target frames, though the paper only tests colon polyp classification.
- A cheaper substitute, longer fine-tuning from random initialization with the same augmentations and weighting, is not compared, so the specific value of the SSL pre-training step over extended supervised training is unmeasured.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript studies self-supervised pre-training for a modified capsule network on the PICCOLO polyp dataset, a small three-class medical imaging benchmark with class imbalance and distribution shift. The authors compare four initialization strategies: Kaiming/Xavier, ImageNet-initialized first convolutional layers, SSL colorisation, and SSL contrastive learning combined with in-painting. They report that the SSL-contrastive model achieves accuracy 0.40 versus 0.38 for ImageNet pre-training and conclude, in Section 5, that it surpasses ImageNet pre-training by 5.26% in accuracy, 6.67% in balanced accuracy, and 1.45% in specificity. The paper also discusses CapsNet training dynamics, including very small weight updates and weak engagement of class capsules during SSL, and proposes longer training and layer-specific learning rates as future work.
Significance. The practical motivation is solid: CapsNets lack off-the-shelf pretrained weights, and target-domain SSL could be a cheap alternative to ImageNet pre-training for small medical datasets. The paper contributes a modified CapsNet architecture for 224x224 inputs, two SSL pretext-task variants, and a candid discussion of optimization and imbalance difficulties. However, the headline claim that SSL-contrastive pre-training surpasses ImageNet pre-training is not currently supported by the evidence: the comparison is single-run, the reported balanced-accuracy values are arithmetically inconsistent with the reported accuracies, and the same table shows ImageNet ahead on MCC and AUROC, metrics the authors themselves identify as primary. With corrected metrics, multi-seed statistics, and a fairer comparison, this would be a useful empirical data point.
major comments (4)
- [Section 3.1, Table 1] The reported balanced-accuracy values are arithmetically incompatible with the reported accuracies under the standard definition of balanced accuracy as mean recall. For a three-class problem, random balanced accuracy is 0.333. Given the test-set class proportions reported in Section 3.1 (0.38, 0.34, 0.28), an overall accuracy of 0.38 requires a mean recall of at least about 0.333, and an accuracy of 0.40 requires at least about 0.353; the tabled values of 0.12-0.16 are therefore below what is possible. Unless 'balanced accuracy' is defined in a nonstandard way, Table 1 cannot be correct as reported. Please provide the confusion matrix, per-class recall, and explicit definitions of balanced accuracy, AUROC, and specificity for this three-class problem.
- [Section 5, Table 1] The conclusion that SSL-contrastive pre-training 'surpasses' ImageNet pre-training is not supported by the reported evidence. The absolute differences are 0.01-0.02 on a 333-frame test set, corresponding to a handful of frames, and no confidence intervals, repeated-seed runs, or significance tests are reported. Moreover, Table 1 itself shows ImageNet pre-training ahead on MCC (0.25 vs 0.22) and AUROC (0.71 vs 0.62), the two metrics the paper in Section 4 identifies as primary, yet these differences are omitted from the Section 5 summary. Please report multiple seeds with variability measures, a pre-specified primary metric, and a discussion of all metrics in the table.
- [Section 4] The proposed mechanism of SSL pre-training is not established by the reported training dynamics. The text reports that primary-capsule and routing-weight updates during SSL training vary only from 10^-6 to 10^-5 and that class capsules 'barely engage' during SSL. If the pre-training changes the weights so little, the downstream accuracy differences may be attributable to data augmentation, the weighted sampler, early stopping, or random variation rather than to the initialization itself. Please quantify the causal effect of the pre-training, for example by reinitializing the class capsules before fine-tuning, ablating the SSL phase, and reporting convergence curves and gradient-norm statistics for each initialization strategy.
- [Sections 3.4 and 4] The comparison between initialization strategies is confounded. ImageNet weights are applied to only the first four convolutional layers, while SSL weights initialize all layers; SSL models are trained for 200 epochs whereas supervised models are trained for a minimum of 50 epochs with early stopping; and hyperparameters were selected separately for each condition on a validation subset. These differences make it impossible to attribute the observed performance differences to the pre-training objective. Please use identical layer coverage, identical training budgets and schedules, a common hyperparameter-selection protocol, and multiple seeds across all four conditions.
minor comments (6)
- [Section 3.1] The validation-set description appears to contain a typo: it says the dominant class remains NICE Type 1 at 66%, then refers to Types 1 and 3 making up 15% and 19%; presumably the dominant class is Type 2. Please correct the class labels.
- [Equations (2), (7), (8)] The contrastive loss equations contain formatting errors, including 'P2N' in the denominator and unclear vector notation in the cosine-similarity expression. Please rewrite these equations cleanly and define all symbols consistently.
- [Section 3.3] The modified architecture is described only through a figure and general statements. For reproducibility, please provide a table with the exact layer configuration: kernel sizes, strides, channel counts, capsule dimensions, routing iterations, and decoder/transposed-convolution layout.
- [Abstract and Section 4] The abstract states that the findings suggest contrastive learning and in-painting are suitable auxiliary tasks, while Section 4 reports that colorisation performs at near-random level. Please clarify in the abstract that colorisation was the underperforming pretext task.
- [Table 1] Please report metrics with more than two decimals, since the relative improvements in Section 5 (e.g., 5.26% from 0.38 to 0.40) are computed from rounded values and may not be stable. Also specify how AUROC is computed for a three-class problem.
- [General] The manuscript does not state whether code, trained model weights, or experiment logs will be made available; given that the central comparison is single-run, releasing these artifacts would substantially increase confidence in the results.
Circularity Check
No significant circularity: the paper is an empirical benchmarking study whose claims rest on measured test-set comparisons, not on a derivation that reduces to its inputs.
full rationale
The paper's central claim is that SSL contrastive pre-training of a CapsNet on PICCOLO achieves test-set results comparable to or slightly better than ImageNet pre-training (Section 5: 'Our SSL-contrastive pre-training approach, despite using a significantly smaller dataset, achieved comparable results to ImageNet pre-training, even surpassing it with increases of 5.26% in accuracy, 6.67% in balanced accuracy and 1.45% in specificity'). This is an empirical outcome of training runs reported in Table 1, not a quantity that is defined in terms of the method's own outputs. The SSL losses (Eqs. 4-10) are standard reconstruction and NT-Xent objectives; the downstream classification uses spread loss (Eq. 3); no equation in the paper defines the reported accuracy, balanced accuracy, or specificity in terms of the SSL pre-training objective itself. Hyperparameters were selected on a validation subset ('Hyperparameters were selected through extensive and systematic experimentation, optimising model performance on a validation subset of the PICCOLO dataset'), which is ordinary model selection rather than fitted-input-called-prediction because the test set is held out and the headline comparison is between initialisation strategies, not a parameter fit to the test metric. The only self-citation, reference [8] ('El-Shimy, H., Zantout, H., Lones, M.A., Gayar, N.E.: A review of capsule networks in medical image analysis'), is used to summarise prior CapsNet DMI applications ('Several studies have successfully implemented CapsNets for DMI as summarised in [8]'); it is not load-bearing for the empirical comparison, and no uniqueness theorem or methodological premise is imported from it. The manuscript itself notes that weight updates during SSL are very small (10^-6 to 10^-5) and that class capsules barely engage, which is a mechanistic limitation that would weaken the causal story but is not circularity. The absent error bars and the arithmetically implausible balanced-accuracy values are serious correctness/statistical concerns, but they are not instances of a derivation being equivalent to its inputs. No step in the paper exhibits the required pattern of X defined in terms of Y, a fitted parameter renamed as a prediction, or a self-citation chain forcing the conclusion. Score 0.
Assumptions & free parameters
free parameters (6)
- Perceptual loss ramp weight (lambda_p) =
0.1 (max, ramped over first 30% of epochs)
- Contrastive temperature (tau) =
not reported
- Learning rate schedule =
1e-3 to 1e-6 warm restarts
- SSL training epochs =
200 (suggests 500)
- Architecture capacities (kernel sizes, strides, capsule counts, routing iterations) =
not fully specified
- Augmentation hyperparameters =
flips 0.35, perspective 0.25, colour jitter 0.2
assumptions (5)
- domain assumption The NICE classification labels in PICCOLO are reliable ground truth.
- domain assumption Capsule networks are inherently interpretable and can capture part-whole relationships via routing.
- domain assumption Histogram matching WL frames to the closest NBI frame of the same polyp does not leak label information or distort features.
- domain assumption Gradient accumulation over 8 mini-batches of size 8 is equivalent to using a true batch size of 64 for CapsNet training.
- domain assumption Spread loss (Eq. 3) is more suitable for multi-class classification than margin loss.
Cite this review
Pith. "Pith review of Self-Supervised Learning for Pre-training Capsule Networks: Overcoming Medical Imaging Dataset Challenges." pith.science (2026). https://pith.science/paper/OJJI4AME
@misc{pith2026250204748,
author = {Pith},
title = {Pith review of: Self-Supervised Learning for Pre-training Capsule Networks: Overcoming Medical Imaging Dataset Challenges},
year = {2026},
howpublished = {\url{https://pith.science/paper/OJJI4AME}},
note = {Machine review of arXiv:2502.04748}
}
read the original abstract
Deep learning techniques are increasingly being adopted in diagnostic medical imaging. However, the limited availability of high-quality, large-scale medical datasets presents a significant challenge, often necessitating the use of transfer learning approaches. This study investigates self-supervised learning methods for pre-training capsule networks in polyp diagnostics for colon cancer. We used the PICCOLO dataset, comprising 3,433 samples, which exemplifies typical challenges in medical datasets: small size, class imbalance, and distribution shifts between data splits. Capsule networks offer inherent interpretability due to their architecture and inter-layer information routing mechanism. However, their limited native implementation in mainstream deep learning frameworks and the lack of pre-trained versions pose a significant challenge. This is particularly true if aiming to train them on small medical datasets, where leveraging pre-trained weights as initial parameters would be beneficial. We explored two auxiliary self-supervised learning tasks, colourisation and contrastive learning, for capsule network pre-training. We compared self-supervised pre-trained models against alternative initialisation strategies. Our findings suggest that contrastive learning and in-painting techniques are suitable auxiliary tasks for self-supervised learning in the medical domain. These techniques helped guide the model to capture important visual features that are beneficial for the downstream task of polyp classification, increasing its accuracy by 5.26% compared to other weight initialisation methods.
Figures
Reference graph
Works this paper leans on
-
[1]
Afshar, P., Oikonomou, A., Naderkhani, F., Tyrrell, P.N., Plataniotis, K.N., Fara- hani, K., Mohammadi, A.: 3d-mcn: A 3d multi-scale capsule network for lung nodule malignancy prediction. Scientific Reports10 (2020)
work page 2020
-
[2]
2021 IEEE/CVF International Conference on Computer Vision (ICCV) (10 2021).https://doi
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging Properties in Self-Supervised Vision Transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (10 2021).https://doi. org/10.1109/iccv48922.2021.00951
-
[3]
International Conference on Machine Learning 1, 1597–1607 (7 2020)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for con- trastive learning of visual representations. International Conference on Machine Learning 1, 1597–1607 (7 2020)
work page 2020
-
[4]
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., Hinton, G.E.: Big self-supervised models are strong semi-supervised learners. ArXivabs/2006.10029 (2020)
arXiv 2020
-
[5]
Deepika, J., Rajan, C., Senthil, T.: Improved CAPSNET model with modified loss function for medical image classification. Signal Image and Video Processing16(8), 2269–2277 (3 2022).https://doi.org/10.1007/s11760-022-02192-5
-
[6]
In: CVPR09 (2009)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large- Scale Hierarchical Image Database. In: CVPR09 (2009)
2009
-
[7]
2015 IEEE International Conference on Computer Vision (ICCV) pp
Doersch, C., Gupta, A.K., Efros, A.A.: Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference on Computer Vision (ICCV) pp. 1422–1430 (2015)
work page 2015
-
[8]
In: IAPR International Workshop on Artificial Neural Networks in Pattern Recognition (2022)
El-Shimy, H., Zantout, H., Lones, M.A., Gayar, N.E.: A review of capsule networks in medical image analysis. In: IAPR International Workshop on Artificial Neural Networks in Pattern Recognition (2022)
work page 2022
Show all 25 references
-
[9]
Endoscopy37(6), 570–578 (Jun 2005)
Endoscopic Classification Review Group: Update on the paris classification of su- perficial neoplastic lesions in the digestive tract. Endoscopy37(6), 570–578 (Jun 2005). https://doi.org/10.1055/s-2005-861352
2005 doi
-
[10]
International Conference on Artificial Intelligence and Self-Supervised Learning for Pre-training Capsule Networks 13 Statistics pp
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedfor- ward neural networks. International Conference on Artificial Intelligence and Self-Supervised Learning for Pre-training Capsule Networks 13 Statistics pp. 249–256 (3 2010),https://proceedings.mlr.pr...
2010
-
[11]
World Journal of Gastrointestinal Endoscopy6(12), 600–605 (Dec 2014)
Hattori, S., Iwatate, M., Sano, W., Hasuike, N., Kosaka, H., Ikumoto, T., Kotaka, M., Ichiyanagi, A., Ebisutani, C., Hisano, Y., Fujimori, T., Sano, Y.: Narrow-band imaging observation of colorectal lesions using NICE classification to avoid discard- ing significant lesions. W...
2014 doi
-
[12]
2015 IEEE International Conference on Computer Vision (ICCV) pp
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV) pp. 1026–1034 (2015), https://api. semanticscholar.org/CorpusID:13740328
2015
-
[13]
Interna- tional Conference on Learning Representations (2 2018)
Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with EM routing. Interna- tional Conference on Learning Representations (2 2018)
2018
-
[14]
Computer Methods and Programs in Biomedicine Update5, 100146 (1 2024)
Khalifa, M., Albadawy, M.: AI in diagnostic imaging: Revolutionising accuracy and efficiency. Computer Methods and Programs in Biomedicine Update5, 100146 (1 2024). https://doi.org/10.1016/j.cmpbup.2024.100146
2024
-
[15]
BMC Medical Imaging23(1) (4 2023).https://doi.org/10.1186/ s12880-023-01007-4
Krenzer, A., Heil, S., Fitting, D., Matti, S., Zoller, W.G., Hann, A., Puppe, F.: Automated classification of polyps using deep learning architectures and few- shot learning. BMC Medical Imaging23(1) (4 2023).https://doi.org/10.1186/ s12880-023-01007-4
2023
-
[16]
Springer, Cham (1 2020)
LaLonde, R., Torigian, D., Bagci, U.: Encoding visual attributes in capsules for explainable medical diagnoses. Springer, Cham (1 2020). https://doi.org/10. 1007/978-3-030-59710-8_29
2020
-
[17]
2017 IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR) pp
Larsson, G., Maire, M., Shakhnarovich, G.: Colorization as a proxy task for visual understanding. 2017 IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR) pp. 840–849 (2017)
2017
-
[18]
arXiv: Learning (2016)
Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv: Learning (2016)
2016
-
[19]
In: International Conference on Learning Representations (2017)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2017)
2017
-
[20]
In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context en- coders: Feature learning by inpainting. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2536–2544 (2016). https://doi. org/10.1109/CVPR.2016.278
2016 doi
-
[21]
Journal of Digital Imaging11(4), 193–200 (11 1998).https: //doi.org/10.1007/bf03178082
Pisano, E.D., Zong, S., Hemminger, B.M., DeLuca, M., Johnston, R.E., Muller, K., Braeuning, M.P., Pizer, S.M.: Contrast Limited Adaptive Histogram Equal- ization image processing to improve the detection of simulated spiculations in dense mammograms. Journal of Digital Imaging...
1998 doi
- [22]
-
[23]
Annual Review of Biomedical Engineering19(1), 221–248 (6 2017).https://doi.org/10
Shen, D., Wu, G., Suk, H.I.: Deep learning in medical image analysis. Annual Review of Biomedical Engineering19(1), 221–248 (6 2017).https://doi.org/10. 1146/annurev-bioeng-071516-044442
2017
-
[24]
Applied Sciences10(23), 8501 (11 2020)
Sánchez-Peralta, L.F., Pagador, J.B., Picón, A., Calderón, A.J., Polo, F., An- draka, N., Bilbao, R., Glover, B., Saratxaga, C.L., Sánchez-Margallo, F.M.: PIC- COLO White-Light and Narrow-Band Imaging Colonoscopic Dataset: A Perfor- mance Comparative of Models and Datasets. Ap...
2020 doi
-
[25]
Springer, Cham (1 2019)
Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Loy, C.C.: ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. Springer, Cham (1 2019). https://doi.org/10.1007/978-3-030-11021-5_5
2019 doi
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.