REVIEW 4 major objections 5 minor 20 references
Thoughts on Objectives of Sparse and Hierarchical Masked Image Model
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper reports that the masking pattern used when pretraining SparK on brain CT slices changes downstream tumor-detection F1 by roughly five points, with patch-level random and mesh masks tied at 87.7 and block-level masks scoring…
desk verdict The central claim rests on validation-set numbers that match the 26-positive Evaluation split, not the 21-positive Test split; the mesh mask ties the random mask anyway. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the mesh mask, a checkerboard-style masking algorithm. For a 7x7 grid of 49 patches, it first randomly picks one of two parity classes of candidate unmasked patches (coordinates satisfying 7j+i≡0 or 1 mod 2), then randomly selects unmasked patches from that class until the target mask ratio is reached; all remaining patches are masked. This design guarantees that unmasked patches are spread evenly across the image, so a small tumor is less likely to be wholly erased. The argument also rests on the SparK architecture itself, a sparse hierarchical ConvNeXt encoder with a light decoder, and on the mask-pattern comparison framework from SimMIM, which already showed that mask choice affects MIM performance.
What would settle it
Run the same SparK pretraining pipeline for square, block-wise, random, and mesh masks with several random seeds and compute bootstrap confidence intervals for F1 on the same 179-slice test set; since one positive slice misclassified shifts recall by about 4.8 points, the claim that patch-level masks outperform block-level masks would be refuted if the intervals overlap by that amount.
Extended reading notes
Core claim
On the paper's own terms, the masking pattern used during SparK pretraining has a non-negligible influence on downstream brain-tumor-detection performance. Random and mesh masks reach F1 87.7 at their best settings, while square masks peak at 83.6 and block-wise masks plateau at 82.8, giving a gap of about five F1 points. The paper proposes the mesh mask, which selects visible patches by randomly choosing one of two parity classes of a 7x7 grid so that visible and masked patches interlace; at a 70 percent mask ratio it ties the random mask but does not exceed it. The authors interpret the gap as evidence that patch-level masking is particularly effective for SparK on this data, and that block-level masks risk hiding an entire small lesion. The paper also shows SparK outperforming SimCLR and MFF when RandomResizedCrop is used, and reports that sparsity in the backbone adds about 7.1 F1 points over a non-sparse ConvNeXt variant.
Load-bearing premise
The result rests on treating the three-to-five-point F1 differences as real signal rather than sampling noise on a 179-slice test set that contains only 21 tumor-positive slices.
Editorial extensions
If this is right
- On the paper's evidence, practitioners should treat the mask pattern as a key pretraining hyperparameter for SparK on small medical datasets, not just the mask ratio.
- Patch-level masks (random or mesh) give roughly five F1 points more than block-level masks in this brain-CT setup.
- The proposed mesh mask matches the random mask's top F1 of 87.7 but does not exceed it, so it is an alternative rather than an improvement in this experiment.
- SparK's downstream performance depends strongly on the augmentation chosen, with RandomResizedCrop giving F1 87.7 versus 16.7 to 52.2 for the other augmentations tested.
- Sparsity in the ConvNeXt backbone contributes to performance: removing it drops F1 from 87.7 to 80.6 in the paper's comparison.
Reading between the lines
- A natural ablation the paper does not run would hide entire tumor regions during pretraining and measure whether downstream F1 drops; this would test the stated mechanism that full occlusion hurts block-level masks.
- The mesh mask's parity rule could be generalized to other grid sizes or combined with block-level masks, making it easy to compare evenly spread masking with random masking on other datasets.
- Because the test set contains only 21 positive slices, re-running each mask condition with several seeds and reporting bootstrap confidence intervals would show whether the five-point gap is stable; the paper reports single runs.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates self-supervised pretraining objectives on a brain-CT slice classification task. It first compares SimCLR, MFF, and SparK, finding SparK with RandomResizedCrop to have the highest F1 (87.7). It then studies how the masking pattern used during SparK pretraining affects downstream tumor-detection performance, comparing square, block-wise, random, and a newly proposed mesh mask. The central claim in Section 8 is that the pretraining mask pattern has a non-negligible effect on downstream performance, with patch-level masks (random and mesh) outperforming block-level masks by more than 5 F1 points, while the proposed mesh mask ties the random mask at F1 87.7.
Significance. If the empirical claims were statistically supported, the paper would provide a useful practical observation for masked image modeling on small medical imaging datasets: for SparK pretraining on brain CT, patch-level masks such as random and mesh appear preferable to block-level masks. The mesh-mask algorithm is clearly described, and the paper includes a Grad-CAM sanity check and a sparsity ablation. However, the proposed mesh mask does not outperform the existing random mask, so the methodological novelty does not translate into an empirical gain. The main limitations are the absence of uncertainty quantification, the small number of positive test samples, post-hoc selection on the evaluation split, and apparent inconsistencies in the reported tables; these issues undermine the central claim as currently stated.
major comments (4)
- [§3.1, Table 1; §7, Tables 2–8] The reported recall values in Tables 2–8 are all consistent with the 26-positive Evaluation split (e.g., 96.2 = 25/26, 92.3 = 24/26, 88.5 = 23/26), not the 21-positive Test split listed in Table 1. The manuscript never states which split is used for the results in Section 7, so it appears that the central comparison is performed on the evaluation/validation split rather than a held-out test set. This must be clarified, and the held-out test results must be reported, because Section 8 bases its conclusion on these numbers.
- [§7, Tables 5–8] No confidence intervals, error bars, significance tests, or repeated seeds are reported. With only 21 (or 26) positive test images, a single true positive changes recall by roughly 4–5 points, which is the same magnitude as the 3–5 point F1 gaps used to distinguish patch-level from block-level masks. The claim that masking pattern has a non-negligible effect is therefore not supported by the reported point estimates; additional runs and an appropriate uncertainty or significance analysis are needed.
- [Table 6] The block-wise mask rows for mask ratios 40%, 60%, and 80% report identical precision (75.0), recall (92.3), and F1 (82.8). This is implausible for three independent pretraining/downstream runs and suggests a copy/paste or reporting error. The authors should verify these numbers and report the actual results for each ratio.
- [§6 and §7] The choice of RandomResizedCrop in Section 6 and the selection of the best mask ratio per mask type in Section 7 appear to be made after inspecting results on the same Evaluation split used for comparison. This is post-hoc selection on the evaluation split and can inflate apparent differences. Model selection should be performed on validation data, and the final comparison should be reported on a held-out test split.
minor comments (5)
- [Abstract] There is a typo: "poplular" should be "popular."
- [Table 1 and §3.1] The text refers to "validation data" while Table 1 labels the split as "Evaluation"; the terminology should be made consistent.
- [Section 5, Algorithm 1 and equations (8)–(9)] The image is divided into a 7×7 grid of patches, so the coordinate sets in lines 3–4 and equations (8)–(9) should range over {0,1,2,3,4,5,6}, not {0,1,2,...,7}.
- [§2.2 and §6] The paper reports SimMIM's random-mask accuracy as 83.0% in Section 2.2 but later states in Section 6 that SimMIM achieved its highest accuracy of 82.6% with a 2×2 square mask; these numbers should be reconciled.
- [Tables 2–4] The tables are introduced as "Results" without stating the evaluation split; the table captions should specify whether they refer to Training, Evaluation, or Test.
Circularity Check
No significant circularity: the paper reports direct experimental comparisons and does not derive its conclusions from fitted inputs or self-referential definitions.
full rationale
The paper's central claim, that pretraining mask pattern affects downstream F1 for SparK on brain CT, is supported by direct empirical comparisons in Tables 5-8. The mesh mask is defined by an explicit geometric algorithm (Algorithm 1) with no dependence on the outcome metric, and the random-mask baseline is an in-experiment condition, not a fitted prediction. No parameter is fitted to a subset of the data and then renamed as a prediction; no uniqueness theorem or ansatz is imported from the authors' own prior work; and the self-citations to SparK, SimMIM, and BEiT serve only as background and experimental setting, not as load-bearing derivations. The reported identical F1 values across some mask ratios and the small test-set size are concerns about statistical reliability and reporting consistency, but they are not circularity. The derivation chain is therefore self-contained as an experimental study.
Assumptions & free parameters
free parameters (2)
- MFF pre-training learning rate =
2.4e-4
- SparK pre-training learning rate and number of epochs =
1e-4, 700 epochs
assumptions (3)
- domain assumption The 179-slice test set with 21 positives yields reliable F1 estimates.
- domain assumption Pre-training and downstream slices are patient-disjoint.
- domain assumption Sparse ConvNeXt prevents mask pattern loss during MIM pre-training.
Cite this review
Pith. "Pith review of Thoughts on Objectives of Sparse and Hierarchical Masked Image Model." pith.science (2026). https://pith.science/paper/4E4HY6XP
@misc{pith2026250508819,
author = {Pith},
title = {Pith review of: Thoughts on Objectives of Sparse and Hierarchical Masked Image Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/4E4HY6XP}},
note = {Machine review of arXiv:2505.08819}
}
read the original abstract
Masked image modeling is one of the most poplular objectives of training. Recently, the SparK model has been proposed with superior performance among self-supervised learning models. This paper proposes a new mask pattern for this SparK model, proposing it as the Mesh Mask-ed SparK model. We report the effect of the mask pattern used for image masking in pre-training on performance.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Bao, H.; Dong, L.; Piao, S.; and Wei, F. 2022. BEiT: BERT Pre-Training of Image Transformers. arXiv preprint arXiv:2106.08254
arXiv 2022
-
[4]
Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020. A Simple Framework for Contrastive Learning of Visual Representations. arXiv preprint arXiv:2002.05709
arXiv 2020
-
[5]
Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09
work page 2009
-
[6]
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
arXiv 2018
-
[7]
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations
2021
-
[8]
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385
arXiv 2015
Show all 20 references
-
[9]
Hoefler, T.; Alistarh, D.; Ben-Nun, T.; Dryden, N.; and Peste, A. 2021. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint arXiv:2102.00554
2021 arXiv
-
[10]
Liu, Y.; Zhang, S.; Chen, J.; Yu, Z.; Chen, K.; and Lin, D. 2023. Improving Pixel-based MIM by Reducing Wasted Modeling Capability. arXiv preprint arXiv:2308.00261
2023 arXiv
-
[11]
Liu, Z.; Mao, H.; Wu, C.-Y.; Feichtenhofer, C.; Darrell, T.; and Xie, S. 2022. A ConvNet for the 2020s. arXiv preprint arXiv:2201.03545
2022 arXiv
-
[12]
OpenMMLab. 2023. MMPreTrain version 1.1.0. GitHub. https://github.com/open-mmlab/mmpretrain/tree/a4c219e05d3ab78c20b9d22dedde7dded6fd206c
2023
-
[13]
pytorch. 2024. PyTorch. GitHub. Https://github.com/pytorch/pytorch/releases/tag/v2.4.1
2024
-
[14]
R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D
Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2019. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. International Journal of Computer Vision, 128(2): 336–359
2019
-
[15]
Shorten, C.; and Khoshgoftaar, T. M. 2019. A survey on image data augmentation for deep learning. Journal of big data, 6(1): 1--48
2019
-
[16]
Tian, K.; Jiang, Y.; Diao, Q.; Lin, C.; Wang, L.; and Yuan, Z. 2023. Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling. arXiv preprint arXiv:2301.03580
2023 arXiv
-
[17]
N.; Kaiser, L.; and Polosukhin, I
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention Is All You Need. CoRR, abs/1706.03762
2017 arXiv
-
[18]
Xie, Z.; Zhang, Z.; Cao, Y.; Lin, Y.; Bao, J.; Yao, Z.; Dai, Q.; and Hu, H. 2022. SimMIM: A Simple Framework for Masked Image Modeling. arXiv preprint arXiv:2111.09886
2022 arXiv
-
[19]
J.; Chun, S.; Choe, J.; and Yoo, Y
Yun, S.; Han, D.; Oh, S. J.; Chun, S.; Choe, J.; and Yoo, Y. 2019. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
2019
-
[20]
N.; and Lopez-Paz, D
Zhang, H.; Cisse, M.; Dauphin, Y. N.; and Lopez-Paz, D. 2018. mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations
2018
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.