Pith. sign in

REVIEW 4 major objections 6 minor 30 references

From Pixels to Components: Eigenvector Masking for Visual Representation Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Masking random principal components of an image, instead of random patches of pixels, produces representations that transfer better to classification and makes the masking ratio easier to tune.

desk verdict A simple, sensible PCA-masking idea with consistent gains on small benchmarks, but missing error bars, a suspect aggregate fine-tuning number, and no scale validation mean the broad claims outrun the evidence. read the letter →

arxiv 2502.06314 v2 pith:BI7GZHOI submitted 2025-02-10 cs.LG cs.AIcs.CV

Alice Bizeul , Thomas Sutter , Alain Ryser , Bernhard Schölkopf , Julius von Kügelgen , Julia E. Vogt This is my paper · ORCID
classification cs.LGcs.AIcs.CV
keywords maskedimagemodelingself-supervisedlearningprincipalcomponentanalysisvisiontransformermaskingstrategyrepresentationclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that masking random principal components, rather than random pixel patches, is a better self-supervised pretext task for learning image representations. It proposes PMAE, which projects images into principal-component space, masks a random subset of components that account for a fixed fraction of the data variance, and trains a ViT autoencoder to reconstruct the hidden components from the visible ones. Because principal components carry global information, the reconstruction task is argued to involve high-level scene content instead of local texture or background redundancy. Across CIFAR10, TinyImageNet, and three MedMNIST datasets, the paper reports consistently higher classification accuracy than the standard MAE, even when the masking ratio is randomized and untuned. If true, this makes PCA-based masking a simple drop-in replacement for patch masking in masked image modeling.

What carries the argument

The central object is the principal masked autoencoder (PMAE), whose machinery is an invertible transformation into principal-component space followed by random component masking. The masking ratio is defined as the fraction of dataset variance explained by the masked components, turning the abstract notion of task complexity into an interpretable and directly tunable quantity. Visible components are mapped back to pixel space with the inverse PCA transform, processed by a standard ViT encoder-decoder, and the loss is the L2 error between the predicted and true masked components in principal-component space.

What would settle it

A concrete test would use a dataset whose principal components are statistically independent, such as images sampled from independent Gaussian coefficients: if PMAE still reconstructs masked components and beats patch-masked MAE on such data, the paper's stated mechanism is not what drives the gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that the masking operation in masked image modeling need not happen in pixel space. By applying a fixed, parameter-free PCA transform, masking a random subset of principal components that explain a chosen fraction of the data variance, and reconstructing those components from the rest, a ViT encoder-decoder learns representations that transfer better to classification than those trained with spatial patch masking. The paper also reports that the widely used default of masking 75% of pixel patches is suboptimal on all five datasets it tests, while masking about 20% of the variance in PCA space is near-optimal across datasets and the method is less sensitive to the masking-ratio hyperparameter.

Load-bearing premise

The method assumes that the different directions of variation found by PCA can be predicted from one another; if they were statistically independent, reconstructing the hidden components from the visible ones would be impossible and the improvement over patch masking would disappear.

Editorial extensions

If this is right

  • Masked autoencoders can adopt PCA masking as a drop-in change without needing new architectures, auxiliary supervision, or a different loss family.
  • The standard default of masking 75% of patches is suboptimal on the tested datasets, and a variance-based ratio near 20% is more robust across datasets.
  • Because components are global, masking them avoids both complete object occlusion and redundant background information, keeping the reconstruction task informative even at high masking levels.
  • The transformation-based masking view extends beyond images to any modality with an applicable PCA decomposition, and to self-supervised pipelines that currently rely on cropping or masking as view generators.
  • Reconstructing in principal-component space matters: computing the reconstruction loss in pixel space still beats MAE but by a smaller margin, indicating that the choice of target space contributes to the gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the stated mechanism: if a dataset's principal components were statistically independent of one another, reconstructing masked components from visible ones would be ill-posed, so PMAE should lose its advantage; measuring mutual information between masked and visible component sets across the paper's datasets would sharpen the claim.
  • Because the variance-explained ratio is dataset-independent, the same roughly-20-percent rule may transfer to other invertible transforms such as Fourier or wavelet bases, which would indicate that global components in general, not PCA specifically, drive the improvement.
  • The appendix's kernel PCA result hints that nonlinear transformations could push the approach further, pointing toward a learned or adaptive transform optimized for the masking objective rather than for raw variance.
  • The observation that evaluation-time augmentations hurt PMAE more than MAE suggests that component-masked representations encode different invariances, a property worth considering when choosing downstream evaluation pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Principal Masked Autoencoders (PMAE), a masked image modeling variant in which masking is performed on principal components rather than on spatial patches. The method first computes a PCA basis on the training set, then masks a random subset of components that account for a fixed ratio of explained variance (with an oracle ratio variant and a randomized-ratio variant). The encoder sees the visible components projected back to pixel space; the decoder reconstructs the masked components in PC space. The paper reports classification gains over standard MAE across CIFAR10, TinyImageNet, and three MedMNIST datasets, using linear, MLP, k-NN, and fine-tuning evaluation, and argues that the masking ratio is more robust and interpretable than the patch-masking ratio. The authors also provide ablations (reconstruction in pixel vs. PC space, kernel PCA) and release code.

Significance. If the reported gains hold, PMAE is a simple, hyperparameter-light alternative to spatial masking in masked image modeling. The randomized-ratio variant (PMAErd) outperforming oracle-tuned MAE on four of five datasets is a strong robustness result, and the exploration of a non-linear extension (kernel PCA) suggests a broader research direction. The paper is clearly written, includes useful ablations, and the promised code release supports reproducibility. The main caveats are the absence of uncertainty estimates and the limited evaluation scale, which leave the generality of the headline claim open.

major comments (4)
  1. [Table 1 and Section 5] The central empirical claim rests on accuracy differences reported in Table 1, but every number is a single run without standard deviations or the number of seeds. Without variance or significance information, a reader cannot tell whether the +14 percentage-point gain on CIFAR10 linear probing, or the smaller gains on other datasets, are real or within run-to-run noise. Please report means and standard deviations over at least three seeds, and ideally a paired test across the same seeds for MAE and PMAE.
  2. [Section 5, Table 1, and Appendix A.1.1] The experimental evidence is limited to 32x32 and 64x64 datasets with a ViT-Tiny backbone. Since MAE's behavior is known to be scale-dependent and the abstract claims PMAE is a 'simple and robust data-driven alternative' to MIM in general, the absence of any mid- or large-scale experiment (e.g., ImageNet-1K at 224x224, or at least a larger dataset such as ImageNet-100 with a ViT-S/B) leaves the external validity of the headline claim unestablished. Please add a larger-scale experiment or explicitly restrict the claim to small-scale, low-resolution settings.
  3. [Figure 5 and Section 5] The claim that PMAE is 'less sensitive to the choice of masking ratio hyperparameter' is supported by visual inspection of Figure 5 over a limited grid (10-50% variance for PMAE, 60-90% for MAE) on five datasets, with no quantitative sensitivity measure (e.g., variance or range of accuracies across ratios). Moreover, the oracle results tune on a held-out set, which is fair but should be complemented by a quantitative comparison of sensitivity. Please report a numerical summary of the ablation or soften the robustness claim.
  4. [Appendix A.1.5 and Appendix A.2.5] The linear probe evaluation applies data augmentations (random crop, flip) at evaluation time, which is nonstandard and can systematically affect learned representations differently. The paper notes in A.2.5 that PMAE accuracy increases substantially when augmentations are dropped, yet the main results keep augmentations for 'fairness.' Please clarify why this protocol was chosen and report results under the standard evaluation protocol (no augmentation at test time) as well, since this could change the magnitude or even the ordering of the reported comparisons.
minor comments (6)
  1. [Section 7] In the Masked Image Modelling paragraph, 'asked Image Modeling' appears to be a typo for 'Masked Image Modeling.'
  2. [Figure 5] The x-axis tick labels in Figure 5 (e.g., '1020304050') are rendered without separators or percent signs, making the ratios hard to read; please format them as '10 20 30 40 50'.
  3. [Table 1 and throughout] Column headers such as 'TinyIN' are not spelled out; please define abbreviations in the caption or at first use in the text.
  4. [Appendix A.2.6] The RBF kernel coefficient is written as '3.10−4'; please use standard notation (e.g., 3 × 10⁻⁴) to avoid ambiguity.
  5. [Figure 12 and Section 4] The paper states in Section 4 that representations are trained for 800 epochs, but Figure 12 reports reconstructions after 1000 epochs; please reconcile these numbers.
  6. [Section 7] The reference to 'VQV AE' should read 'VQVAE'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method and its empirical claims are self-contained; the randomized-ratio variant makes the main comparison independent of fitted hyperparameters.

full rationale

The paper does not present a formal derivation whose conclusion is equivalent to its premises; it proposes an empirical method (masking principal components instead of pixel patches) and evaluates it against MAE baselines. The only quantities fitted to downstream performance are the oracle masking ratios (PMAE_ocl and MAE_ocl), and these are explicitly labeled as oracle settings. The paper's central comparison does not rely on those fitted ratios, because PMAE_rd, with the masking ratio sampled uniformly per batch and no hyperparameter tuning, independently outperforms the MAE baselines across datasets (Table 1: e.g., CIFAR10 linear probing 44.0 vs. MAE_std 41.7 and MAE_rd 41.9; BloodMNIST 90.0 vs. 73.4/83.2). No 'prediction' reduces to a fitted quantity by construction. The self-citation Bizeul et al., 2024 appears only in Related Work as a contextual example of generative SSL and is not load-bearing. The paper's reliance on Balestriero & LeCun (2024) and Chen et al. (2024b) is for motivation and prior insight, not to prove the paper's central claim. The empirical premise that principal components are non-linearly predictive of each other is stated explicitly (Section 2) and is supported by reconstruction examples and the downstream results rather than by an assumption smuggled in from the authors' own prior work. There is no uniqueness theorem, no fitted parameter renamed as a prediction, and no derivation chain in which Eq. (3.1) reduces to Eq. (2.1) except for the intentional special case t = identity, which is acknowledged. The concerns about small-dataset scope and lack of ImageNet-scale evidence are external-validity risks, not circularity. The appropriate finding is therefore no significant circularity, score 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new entities. It relies on standard PCA mathematics and two empirical assumptions about image statistics and the MIM objective. The masking ratio is a tunable hyperparameter, but the randomized variant demonstrates robustness to it.

free parameters (2)
  • masking ratio r for PMAE oracle (percentage of variance masked) = varies per dataset (approximately 10-30%)
    Tuned on held-out linear probe performance for the PMAEocl results. The PMAErd variant shows the method works without tuning, reducing the load on this parameter.
  • RBF kernel coefficient for KMAE (appendix) = 3e-4
    Used in the appendix's kernel PCA experiment. Preliminary and not load-bearing for the main claim.
assumptions (3)
  • standard math PCA is a lossless invertible linear transformation of images when L=D
    Background result used in Section 3 to define t(x)=xV and the inverse transformation.
  • domain assumption Principal components of natural and medical images are non-linearly predictive of each other
    Stated in Section 2: uncorrelatedness does not imply independence. The feasibility of reconstructing masked PCs from visible PCs rests on this. Supported only by reconstruction examples in Appendix A.2.1.
  • domain assumption Shared information between visible and masked views in MIM should contain high-level variables for good downstream features (Kong et al. hypothesis)
    Used in Sections 2 and 6 to argue why masking in PC space aligns with classification. Not proven by the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Pixels to Components: Eigenvector Masking for Visual Representation Learning." pith.science (2026). https://pith.science/paper/BI7GZHOI

@misc{pith2026250206314,
  author       = {Pith},
  title        = {Pith review of: From Pixels to Components: Eigenvector Masking for Visual Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BI7GZHOI}},
  note         = {Machine review of arXiv:2502.06314}
}
read the original abstract

Predicting masked from visible parts of an image is a powerful self-supervised approach for visual representation learning. However, the common practice of masking random patches of pixels exhibits certain failure modes, which can prevent learning meaningful high-level features, as required for downstream tasks. We propose an alternative masking strategy that operates on a suitable transformation of the data rather than on the raw pixels. Specifically, we perform principal component analysis and then randomly mask a subset of components, which accounts for a fixed ratio of the data variance. The learning task then amounts to reconstructing the masked components from the visible ones. Compared to local patches of pixels, the principal components of images carry more global information. We thus posit that predicting masked from visible components involves more high-level features, allowing our masking strategy to extract more useful representations. This is corroborated by our empirical findings which demonstrate improved image classification performance for component over pixel masking. Our method thus constitutes a simple and robust data-driven alternative to traditional masked image modeling approaches.

Figures

Figures reproduced from arXiv: 2502.06314 by the authors.

Figure 1
Figure 1. From Pixels to Components. Masked image modeling involves reconstructing masked-out patches of pixels from visible ones. Instead of masking in pixel space (top), we propose applying random masks to a transformed version of the image, specifically to its principal component representation (bottom). Two disjoint sets of components are used as input and reconstruction target. on downstream tasks, such as image classifi… view at source ↗
Figure 2
Figure 2. Overview of the Principal Masked Autoencoder. A principal masked autoencoder (PMAE) differs from a vanilla MAE (He et al., 2021) by performing the masking in the space of principal components xPC = PCA(x) rather than in pixel space. The visible principal components m ⊙ xPC are then projected back into the observation space and serve as the input for a ViT encoder-decoder architecture. Masked principal components, (1… view at source ↗
Figure 3
Figure 3. Masking in pixel space. Image (left) with a random spatial mask partially removing relevant information (middle) and removing all semantic information (right). The latter is an example in which MIM would likely fail to learn useful representations. 2. Background Principal Component Analysis. Principal component analysis (PCA; Pearson, 1901; Hotelling, 1933) identifies components in data that ex￾hibit the highest var… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Mask Design in PMAE. 1. Perform PCA 2. For each batch, randomly shuffle the principal components and select a subset to construct the input (light blue), while the remaining components are used to create the reconstruction target (dark blue). In PMAEocl, the input comp…
Figure 5
Figure 5. Figure 5: Impact of the Masking Ratio. MAE (top) and PMAE (bottom) linear probing accuracy for varying masking ratios. The masking ratio is a sensitive and data-dependent hyper-parameter. While for MAE a clear masking guideline is hard to extract, for PMAE we observe close-to-op…
Figure 6
Figure 6. Figure 6: Performance across masking ratios. Average and stan￾dard error of the linear probe accuracy across masking ratios. patches share any information with the object. On the contrary, if the masked-out information is redundant with the information carried by visible patches…
Figure 7
Figure 7. Figure 7: From Principal Components to Spatial Features. Overview of the spatial features associated with distinct regions of the principal component spectrum; Images depict the features captured by the top (light blue), middle (mild blue) and bottom (dark blue) PCs. 8. Discussi…
Figure 8
Figure 8. Figure 8: MedMNIST datasets. Example images from the (from left to right) DermaMNIST, PathMNIST, and BloodMNIST datasets used for image classification. We apply an equivalent data augmentation strategy to all datasets and for all learning objectives during training; Following He…
Figure 9
Figure 9. Figure 9: Mask Design Strategies. An overview of the different mask design strategies used in our experimental setup: spatial masking (green) and principal component masking (blue). std refers to the standard approach of masking out 75% of image patches, ocl denotes masking with…
Figure 10
Figure 10. Figure 10: ViT-T/8 hyperparameters. We train a tiny Vision Transformer encoder architecture (ViT-T) with image patch size 8×8 for all datasets (ViT-T/8). The specifics of this architecture can be found in [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Training time. We report the training time in minutes for 800 training epochs using a ViT-T/8 architecture. For standard MAE we report numbers for various masking ratios. A.2. Additional Results A.2.1. RECONSTRUCTION OF MASKED INFORMATION [PITH_FULL_IMAGE:figures/ful…
Figure 12
Figure 12. Figure 12: depicts the output of the decoder networks after 1000 training epochs for both MAE and PMAE. Interestingly, we observe that for PMAE, the model is able to well estimate the masked out principal components. Original Input Output Target MAE PMAE [PITH_FULL_IMAGE:figure…
Figure 13
Figure 13. Figure 13: TinyImageNet 200 400 600 800 75 80 85 90 95 MAE PMAE Epochs Lin. P robe Acc. (%) [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 15
Figure 15. Figure 15: DermaMNIST 200 400 600 800 86 88 90 92 94 96 MAE PMAE Epochs Lin. P robe Acc. (%) [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 18
Figure 18. Figure 18: Overview of the Principal Masked Autoencoder (PMAE) with reconstruction targets in the observation space. A Principal Masked Autoencoder (PMAE) differs from a vanilla MAE by performing the masking in the space of principal components xPC =PCA(x) rather than in the pix…
Figure 19
Figure 19. Figure 19: Impact of the Masking Ratio. (top) PMAE trained with Equation (A.1) linear probing accuracy for varying masking ratios. We observe a close-to-optimal performance across datasets for 10 to 20% of the variance masked. (bottom) PMAE trained with Equation (A.1) linear pro…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 12 canonical work pages

  1. [2]

    We use the linear lr scaling rule: lr = base lr×batchsize / 256 (Goyal et al., 2017)

    These hyperparameters are taken from (He et al., 2021). We use the linear lr scaling rule: lr = base lr×batchsize / 256 (Goyal et al., 2017). Note that for our oracle masking settings, we conduct ablation studies across a masking ratio range of [10, 90]. A.1.5. E VALUATION HYPERPARAMETERS We evaluate the learned representation (i.e.,[CLS] token) using a l...

    work page 2021

  2. [3]

    [Cited on p

    URL https://arxiv.org/pdf/2204.01678. [Cited on p. 7.] Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M. data2vec: A general framework for self-supervised learning in speech, vision and language. In Proceedings of the 39th International Conference on Machine Learn- ing,

    arXiv

  3. [4]

    [Cited on p

    URL http://arxiv.org/abs/2106.08254. [Cited on p. 1, 3, and 7.] Bengio, Y ., Courville, A., and Vincent, P. Representa- tion learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intel- ligence,

    arXiv

  4. [7]

    [Cited on p

    URL https: //arxiv.org/abs/2202.04200. [Cited on p. 7.] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, 2020a. URL http://proceedings.mlr. press/v119/chen20j/chen20j.pdf. [Cited on p. 7 and 8.] Chen, X., Fan, H., Girshick, R., and...

    arXiv 2003

  5. [8]

    Example images from the (from left to right) DermaMNIST, PathMNIST, and BloodMNIST datasets used for image classification

    MedMNIST datasets. Example images from the (from left to right) DermaMNIST, PathMNIST, and BloodMNIST datasets used for image classification. We apply an equivalent data augmentation strategy to all datasets and for all learning objectives during training; Following He et al. (2021), our augmentation strategy consists of a random cropping followed by imag...

    work page 2021

  6. [10]

    [Cited on p

    URL https://arxiv.org/pdf/ 2111.12710. [Cited on p. 1, 3, and 7.] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the Tenth International Co...

    arXiv

  7. [13]

    [Cited on p

    URL http:// arxiv.org/abs/2111.06377. [Cited on p. 1, 2, 3, 4, 5, 6, 7, 12, 13, and 16.] Hotelling, H. Analysis of a complex of statistical variables into principal components. Journal of educational psy- chology,

    arXiv

  8. [14]

    [Cited on p

    URL https://link.springer.com/chapter/ 10.1007/978-3-031-20056-4_18 . [Cited on p. 2, 7, and 8.] Kong, L., Ma, M. Q., Chen, G., Xing, E. P., Chi, Y ., Morency, L.-P., and Zhang, K. Understanding masked autoencoders via hierarchical latent variable models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

    work page doi:10.1007/978-3-031-20056-4_18

Show all 30 references
  1. [15]

    [Cited on p

    URL https: //arxiv.org/abs/2306.04898. [Cited on p. 1, 2, 3, 6, and 8.] Lehner, J., Alkin, B., F¨urst, A., Rumetshofer, E., Miklautz, L., and Hochreiter, S. Contrastive tuning: A little help to make masked autoencoders forget. In Proceedings of the AAAI Conference on Artificia...

    arXiv

  2. [16]

    [Cited on p

    URL http://arxiv.org/abs/2304.10520. [Cited on p. 7.] Li, G., Zheng, H., Liu, D., Wang, C., Su, B., and Zheng, C. SemMAE: Semantic-guided masking for learning masked autoencoders. Advances in Neural Information Processing Systems, 2022a. URL https://arxiv. org/abs/2206.10207. ...

    arXiv

  3. [17]

    [Cited on p

    URL https://arxiv.org/ pdf/2006.08218. [Cited on p. 7.] Madan, N., Ristea, N.-C., Nasrollahi, K., Moeslund, T. B., and Ionescu, R. T. CL-MAE: Curriculum-learned masked autoencoders. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion,

    2006 arXiv

  4. [18]

    [Cited on p

    URL https: //arxiv.org/pdf/2304.07193. [Cited on p. 7 and 8.] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A. A. Context encoders: Feature learning by in- painting. In Proceedings of the IEEE conference on computer vision and pattern recognition ,

    arXiv

  5. [21]

    [Cited on p

    URL https://arxiv.org/abs/ 2201.13100. [Cited on p. 2 and 7.] Touvron, H., Cord, M., Douze, M., Massa, F., Sablay- rolles, A., and J ´egou, H. Training data-efficient image transformers & distillation through attention. In Inter- national conference on machine learning ,

    arXiv

  6. [22]

    [Cited on p

    URL https://arxiv.org/abs/2012.12877. [Cited on p. 5.] Turk, M. and Pentland, A. Eigenfaces for Recognition.Jour- nal of Cognitive Neuroscience,

    2012 arXiv

  7. [24]

    [Cited on p

    URL https: //arxiv.org/abs/2304.05919. [Cited on p. 7.] Xie, Z., Zhang, Z., Cao, Y ., Lin, Y ., Bao, J., Yao, Z., Dai, Q., and Hu, H. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion,

    arXiv

  8. [26]

    [Cited on p

    URL http://arxiv.org/ abs/2310.01994. [Cited on p. 1.] Zhang, Q., Wang, Y ., and Wang, Y . How mask matters: Towards theoretical understandings of masked autoen- coders. Advances in Neural Information Processing Systems,

    arXiv

  9. [27]

    [Cited on p

    URL https://arxiv.org/abs/ 2210.08344. [Cited on p. 1 and 3.] Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre-training with on- line tokenizer. In International Conference on Learning Representations,

    arXiv

  10. [28]

    [Cited on p

    URL https://arxiv.org/ pdf/2111.07832. [Cited on p. 1, 3, and 7.] 11 Eigenvector Masking for Visual Representation Learning A. Appendix A.1. Experimental Setup A.1.1. D ATASETS CIFAR-10 is a widely used benchmark dataset containing 50,000 training and 10,000 validation 32×32 R...

    arXiv

  11. [30]

    Note that while for MAE, we keep the normalized pixel values of each masked patch as reconstruction targets (He et al., 2021), we find it to not have a clear positive impact for PMAE and hence enforce the reconstruction of the raw target values for PMAE. A.1.4. T RAINING HYPER...

    2021

  12. [32]

    We also use the linear lr scaling rule: lr = base lr×batchsize / 256 (Goyal et al., 2017)

    The same hyperparameters were used for the MLP probing. We also use the linear lr scaling rule: lr = base lr×batchsize / 256 (Goyal et al., 2017). For the fine-tuning setup, we also follow (He et al., 2021), we fine-tune the encoder and a linear probe for 100 epochs and resort...

    2017

  13. [1991]

    [Cited on p

    URL https:// doi.org/10.1162/jocn.1991.3.1.71. [Cited on p. 7.] Wang, H., Song, K., Fan, J., Wang, Y ., Xie, J., and Zhang, Z. Hard patches mining for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

    1991 doi

  14. [1992]

    1137/1.9781611970104.fm

    URL https://epubs.siam.org/doi/pdf/10. 1137/1.9781611970104.fm. [Cited on p. 8.] Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computatio...

    2019

  15. [1997]

    [Cited on p

    URL https://link.springer.com/chapter/ 10.1007/BFb0020217. [Cited on p. 8 and 16.] Shi, Y ., Siddharth, N., Torr, P., and Kosiorek, A. R. Adver- sarial masking for self-supervised learning. In Proceed- ings of the 39th International Conference on Machine Learning,

    doi

  16. [2013]

    [Cited on p

    URL https://arxiv.org/pdf/ 1206.5538. [Cited on p. 3.] Bizeul, A., Sch ¨olkopf, B., and Allen, C. A probabilistic model to explain self-supervised representation learning. Transactions on Machine Learning Research,

    arXiv

  17. [2016]

    [Cited on p

    URL https://arxiv.org/pdf/1604.07379. [Cited on p. 1, 3, and 7.] Pearson, K. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Sci- ence,

    arXiv

  18. [2017]

    [Cited on p

    URL https://arxiv.org/abs/1706.02677. [Cited on p. 13.] He, K., Chen, X., Xie, S., Li, Y ., Doll´ar, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ,

    arXiv

  19. [2019]

    [Cited on p

    URL https: //arxiv.org/abs/1810.04805. [Cited on p. 1.] 9 Eigenvector Masking for Visual Representation Learning Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F., Yu, N., and Guo, B. Peco: Percep- tual codebook for bert pre-training of vision tran...

    arXiv

  20. [2022]

    [Cited on p

    URL http://arxiv.org/abs/2204.07141. [Cited on p. 7.] Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y ., and Ballas, N. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Confe...

    arXiv

  21. [2023]

    [Cited on p

    URL https://arxiv.org/abs/2301.08243. [Cited on p. 7 and 8.] Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A. Mul- timae: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vision,

    arXiv

  22. [2024]

    [Cited on p

    URL https://arxiv.org/pdf/2402.01399. [Cited on p. 7.] Bracewell, R. and Kahn, P. B. The fourier transform and its applications. American Journal of Physics,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.