Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Masked Autoencoders for Ultrasound Signals: Robust Representation Learning for Downstream Applications

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Masked Autoencoders pre-trained on unlabeled synthetic ultrasound signals learn representations that transfer to real measurements, lifting time-of-flight classification accuracy from 68% to 87% and beating CNNs most clearly when labeled da

desk verdict First clean MAE-for-1D-ultrasound study with careful experiments, but the synthetic-vs-real pre-training claim is confounded by a 5x dataset-size gap. read the letter →

arxiv 2508.20622 v1 pith:GIJES637 submitted 2025-08-28 cs.LG cs.CV

Immanuel Ro{ss}teutscher , Klaus S. Drese , Thorsten Uphues This is my paper
classification cs.LGcs.CV
keywords maskedautoencoderself-supervisedlearningultrasoundsignalprocessingtime-of-flightclassificationvisiontransformersynthetic-to-realtransferrepresentationnon-destructivetesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that Masked Autoencoders (MAEs), a self-supervised method that learns by hiding and reconstructing parts of the input, work on one-dimensional ultrasound signals, a domain where labeled data are scarce and signal processing is usually hand-crafted per application. The central claim is that pre-training the model on unlabeled synthetic ultrasound tone bursts, then fine-tuning on a small set of real measured signals, beats both training from scratch and pre-training on the real data itself. In the headline experiment, synthetic pre-training reaches 87.46% top-1 accuracy on the 200-class time-of-flight task (identifying the exact sample where a tone burst begins), versus 68.03% from scratch and 78.68% pre-trained on measured data. The advantage grows as labeled data shrink: with only 2,000 training samples the MAE scores 48.00% against 38.50% for a strong CNN baseline. If this holds, it offers a data-efficient route to general-purpose ultrasound analysis for industrial non-destructive testing without task-specific feature engineering.

What carries the argument

The load-bearing mechanism is the Masked Autoencoder adapted to 1D: a 512-sample signal is cut into non-overlapping patches, 75% are masked, and only visible patches enter the encoder; a small decoder reconstructs the masked patches under L1 loss. Reconstruction forces the model to infer missing parts of a periodic, noisy tone burst from context, teaching the physical features — frequency, onset, amplitude, noise — that downstream classification needs. Two adaptations carry the argument: non-square query/key/value matrices (head dim 64, model dim 128) keep attention subspaces from collapsing, and patch size 32 with 75% masking balances reconstruction difficulty against representation quality

What would settle it

Decisive test: hold architecture and fine-tuning fixed and pre-train on three synthetic corpora from explicitly specified generators — one matching the paper's parameter table, one matched to the measured signals (burst length 75–180 samples, 2.0–2.4 MHz), one with alien physics (different noise model, no ring-down). If real-signal accuracy stays near 87% across all three, transfer is robust and the diversity reading holds; if it collapses when the generator changes, the headline result depends on one unspecified generative model. Second check: fine-tune the same checkpoints on a different tra

Watch

Extended reading notes

Core claim

A Masked Autoencoder with a small Vision Transformer encoder (~1.2M parameters) learns transferable representations of raw 1D ultrasound by reconstruction; pre-training diversity matters more than realism. Pre-training on 60K unlabeled synthetic signals generalized to real measured signals whose burst length lay outside the synthetic range: 87.46% top-1 accuracy on the 200-class time-of-flight task, versus 78.68% after pre-training on 12K measured signals and 68.03% from scratch. Adding real data changed little (87.58%). The paper concludes that broad variation in frequency, amplitude, and burst length drives generalization.

Load-bearing premise

The transfer result assumes the synthetic signal generator reproduces the real transducer's physics closely enough that features learned from simulated bursts transfer to measured ones; the paper gives only parameter ranges (frequency, burst length, peak SNR) and never describes or verifies the generator.

Editorial extensions

If this is right

  • Pre-training on unlabeled synthetic signals substitutes for large labeled real datasets: with half the downstream data (6K samples) the pre-trained MAE holds 80.82% top-1 accuracy versus 74.45% for the optimized CNN baseline.
  • The data-efficiency advantage grows as labels shrink: at 2K training samples the MAE reaches 48.00% (51.25% with combined synthetic+real pre-training) against 38.50% for the strong CNN, whose naive CNN counterpart collapses to chance.
  • Small models suffice for ultrasound: the best encoder has roughly 1.2M parameters, which the paper argues suits edge deployment in industrial measurement, and reconstruction quality plateaus beyond the M-size model.
  • Synthetic diversity, not realism, drives the transfer: adding 12K measured signals to the 60K synthetic pool improves real-signal accuracy by only 0.12 percentage points (87.58% vs 87.46%).
  • Pre-training stabilizes training: standard deviation drops from ±3.16 points from scratch to ±0.30 after synthetic pre-training, and the overfitting visible in larger from-scratch models disappears.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same recipe — MAE pre-training on broad synthetic distributions, fine-tuning on limited real data — should transfer to other 1D sensing modalities with simulators (radar, sonar, seismics, machine vibration); the optimal masking ratio and patch size would need re-testing per modality, as would the paper's entropy-based size argument.
  • The paper treats the synthetic generator as a black box; its own 'diversity, not realism' reading is testable by pre-training on synthetic corpora whose ranges deliberately mismatch the real transducer (e.g., frequencies outside 2.0–2.4 MHz) and watching how the transfer gap moves.
  • Because measured-signal labels come from cross-correlation with the known excitation signal, the real-data numbers inherit a matched-filter ceiling; whether the MAE's 87.46% approaches or exceeds the labeling accuracy itself is an open question that bears on how much headroom the method has.
  • The results sketch, without testing, a scaling path to an ultrasound foundation model: if synthetic diversity is the driver, much larger corpora spanning transducer types, propagation paths, and noise regimes should push transfer further — a claim the presented data can neither confirm nor refute.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper adapts Masked Autoencoders (MAEs) with Vision Transformer backbones to one-dimensional raw ultrasound signals. Pre-training is performed on unlabeled synthetic tone bursts; the encoder is then fine-tuned for a 200-class absolute time-of-flight (ToF) classification task on synthetic and on real measured ultrasound data. The authors systematically vary model size, attention-head dimensionality, patch size, and masking ratio, reporting ten-run means and standard deviations. Their central empirical claims are (i) MAE pre-training substantially improves downstream ToF accuracy over training from scratch and over two CNN baselines, (ii) this advantage grows when labeled downstream data are scarce, and (iii) pre-training on synthetic data transfers to real signals better than pre-training on measured data alone.

Significance. The study is potentially useful for industrial NDT/SHM, where labeled data are scarce and task-specific DSP is the norm; a 1D ultrasound MAE is a reasonable and underexplored direction. The paper has real strengths: ten-run averages with standard deviations, a two-stage design (hyperparameters chosen on synthetic data, then evaluated on real data), from-scratch comparisons, and explicit low-label-data curves. The authors also acknowledge in Section V that the method's breadth across downstream tasks and hyperparameter transferability remain open. However, the headline synthetic-versus-real pre-training comparison is currently confounded by dataset size, and the synthetic generation process is underspecified; the significance is conditional on the results surviving an equal-size, leakage-free comparison.

major comments (4)
  1. [Section IV-D, Table 6, and Table 1] The synthetic pre-training row uses 60K synthetic signals, while the measured pre-training row uses 12K measured signals; Table 1 confirms this 5x size difference (48K/6K/6K versus 9.6K/1.2K/1.2K). Because MAE pre-training is known to benefit from larger corpora, the 87.46% versus 78.68% gap cannot be attributed to the synthetic domain. Add an equal-size control (e.g., pre-train on a 12K subsample of synthetic data) and report all conditions; this is load-bearing for the abstract and for the conclusion that synthetic data 'enables the model to learn more generalized and robust representations.'
  2. [Section IV-D, Table 6] The 'real' pre-training condition is described as using 'measured data (12K samples)', which matches the total size of the measured dataset in Table 1 (train 9.6K + val 1.2K + test 1.2K). If pre-training includes the validation/test portions, then the pre-training corpus overlaps the fine-tuning/test data and the real pre-training baseline is contaminated. Specify exactly which split is used for pre-training; if only the training split is used, the stated 12K figure must be corrected to 9.6K.
  3. [Section III-C, Table 1] The synthetic data generator is not described: only parameter ranges are given (frequency, amplitude, burst length, peak SNR), with no equations for the tone burst envelope, noise type/color, quantization, or transducer-like impulse response. The central transfer claim depends on synthetic-to-real distributional similarity, so the experiment cannot be reproduced or its generality assessed from the manuscript. Provide the generative model or code, and ideally a quantitative spectral/statistical comparison between synthetic and measured signals.
  4. [Section IV-D, Table 7] The 'strong' CNN baseline is described only as a 'VGG-like' architecture with batch normalization, dropout, and a cosine learning rate scheduler, and the 'naive' baseline is even less specified. Since a central claim is that the MAE outperforms 'strong convolutional neural network (CNN) baselines optimized for the downstream task', the manuscript should include full architecture details (layers, kernel sizes, parameter counts, training schedule, hyperparameter selection) or release code. Without this, the comparison is not verifiable.
minor comments (5)
  1. [Table 7] In the 12K row, the syn and syn+real columns appear swapped relative to Table 6: Table 6 reports syn 87.46 and syn+real 87.58, while Table 7 reports syn 87.58 and syn+real 87.46. Please correct the inconsistency.
  2. [Section IV-A] The sentence 'no additional or alternative training data were used for the pre-trained models' is confusing because the models are pre-trained on synthetic data. Rephrase to clarify that no external checkpoints or data beyond the described datasets were used.
  3. [Section III-B] The Shannon-entropy comparison is computed from aggregate amplitude histograms, which conflates signal content with noise and does not measure the structural complexity relevant to the MAE reconstruction task. If kept, state this limitation explicitly, and consider reporting per-signal entropy statistics.
  4. [Tables 3-5] Several scratch baselines have very large standard deviations (e.g., 44.25 ± 18.81, 45.45 ± 19.84). Consider reporting paired differences or significance tests across the ten runs to support the claim of consistent improvement.
  5. [Section III-E, Table 6] Top-5 accuracy saturates at 100% in several key comparisons, providing little differentiation; top-2 accuracy or calibrated confidence intervals would be more informative for the ToF task.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims rest on held-out empirical comparisons against external baselines; the synthetic-vs-real pre-training comparison is confounded by dataset size, but that is a validity limitation, not a circular derivation.

full rationale

The paper's central comparisons are held-out empirical evaluations, not reductions. The pre-training loss (Eq. 1) is an L1 reconstruction over masked patches and contains no downstream ToF information; the ToF labels for measured signals are produced independently by cross-correlation (Eq. 5), and the fine-tuning accuracy is measured against these held-out labels. The model configuration (M-base, mask 75%, P=32) is selected on synthetic validation data in Sections IV-A through IV-C before the real-signal experiments, so the Table 6 results are not fit to the real test labels. References to MAE [17], ViT [2], and time-series MAEs are external prior work; there are no load-bearing self-citations, uniqueness theorems, or ansatz-smuggling citations. The main caveat is a size confound: Section III-C states "The synthetic dataset is five times larger" (60K vs. 12K), and Section IV-D compares pre-training on synthetic 60K with pre-training on measured 12K, so the "synthetic transfers better" claim is not size-controlled. That is a validity limitation, not circularity: the superior accuracy does not follow by construction from any fitted parameter or defined quantity. The paper also acknowledges open limitations in Section V ("its effectiveness across a diverse set of downstream tasks remains to be validated systematically"), consistent with an honest empirical study rather than a self-referential derivation.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities or conserved quantities. Its free parameters are hyperparameters and simulation ranges selected by hand; they do not define a parametric law. The key domain assumptions are the representativeness of the synthetic generator and the validity of cross-correlation labels, both load-bearing for the transfer claim and neither externally verified.

free parameters (2)
  • Synthetic signal distribution ranges = frequency 1.0-4.0 MHz; burst length 200-400 samples; peak SNR 18-38 dB; amplitude 0.2-1.0 normalized
    Hand-chosen ranges (Table 1) that define the pre-training dataset; the synthetic-to-real transfer result depends on these ranges spanning real signal variability.
  • Optimal hyperparameters (P=32, mask=0.75, M-dh64) = patch size 32, masking ratio 0.75, d_model=128, h=4, d_head=64, 6 encoder/2 decoder layers
    Selected empirically on the synthetic dataset (Sections IV-A to IV-C). These are standard model-selection choices, not fitted physical constants, but they shape the reported performance.
assumptions (5)
  • domain assumption MAE reconstruction of masked patches induces representations transferable to downstream ToF classification.
    The entire method assumes that masked-signal reconstruction features are useful for a supervised time-of-flight task; supported by prior work in vision but assumed for ultrasound.
  • domain assumption Synthetic ultrasound signals are representative of real measured ultrasound bursts.
    Load-bearing for the synthetic-to-real transfer claim; the generator is not fully specified, so this cannot be checked from the paper.
  • domain assumption Cross-correlation with the excitation signal yields the ground-truth absolute ToF label for measured signals.
    Used in Section III-D to label all real data; if cross-correlation is biased at low SNR, the labels (and hence reported accuracy) inherit that bias.
  • domain assumption Shannon entropy of amplitude distributions is a valid proxy for data complexity, justifying smaller models.
    Motivates the small-model design in Section III-B; not central to the empirical comparison, but an unproven heuristic.
  • standard math Standard MAE (He et al. 2022) and ViT (Dosovitskiy et al. 2021) architectures, training procedures, and optimizer settings work as assumed.
    The paper builds directly on these published methods and standard deep-learning practice (AdamW, cosine schedule, dropout).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Masked Autoencoders for Ultrasound Signals: Robust Representation Learning for Downstream Applications." pith.science (2026). https://pith.science/paper/GIJES637

@misc{pith2026250820622,
  author       = {Pith},
  title        = {Pith review of: Masked Autoencoders for Ultrasound Signals: Robust Representation Learning for Downstream Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GIJES637}},
  note         = {Machine review of arXiv:2508.20622}
}
read the original abstract

We investigated the adaptation and performance of Masked Autoencoders (MAEs) with Vision Transformer (ViT) architectures for self-supervised representation learning on one-dimensional (1D) ultrasound signals. Although MAEs have demonstrated significant success in computer vision and other domains, their use for 1D signal analysis, especially for raw ultrasound data, remains largely unexplored. Ultrasound signals are vital in industrial applications such as non-destructive testing (NDT) and structural health monitoring (SHM), where labeled data are often scarce and signal processing is highly task-specific. We propose an approach that leverages MAE to pre-train on unlabeled synthetic ultrasound signals, enabling the model to learn robust representations that enhance performance in downstream tasks, such as time-of-flight (ToF) classification. This study systematically investigated the impact of model size, patch size, and masking ratio on pre-training efficiency and downstream accuracy. Our results show that pre-trained models significantly outperform models trained from scratch and strong convolutional neural network (CNN) baselines optimized for the downstream task. Additionally, pre-training on synthetic data demonstrates superior transferability to real-world measured signals compared with training solely on limited real datasets. This study underscores the potential of MAEs for advancing ultrasound signal analysis through scalable, self-supervised learning.

Figures

Figures reproduced from arXiv: 2508.20622 by the authors.

Figure 1
Figure 1. FIGURE 1 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. FIGURE 2 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. FIGURE 3 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: FIGURE 4 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: FIGURE 5 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: FIGURE 6 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 46 canonical work pages

  1. [1]

    Attention Is All You Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need,” in Advances in Neural Information Processing Systems, vol. 30. Long Beach, California, USA: Curran Associates, Inc., 2017, pp. 5998–

    work page 2017

  2. [2]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in International Conference on Learning Representations (ICLR), 2021. [Online]. Available: https: //openreview.net/pdf?...

    work page 2021

  3. [3]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in The North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1. Minneapolis, Minnesota: Association for Computational Linguistics, 2019, pp. 4171–

    work page 2019

  4. [4]

    Improving Language Understanding by Generative Pre-Training,

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving Language Understanding by Generative Pre-Training,”

    work page

  5. [5]

    BEiT: BERT Pre-Training of Image Transformers,

    H. Bao, L. Dong, S. Piao, and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” in International Conference on Learning Representations (ICLR), 2022. [Online]. Available: https://openreview. net/pdf?id=p-BhZSz59o4

    work page 2022

  6. [6]

    DNABERT: pre- trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome,

    Y . Ji, Z. Zhou, H. Liu, and R. V . Davuluri, “DNABERT: pre- trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome,” Bioinformatics, vol. 37, no. 15, pp. 2112–2120, Aug. 2021. [Online]. Available: https://doi.org/10.1093/ bioinformatics/btab083

    work page 2021

  7. [7]

    wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,” in Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020, pp. 12 449–12 460. [Online]. Available: https://proceedings.neurips. cc/paper/2020/file/92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf

    work page 2020

  8. [8]

    FinBERT: A Pretrained Language Model for Financial Communications,

    Y . Yang, M. C. Siy UY , and A. Huang, “FinBERT: A Pretrained Language Model for Financial Communications,” 2020. [Online]. Available: https://arxiv.org/abs/2006.08097

    arXiv 2020

Show all 52 references
  1. [9]

    Don’t Pay Attention to the Noise: Learning Self-supervised Representations of Light Curves with a Denoising Time Series Transformer,

    M. Morvan, N. Nikolaou, K. H. Yip, and I. Waldmann, “Don’t Pay Attention to the Noise: Learning Self-supervised Representations of Light Curves with a Denoising Time Series Transformer,” 2022, arXiv:2207.02777 [astro-ph]. [Online]. Available: http://arxiv.org/abs/ 2207.02777

    2022 arXiv

  2. [10]

    A Systematic Review of Advanced Sensor Technologies for Non-Destructive Testing and Structural Health Monitoring,

    S. Hassani and U. Dackermann, “A Systematic Review of Advanced Sensor Technologies for Non-Destructive Testing and Structural Health Monitoring,” Sensors, vol. 23, no. 4, p. 2204, Feb. 2023. [Online]. Available: https://www.mdpi.com/1424-8220/23/4/2204

    2023

  3. [11]

    Structural Health and Condition Monitoring with Acoustic Emission and Guided Ultrasonic Waves: What about Long-Term Durability of Sensors, Sensor Coupling and Measurement Chain?

    A. J. Brunner, “Structural Health and Condition Monitoring with Acoustic Emission and Guided Ultrasonic Waves: What about Long-Term Durability of Sensors, Sensor Coupling and Measurement Chain?” Applied Sciences, vol. 11, no. 24, p. 11648, Dec. 2021. [Online]. Available: https...

    2021

  4. [12]

    Recent Advancements in Non-Destructive Testing Techniques for Structural Health Monitoring,

    P. Kot, M. Muradov, M. Gkantou, G. S. Kamaris, K. Hashim, and D. Yeboah, “Recent Advancements in Non-Destructive Testing Techniques for Structural Health Monitoring,” Applied Sciences, vol. 11, no. 6, p. 2750, Mar. 2021. [Online]. Available: https: //www.mdpi.com/2076-3417/11/6/2750

    2021

  5. [13]

    Review of Ultrasonic Ranging Methods and Their Current Challenges,

    Z. Qiu, Y . Lu, and Z. Qiu, “Review of Ultrasonic Ranging Methods and Their Current Challenges,” Micromachines, vol. 13, no. 4, p. 520, Mar

  6. [14]

    Analysis and Validation of Ultrasonic Probes in Liquid Level Monitoring Systems,

    W. Gao, W. Liu, F. Li, and Y . Hu, “Analysis and Validation of Ultrasonic Probes in Liquid Level Monitoring Systems,” Sensors, vol. 21, no. 4, p. 1320, Feb. 2021. [Online]. Available: https: //www.mdpi.com/1424-8220/21/4/1320

    2021

  7. [15]

    Design and Implementation of an Ultrasonic Flowmeter Based on the Cross-Correlation Method,

    R. Ren, H. Wang, X. Sun, and H. Quan, “Design and Implementation of an Ultrasonic Flowmeter Based on the Cross-Correlation Method,” 12 Immanuel Roßteutscher et al.: et al.: Masked Autoencoders for Ultrasound Signals Sensors, vol. 22, no. 19, p. 7470, Oct. 2022. [Online]. Avail...

    2022

  8. [16]

    A comprehensive review on accuracy in ultrasonic flow measurement using reconfigurable systems and deep learning approaches,

    S. K. Jagatheesaperumal, K. Arunachalam, K. Sundaram, S. Ganesan, and K. Ganesan, “A comprehensive review on accuracy in ultrasonic flow measurement using reconfigurable systems and deep learning approaches,” AIP Advances, vol. 10, no. 10, p. 105221, Oct

  9. [17]

    Masked Autoencoders Are Scalable Vision Learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA: IEEE, 2022, pp. 15 979–15 988. [Online]. Available: http://arxiv...

    2022 arXiv

  10. [18]

    SimMIM: a Simple Framework for Masked Image Modeling,

    Z. Xie, Z. Zhang, Y . Cao, Y . Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “SimMIM: a Simple Framework for Masked Image Modeling,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA: IEEE, 2022, pp. 9643–9653. [Online]. Available: h...

    2022

  11. [19]

    Generative Pretraining from Pixels,

    M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, “Generative Pretraining from Pixels,” in Proceedings of the 37th Inter- national Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 1691–1703

    2020

  12. [20]

    Emerging Properties in Self-Supervised Vision Transformers,

    M. Caron, H. Touvron, I. Misra, H. Jegou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging Properties in Self-Supervised Vision Transformers,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada: IEEE, 2021, pp. 9630–9640. [Online]. Avai...

    2021

  13. [21]

    Masked Autoencoders in Computer Vision: A Comprehensive Survey,

    Z. Zhou and X. Liu, “Masked Autoencoders in Computer Vision: A Comprehensive Survey,” IEEE Access, vol. 11, pp. 113 560– 113 579, 2023. [Online]. Available: https://ieeexplore.ieee.org/document/ 10278410/

    2023

  14. [22]

    Pose Mask: A Model-Based Augmentation Method for 2D Pose Estimation in Classroom Scenes Using Surveillance Images,

    S. Liu, M. Ma, H. Li, H. Ning, and M. Wang, “Pose Mask: A Model-Based Augmentation Method for 2D Pose Estimation in Classroom Scenes Using Surveillance Images,” Sensors, vol. 22, no. 21, p. 8331, Oct. 2022. [Online]. Available: https://www.mdpi.com/1424-8220/22/21/8331

    2022

  15. [23]

    An Advanced Chicken Face Detection Network Based on GAN and MAE,

    X. Ma, X. Lu, Y . Huang, X. Yang, Z. Xu, G. Mo, Y . Ren, and L. Li, “An Advanced Chicken Face Detection Network Based on GAN and MAE,” Animals, vol. 12, no. 21, p. 3055, Nov. 2022. [Online]. Available: https://www.mdpi.com/2076-2615/12/21/3055

    2022

  16. [24]

    Transfer Learning with Self-Supervised Vision Transformer for Large-Scale Plant Identification,

    M. Xu, S. Yoon, Y . Jeong, J. Lee, and D. S. Park, “Transfer Learning with Self-Supervised Vision Transformer for Large-Scale Plant Identification,” in Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, 2022. [Online]. Available: https://ceur-ws...

    2022

  17. [25]

    Cross-Attention Based Multi-Resolution Feature Fusion Model for Self-Supervised Cervical OCT Image Classification,

    Q. Wang, K. Chen, W. Dou, and Y . Ma, “Cross-Attention Based Multi-Resolution Feature Fusion Model for Self-Supervised Cervical OCT Image Classification,” IEEE/ACM Transactions on Computational Biology and Bioinformatics, vol. 20, no. 4, pp. 2541–2554, Jul. 2023. [Online]. Ava...

    2023

  18. [26]

    Self-supervised Learning Based on a Pre-trained Method for the Subtype Classification of Spinal Tumors,

    M. Jiao, H. Liu, Z. Yang, S. Tian, H. Ouyang, Y . Li, Y . Yuan, J. Liu, C. Wang, N. Lang, L. Jiang, H. Yuan, Y . Qian, and X. Wang, “Self-supervised Learning Based on a Pre-trained Method for the Subtype Classification of Spinal Tumors,” in Computational Mathematics Modeling i...

    2022

  19. [27]

    Self-supervised learning for medical image classification: a systematic review and implementation guidelines,

    S.-C. Huang, A. Pareek, M. Jensen, M. P. Lungren, S. Yeung, and A. S. Chaudhari, “Self-supervised learning for medical image classification: a systematic review and implementation guidelines,” Springer Nature, vol. 6, no. 1, p. 74, Apr. 2023. [Online]. Available: https://www.n...

    2023

  20. [28]

    Efficient Medical Image Assessment via Self-supervised Learning,

    C.-Y . Huang, Q. Lei, and X. Li, “Efficient Medical Image Assessment via Self-supervised Learning,” in Data Augmentation, Labelling, and Imperfections. Springer Nature Switzerland, 2022

    2022

  21. [29]

    Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification,

    J. Xiao, Y . Bai, A. Yuille, and Z. Zhou, “Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification,” in 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV). Waikoloa, HI, USA: IEEE, Jan. 2023, pp. 3577–3589. [Online]. Available: ht...

    2023

  22. [30]

    Unsupervised Anomaly Detection in Medical Images with a Memory-Augmented Multi-level Cross-Attentional Masked Autoencoder,

    Y . Tian, G. Pang, Y . Liu, C. Wang, Y . Chen, F. Liu, R. Singh, J. W. Verjans, M. Wang, and G. Carneiro, “Unsupervised Anomaly Detection in Medical Images with a Memory-Augmented Multi-level Cross-Attentional Masked Autoencoder,” in Machine Learning in Medical Imaging - MLMI ...

    2023

  23. [31]

    Self Pre-Training with Masked Autoencoders for Medical Image Classification and Segmentation,

    L. Zhou, H. Liu, J. Bae, J. He, D. Samaras, and P. Prasanna, “Self Pre-Training with Masked Autoencoders for Medical Image Classification and Segmentation,” in 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI). Cartagena, Colombia: IEEE, Apr. 2023, pp. 1–6. [...

    2023

  24. [32]

    How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks?

    J. Sosa, M. Aloulou, D. Rukhovich, R. Sleimi, B. Changaival, A. Kacem, and D. Aouada, “How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks?” Sep. 2024, arXiv:2409.18536 [cs]. [Online]. Available: http://arxiv.org/abs/2409. 18536

    2024 arXiv

  25. [33]

    Land Cover Classification for Polarimetric SAR Images Based on Vision Transformer,

    H. Wang, C. Xing, J. Yin, and J. Yang, “Land Cover Classification for Polarimetric SAR Images Based on Vision Transformer,” Remote Sensing, vol. 14, no. 18, p. 4656, Sep. 2022. [Online]. Available: https://www.mdpi.com/2072-4292/14/18/4656

    2022

  26. [34]

    SatViT: Pretraining Transformers for Earth Observation,

    A. Fuller, K. Millard, and J. R. Green, “SatViT: Pretraining Transformers for Earth Observation,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022. [Online]. Available: https://ieeexplore.ieee.org/ document/9866058/

    2022

  27. [35]

    Detecting Anomalous Events from Unlabeled Videos via Temporal Masked Auto-Encoding,

    J. Hu, G. Yu, S. Wang, E. Zhu, Z. Cai, and X. Zhu, “Detecting Anomalous Events from Unlabeled Videos via Temporal Masked Auto-Encoding,” in 2022 IEEE International Conference on Multimedia and Expo (ICME). Taipei, Taiwan: IEEE, Jul. 2022, pp. 1–6. [Online]. Available: https://...

    2022

  28. [36]

    Anomaly and activity recognition in a video surveillance using Masked Autoencoder,

    K. Datchanamoorthy, B. Padmavathi, V . Prakash, S. Yogesh, and S. Sahabudeen, “Anomaly and activity recognition in a video surveillance using Masked Autoencoder,” in 2022 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (...

    2022

  29. [37]

    Masked Autoencoders As Spatiotemporal Learners,

    C. Feichtenhofer, H. Fan, Y . Li, and K. He, “Masked Autoencoders As Spatiotemporal Learners,” in NeurIPS’22: Proceedings of the 36th International Conference on Neural Information Processing Systems. Association for Computing Machinery, 2022, pp. 35 946 – 35 958. [Online]. Av...

    2022

  30. [38]

    MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting,

    P. Tang and X. Zhang, “MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting,” in 2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI). Macao, China: IEEE, Oct. 2022, pp. 982–989. [Online]. Available: https: //ieeexplore.ieee.org...

    2022

  31. [39]

    Ti-MAE: Self-Supervised Masked Time Series Autoencoders,

    Z. Li, Z. Rao, L. Pan, P. Wang, and Z. Xu, “Ti-MAE: Self-Supervised Masked Time Series Autoencoders,” Jan. 2023, arXiv:2301.08871 [cs]. [Online]. Available: http://arxiv.org/abs/2301.08871

    2023 arXiv

  32. [40]

    GMAEEG: A Self-Supervised Graph Masked Autoencoder for EEG Representation Learning,

    Z. Fu, H. Zhu, Y . Zhao, R. Huan, Y . Zhang, S. Chen, and Y . Pan, “GMAEEG: A Self-Supervised Graph Masked Autoencoder for EEG Representation Learning,” IEEE Journal of Biomedical and Health Informatics, vol. 28, no. 11, pp. 6486–6497, Nov. 2024. [Online]. Available: https://i...

    2024

  33. [41]

    MAEEG: Masked Auto-encoder for EEG Representation Learning,

    H. Chien, H. Goh, C. M. Sandino, and J. Y . Cheng, “MAEEG: Masked Auto-encoder for EEG Representation Learning,” Oct. 2022, arXiv:2211.02625 [eess]. [Online]. Available: http://arxiv.org/abs/2211. 02625

    2022 arXiv

  34. [42]

    Wavefield MAE: Leveraging Large Vision Models for Ultrasonic Wavefield Pattern Analysis,

    J. Ye, T. Kobayashi, and N. Toyama, “Wavefield MAE: Leveraging Large Vision Models for Ultrasonic Wavefield Pattern Analysis,” in 2024 32nd European Signal Processing Conference (EUSIPCO). Lyon, France: IEEE, Aug. 2024, pp. 1856–1860. [Online]. Available: https://ieeexplore.ie...

    2024

  35. [43]

    Learning Multiple Layers of Features from Tiny Images,

    A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” Department of Computer Science, University of Toronto, Tech. Rep., 2009. [Online]. Available: https://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf

    2009

  36. [44]

    FlexiViT: One Model for All Patch Sizes,

    L. Beyer, P. Izmailov, A. Kolesnikov, M. Caron, S. Kornblith, X. Zhai, M. Minderer, M. Tschannen, I. Alabdulmohsin, and F. Pavetic, “FlexiViT: One Model for All Patch Sizes,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver, BC, Canada: ...

    2023

  37. [45]

    Exploring Long-Sequence Masked Autoencoders,

    R. Hu, S. Debnath, S. Xie, and X. Chen, “Exploring Long-Sequence Masked Autoencoders,” Oct. 2022, arXiv:2210.07224 [cs]. [Online]. Available: http://arxiv.org/abs/2210.07224 13 Immanuel Roßteutscher et al.: et al.: Masked Autoencoders for Ultrasound Signals

    2022 arXiv

  38. [46]

    Towards Optimal Patch Size in Vision Transformers for Tumor Segmentation,

    R. Mojtahedi, M. Hamghalam, R. Do, and A. Simpson, “Towards Optimal Patch Size in Vision Transformers for Tumor Segmentation,” in Multiscale Multimodal Medical Imaging. Springer Nature Switzerland AG, 2022, pp. 110–120

    2022

  39. [47]

    ImageNet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Miami, FL, USA: IEEE, 2009, pp. 248–255. [Online]. Available: https://ieeexplore.ieee...

    2009

  40. [2018]

    Available: https://cdn.openai.com/research-covers/ language-unsupervised/language_understanding_paper.pdf

    [Online]. Available: https://cdn.openai.com/research-covers/ language-unsupervised/language_understanding_paper.pdf

  41. [2020]

    Available: https://pubs.aip.org/adv/article/10/10/105221/ 1071166/A-comprehensive-review-on-accuracy-in-ultrasonic

    [Online]. Available: https://pubs.aip.org/adv/article/10/10/105221/ 1071166/A-comprehensive-review-on-accuracy-in-ultrasonic

  42. [2022]

    Available: https://www.mdpi.com/2072-666X/13/4/520

    [Online]. Available: https://www.mdpi.com/2072-666X/13/4/520

  43. [4186]

    Available: https://aclanthology.org/N19-1423/

    [Online]. Available: https://aclanthology.org/N19-1423/

  44. [6008]

    Available: https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

    [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

    2017

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.