REVIEW 14 cited by
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Masked image modeling (MIM) has demonstrated impressive results in self-supervised representation learning by recovering corrupted image patches. However, most existing studies operate on low-level image pixels, which hinders the exploitation of high-level semantics for representation models. In this work, we propose to use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing a systematic way to promote MIM from pixel-level to semantic-level. Specifically, we propose vector-quantized knowledge distillation to train the tokenizer, which discretizes a continuous semantic space to compact codes. We then pretrain vision Transformers by predicting the original visual tokens for the masked image patches. Furthermore, we introduce a patch aggregation strategy which associates discrete image patches to enhance global semantic representation. Experiments on image classification and semantic segmentation show that BEiT v2 outperforms all compared MIM methods. On ImageNet-1K (224 size), the base-size BEiT v2 achieves 85.5% top-1 accuracy for fine-tuning and 80.1% top-1 accuracy for linear probing. The large-size BEiT v2 obtains 87.3% top-1 accuracy for ImageNet-1K (224 size) fine-tuning, and 56.7% mIoU on ADE20K for semantic segmentation. The code and pretrained models are available at https://aka.ms/beitv2.
Forward citations
Cited by 14 Pith papers
-
Multiplayer Interactive World Models with Representation Autoencoders
A 5B-parameter latent diffusion model generates real-time four-player Rocket League matches conditioned on all players' actions, staying stable far beyond its training horizon.
-
Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentation
ConfiG uses teacher-student confidence disagreement to guide diffusion-based augmentation and improves worst-group accuracy under unknown covariate shift in knowledge distillation.
-
Time Imprint: Learning Time-Aware Representations in Multi-Modal Knowledge Graphs
Treating time as an entity-level modality with median-K timestamp selection, attention pooling, and three-stage temporal injection yields large link-prediction gains on the hardest multi-modal ambiguity cases.
-
Bitrate-Controlled Diffusion for Disentangling Motion and Content in Video
A self-supervised diffusion framework with a low-bitrate vector-quantization bottleneck learns disentangled motion and content latents supporting motion transfer and auto-regressive generation.
-
BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
A self-supervision method makes multimodal LLMs align their input image embeddings with the model's own refined internal representations, improving visual QA scores over LLaVA baselines.
-
Dataset Ownership Verification for Pre-trained Masked Models
DOV4MM detects whether a masked pre-trained model was trained on a given dataset via relative embedding reconstruction difficulty, reporting p<0.05 in tests on ImageNet-1K and WikiText-103.
-
UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation
UniCode² builds a 500K-entry codebook from clustered SigLIP embeddings and uses a cascaded frozen-plus-trainable codebook to unify multimodal understanding and generation with stable training and high token utilization.
-
BAT: Better Audio Transformer Guided by Convex Gated Probing
CGP probing—layer-gating plus prototypes—closes much of the gap between frozen and fine-tuned audio SSL evaluation, and guides a re-engineered audio transformer (BAT) that improves on the authors' reproduced baselines.
-
Robustifying Diffusion-Denoised Smoothing Against Covariate Shift
Adversarially perturbing the noise term of a diffusion denoiser during training improves the certified l2 robustness of denoised randomized smoothing on MNIST, CIFAR-10, and ImageNet, with the largest gains at large p...
-
TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers
TinyDrop uses a lightweight model's confidence and attention map to early-exit easy samples and drop uninformative tokens in frozen ViTs, cutting FLOPs by up to 87%.
-
MINR: Implicit Neural Representations with Masked Image Modelling
A hybrid of implicit neural representations and masked image modeling, called MINR, reconstructs masked image patches better than MAE in the reported in-domain and out-of-distribution tests with fewer parameters.
-
ProtoConNet: Prototypical Augmentation and Alignment for Open-Set Few-Shot Image Classification
ProtoConNet improves open-set few-shot classification by combining clustering-based sample selection, contextual feature fusion, and prototypical alignment with a threshold-based unknown-class detector.
-
Towards Structure-aware Model for Multi-modal Knowledge Graph Completion
TSAM combines token-level fusion of visual and textual data with structure-anchored contrastive learning, outperforming prior multi-modal KGC models on DB15K, MKG-W, and MKG-Y.
-
Object Concepts Emerge from Motion
Motion-based pseudo-labels from optical flow clustering, used with contrastive pretraining, produce visual features that transfer well to depth, 3D detection, and occupancy tasks.
Discussion (0). Sign in to comment.