REVIEW 8 cited by
Patches Are All You Need?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite its simplicity, we show that the ConvMixer outperforms the ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet. Our code is available at https://github.com/locuslab/convmixer.
Forward citations
Cited by 8 Pith papers
-
Gene Ontology-Guided Hierarchical Spatial Gene Expression Prediction from Histopathology Images
MSGR's Gene Ontology-guided hierarchical decoder improves spatial gene expression prediction from histology images, with the biological structure adding a +0.027 gain over an equivalent random hierarchy.
-
UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
A heterogeneous pyramidal vision encoder plus two-stage dense-then-generative pre-training yields faster, stronger on-device LVLM vision encoding than FastVLM.
-
SWIFT: Spatio-temporal Wavelet Integrated Forecasting Framework for Workload Traces
SWIFT forecasts cloud workloads at SOTA accuracy with linear complexity by combining learnable cascaded wavelet convolutions and sequential inter/intra-variable mixing.
-
Learning High-Level Decision Making with an Interaction-Aware Attention-Based Network in Autonomous Driving
An attention architecture that bottlenecks traffic agents into fixed latent queries plus a finer discrete action set yields higher simulated speeds and lower early-termination rates than DeepSet and Ego-attention on t...
-
Lightweight Backbone Networks Only Require Adaptive Lightweight Self-Attention Mechanisms
LOLViT, a GhostNet-based lightweight backbone using adaptive window attention, reports CNN-like CPU speed with MobileViT-level accuracy.
-
LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation
LaVi encodes visual context into LayerNorm affine parameters, bypassing visual token concatenation, and reports LLaVA-comparable accuracy at a 94% FLOP reduction.
-
Accurate and efficient zero-shot 6D pose estimation with frozen foundation models
A training-free 6D pose estimator using sparse-to-dense matching of frozen foundation model features achieves new state-of-the-art results on BOP with large speedups.
-
Modernizing CNN-based Weather Forecast Model towards Higher Computational Efficiency
A 7-million-parameter convolutional weather model trains in 12 hours on one GPU and is reported to match or beat much larger AI and numerical weather models in medium-range forecasts.
Discussion (0). Sign in to comment.