REVIEW 6 cited by
Demystify Mamba in Vision: A Linear Attention Perspective
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba's success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba's success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-Inspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https://github.com/LeapLabTHU/MLLA.
Forward citations
Cited by 6 Pith papers
-
How Can Mamba Learn In Context with Outliers and Generalize Provably?
A simplified one-layer Mamba provably learns in-context binary classification tolerating outlier fractions approaching 1, whereas a linear Transformer can only tolerate α < 1/2.
-
Training-free Token Reduction for Vision Mamba
MTR uses Mamba's timescale parameter Δ as a token importance score to merge unimportant tokens, giving training-free inference speedups with small accuracy loss.
-
PMA: Towards Parameter-Efficient Point Cloud Understanding via Point Mamba Adapter
PMA adapts frozen point cloud models by ordering and fusing all intermediate layer features with Mamba, achieving parameter-efficient gains on ScanObjectNN, ModelNet40, and ShapeNetPart.
-
Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks
FlowHN splits input tokens between parallel attention and SSM branches to balance compute load, improving throughput and hardware utilization in small autoregressive language models.
-
Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization
Central-difference random gradient estimation (CD-RGE) trains RNNs up to 1.1B parameters without backpropagation through time, matching or beating BPTT on several small tasks while using less GPU memory.
-
VCMamba: Bridging Convolutions with Multi-Directional Mamba for Efficient Visual Representation
VCMamba reports that using convolutional feed-forward blocks for the first three stages followed by multi-directional Mamba blocks in the final stage yields 82.6% ImageNet-1K and 47.1 ADE20K mIoU at 31.5M parameters, ...
Discussion (0). Continue with ORCID to comment.