The reviewed record of science sign in
Pith

arxiv: 2306.06635 · v1 · pith:MS5L37HL · submitted 2023-06-11 · cs.CV · cs.LG

2-D SSM: A General Spatial Layer for Visual Transformers

Ethan Baron , Itamar Zimerman , Lior Wolf This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:MS5L37HLrecord.jsonopen to challenge →

classification cs.CV cs.LG
keywords layerbiasinductivetransformersvisioneffectiveevenperformance
0
0 comments X
read the original abstract

A central objective in computer vision is to design models with appropriate 2-D inductive bias. Desiderata for 2D inductive bias include two-dimensional position awareness, dynamic spatial locality, and translation and permutation invariance. To address these goals, we leverage an expressive variation of the multidimensional State Space Model (SSM). Our approach introduces efficient parameterization, accelerated computation, and a suitable normalization scheme. Empirically, we observe that incorporating our layer at the beginning of each transformer block of Vision Transformers (ViT) significantly enhances performance for multiple ViT backbones and across datasets. The new layer is effective even with a negligible amount of additional parameters and inference time. Ablation studies and visualizations demonstrate that the layer has a strong 2-D inductive bias. For example, vision transformers equipped with our layer exhibit effective performance even without positional encoding

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

    cs.CV 2024-01 conditional novelty 7.0

    Vim is a bidirectional Mamba vision backbone that outperforms DeiT in accuracy on standard tasks while being substantially faster and more memory-efficient for high-resolution images.

  2. Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders

    cs.CV 2026-05 unverdicted novelty 6.0

    C-GSPN scales 2D spatial propagation to foundation vision encoders via a fast CUDA kernel, compressed blocks, and two-stage distillation, matching ViT performance with 15% fewer parameters and 4x block speedup at 2K r...

  3. EmambaIR: Efficient Visual State Space Model for Event-guided Image Reconstruction

    cs.CV 2026-05 unverdicted novelty 6.0

    EmambaIR is a visual state space model with cross-modal top-k sparse attention and gated SSM components that outperforms prior CNN and ViT methods on event-guided deblurring, deraining, and HDR reconstruction while re...

  4. Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

    cs.CV 2026-04 unverdicted novelty 5.0

    UniME combines a pretrained unified ViT encoder with modality-specific CNN encoders to improve brain tumor segmentation performance when some MRI modalities are missing.