Pith. sign in

REVIEW 3 major objections 7 minor 94 references

Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing

T0 review · 3 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FastVim claims that a Vision Mamba encoder can match full-model accuracy while scanning half the tokens, by mean-pooling the token grid along alternating spatial dimensions each layer.

desk verdict A simple, plausible efficiency trick for Vision Mamba with solid throughput measurements, but the 'no degradation' claim is confounded by a normalization change and needs a controlled baseline. read the letter →

arxiv 2502.00594 v1 pith:CH5UOKJK submitted 2025-02-01 cs.CV cs.AI

Saarthak Kapse , Robin Betz , Srinivasan Sivanandan This is my paper · ORCID
classification cs.CVcs.AI
keywords visionstatespacemodelsMambameanpoolingparallelscaninferencespeedupmaskedautoencodersper-channeltokenizationhigh-resolutionimaging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FastVim claims that a Vision Mamba (Vim) encoder can match the accuracy of the full model while scanning far fewer tokens, by mean-pooling the 2D token grid along one spatial dimension before each selective state-space scan and alternating the pooled dimension across layers. This cuts the SSM's parallel scan steps from log2($h^{2}$) to log2(h), reducing the per-block SSM scan time at 2048x2048 images from 58.20 ms to 0.30 ms and delivering up to a 72.5% overall inference speedup. The paper reports parity with Vim on ImageNet-1k classification, ADE20K segmentation, COCO detection and instance segmentation, and JUMP-CP cell perturbation prediction, and a new state-of-the-art 86.7% ImageNet-1k accuracy for a Mamba-based encoder when combined with masked autoencoding. If these results hold, high-resolution and long-token vision tasks become far more practical on SSM backbones without an accuracy penalty.

What carries the argument

The load-bearing mechanism is the pool-scan-repeat roundtrip: a parameter-free mean pooling over one spatial axis of the token grid before the selective SSM scan, a transposition to alternate the pooled axis every layer, and a repetition operation that restores the original token count before the Dxt skip connection. This compresses the sequence seen by the parallel scan from L = h x w tokens to h tokens, halving the number of parallel scan steps, while the residual path and subsequent blocks are what the paper relies on to preserve information that pooling may have discarded.

What would settle it

Take two 224x224 images that are provably identical after the column-mean and row-mean pooling steps used in FastVim's early blocks (for example, images that differ only in high-frequency variation within each pooled row), and test whether a full Vim model classifies them differently while FastVim produces identical logits.

Watch

Extended reading notes

Core claim

The central claim is that the full pairwise token contextualization inside Vim's SSM scan is not necessary at every layer: after a 1D convolution, mean-pooling tokens across columns (or rows) reduces the scan input from $h^{2}$ to h tokens, the SSM processes only the pooled tokens, and the output is repeated back to full resolution before the Dxt skip connection and norm. By transposing the token grid at each block, FastVim alternates which dimension is pooled, so every token still interacts with tokens in other rows and columns across successive layers. The paper argues empirically that this alternation is required, not optional, for accuracy, that the scheme works in Mamba but fails in Vision Transformers, and that the resulting architecture shows no performance degradation relative to the Vim baseline across classification, segmentation, detection, and cell-perturbation tasks, despite contextualizing significantly fewer tokens per scan.

Load-bearing premise

The paper's results rest on the empirical premise that alternating mean pooling over rows and columns preserves enough token-interaction information that the residual skip connection and later blocks can recover what a full per-token scan would have provided; if mean pooling discards signal that the skip connections cannot restore, the claimed parity with Vim would fail.

Editorial extensions

If this is right

  • FastVim cuts the SSM scan time in a block from 58.20 ms to 0.30 ms at 2048x2048 input, and the whole model runs up to 72.5% faster than Vim at that resolution.
  • The speed gap over Vim widens as resolution increases, and FastVim becomes faster than ViT at 1024x1024 and above while consuming less memory.
  • FastMaskVim reaches 86.7% ImageNet-1k top-1 accuracy, the current best for a Mamba-based visual encoder, with faster pre-training and fine-tuning than Vim.
  • FastChannelVim matches ChannelVim accuracy on JUMP-CP while increasing throughput by 62.3% at patch size 8, and improves over ChannelViT by 8.3 points.
  • Because the pooling is parameter-free, the technique can be dropped into other Mamba-based vision architectures to accelerate their SSM scans as well.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the success of such sparse, alternating scans suggests Mamba's per-layer contextualization is highly redundant, so even sparser interaction patterns, such as pooling every other layer or fusing tokens across multiple blocks, may preserve accuracy while cutting cost further.
  • Beyond the paper: the same pool-scan-repeat cycle could be applied along the channel or time axis for video and 3D microscopy data, reducing scan steps by additional log factors; the paper's 2D pooling results already hint this works at smaller patch sizes.
  • Beyond the paper: the contrast between pooling succeeding in Mamba and failing in ViT points to a structural difference in how recurrent state propagation distributes information across steps, which could be characterized theoretically as a rank or memory property of the SSM hidden state.
  • A testable extension the paper does not run: ablate the Dxt skip connection in FastVim to measure how much of the preserved accuracy comes from the residual path versus the pooled scan; the authors only moved the decompression after the skip connection, not removed the connection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes FastVim, a modification of Vision Mamba (Vim) that applies mean pooling across one spatial dimension of the token grid before the SSM scan, alternating row/column pooling across layers, and then repeats the scan output to restore the original resolution. This reduces the number of parallel scan steps from log(h^2) to log(h). The authors report up to 72.5% inference speedup at 2048x2048 resolution, comparable or better accuracy on ImageNet-1k, ADE20K, COCO, and JUMP-CP, and a Mamba-based state of the art of 86.7% with MAE pretraining. They also introduce FastMaskVim and FastChannelVim for masked and per-channel tokenization settings. Code is provided at github.com/insitro/FastVim.

Significance. The complexity reduction is exact and the throughput measurements are detailed and internally consistent: Table 23 shows Vim's SSM scan time growing by roughly 74x from 224 to 2048 resolution, while FastVim's scan time stays nearly flat. The pooling idea is simple, parameter-free, and backed by useful ablations on alternating dimensions, pooling types, and norm combinations. The code release and reproduction details are strengths. However, the headline accuracy-parity claim is not yet cleanly established because the normalization recipe is changed at the same time as the pooling strategy, and the reported Tiny results already show a 0.7-point drop relative to the Vim baseline.

major comments (3)
  1. [Sec. 4.1, Table 1; Sec. 9, Table 17] The comparison against Vim-T and Vim-S is confounded by a simultaneous architectural change: FastVim uses an RMS input norm plus a post-SSM LayerNorm (Table 17) and adds the post-SSM LayerNorm, while the Vim-T and Vim-S rows in Table 1 are the original published numbers without that norm. Section 4.1 reports that adding post-SSM LayerNorm improved Vim-B from 80.7 to 82.6, a +1.9 gain. Without Vim-T and Vim-S baselines retrained with the identical normalization recipe and matching hyperparameters (including drop path), the observed parity cannot be attributed to the pooling strategy; the reported numbers are equally consistent with the added norm being responsible for part or all of the difference. The authors should provide controlled Vim-T/S baselines with the same RMS-LN and post-SSM LayerNorm settings before claiming that pooling causes no performance degradation.
  2. [Conclusion; Table 1] The conclusion states that FastVim 'achieves this without any performance degradation compared to the baseline Vim model across multiple tasks,' but Table 1 reports FastVim-T at 75.4 versus Vim-T at 76.1, a 0.7-point drop. Regardless of whether this is within run-to-run noise, the sentence as written is internally inconsistent with the reported numbers. Please qualify the claim (e.g., 'within noise' or 'for Small/Base and downstream tasks') or provide multiple seeds to demonstrate that the Tiny gap is not real.
  3. [Supplement 8, Tables 14-15] The MAE experiments depend on an additional, non-architectural scaling factor of 0.25 applied when pooling during fine-tuning and linear probing. Table 15 shows the effect is drastic: linear probing collapses from 60.2% to 0.02% without the scaling factor. This indicates that the constant-divide pooling is not scale-invariant across masking regimes and that the transfer recipe contains a dataset-specific correction. The paper should present this as a limitation of the current pooling scheme rather than as part of a fully parameter-free method, and it should acknowledge that the no-degradation claim for MAE pretraining rests on this correction.
minor comments (7)
  1. [Algorithm 1 and Sec. 3.1] Algorithm 1 shows pooling along the second spatial dimension with no transpose, while the main text explains that the token grid is transposed every block to alternate pooling dimensions. Please make the algorithm consistent with the described implementation.
  2. [Sec. 4.2] The phrase 'a 324% speedup' should be 'a 3.24x speedup' (or 'a 224% speedup') to avoid ambiguity about whether the total is 324% of the original or an increase of 324%.
  3. [Table 2 caption] The caption contains a typo: 'btoh' should be 'both'.
  4. [Sec. 4.4] The word 'insipired' should be 'inspired'.
  5. [References] References [51] and [52] are the same paper (Ren et al., 'Autoregressive pretraining with mamba in vision') and should be merged.
  6. [Sec. 4.2 and Supplement Fig. 10] The cross-reference for the LayerNorm post-SSM throughput comparison points to 'Fig. 3', which is the loss-stability figure, while the actual comparison appears in Supplement Fig. 10. Please fix the cross-reference.
  7. [Abstract and Sec. 4.2] The 72.5% speedup is stated without conditions; please specify that it is measured at 2048x2048 resolution, batch size 128, on an H100, as reported in Sec. 4.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 2x parallel-step reduction follows directly from the pooling definition in Algorithm 1, and the performance claims are external benchmark measurements; no fitted constant is relabeled as a prediction.

full rationale

The paper's efficiency claim is definitional rather than circular: Algorithm 1 pools the h*w token grid to h tokens before the SSM scan, so the scan operates on h tokens and the parallel-scan depth drops from log(h^2)=2log(h) to log(h). Table 23 then reports measured SSM scan times (58.20 ms down to 0.30 ms at 2048x2048) rather than deriving them from a fitted constant. The 'no performance degradation' claim is an empirical comparison on external benchmarks (ImageNet-1k, ADE20K, COCO, JUMP-CP) and is not obtained by construction from the architecture. The MAE transfer scaling factor of 0.25 is set by the 75% masking ratio (each pooled row contains on average 25% of the columns, so the divide-by-columns pooling amplitude changes by 0.25 between pretraining and full-context transfer), and the paper ablates it in Table 14; it is a training-stability constant, not a fitted prediction. One architectural confound is present and disclosed: post-SSM LayerNorm is part of FastVim but not of the original Vim-T/S baselines in Table 1, and Sec. 4.1 reports that adding LayerNorm raised Vim-B from 80.7 to 82.6. This weakens the no-degradation comparison for Tiny and Small sizes, but that is a benchmarking fairness issue, not circularity. The self-citation to ChannelViT [3] for per-channel tokenization is used as an external baseline and is directly benchmarked in Table 3; it does not carry the central result. No self-definitional step, no fitted input relabeled as prediction, and no load-bearing self-referential chain was found.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central efficiency identity (log(h^2) = 2 log h) is definitional and not circular. The method introduces no new entities and no new fitted constants into the core SSM, but it relies on two domain assumptions about information preservation under pooled scans, validated only empirically. The MAE scaling factor 0.25 is a hand-set training constant tied to the masking ratio; the drop path rate is a standard regularization choice.

free parameters (2)
  • MAE fine-tuning scaling factor = 0.25
    Applied during MAE fine-tuning and linear probing to rescale pooled token sums (Supplement Sec. 8, Tables 14-15). Set to 1 minus the 0.75 masking ratio, selected to match the pretraining distribution rather than fitted to the target metric.
  • FastVim-B drop path rate = 0.4
    Higher drop path rate used for the base model to avoid overfitting (Sec. 4.1). A standard regularization hyperparameter, not part of the pooling method and not load-bearing for the complexity claim.
assumptions (5)
  • standard math Parallel scan on an associative operator of length L requires ceil(log2 L) sequential steps.
    Invoked for both Vim and FastVim complexity comparisons (Preliminaries, Sec. 2, citing Smith et al. [57]).
  • domain assumption Alternating column-pooling and row-pooling across blocks lets all tokens interact implicitly.
    Stated in Sec. 3.1 and Fig. 1; empirically supported by the alternation ablation (Table 6), but no formal proof is given.
  • domain assumption Mean pooling of tokens before the SSM scan preserves enough visual information for downstream tasks.
    The core of the method (Sec. 3.1); validated only empirically through the accuracy-parity results in Sec. 4.
  • domain assumption In FastMaskVim, dividing pooled sums by the number of grid columns rather than the number of unmasked tokens preserves token-count information.
    Introduced in Sec. 3.2 and justified by the ablation in Supplement Table 12.
  • standard math SSM discretization via zero-order hold and the Mamba selective-scan formulation.
    Standard background from Sec. 2 (Eqs. 1-4), taken directly from prior SSM literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing." pith.science (2026). https://pith.science/paper/CH5UOKJK

@misc{pith2026250200594,
  author       = {Pith},
  title        = {Pith review of: Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CH5UOKJK}},
  note         = {Machine review of arXiv:2502.00594}
}
abstract

State Space Models (SSMs) with selective scan (Mamba) have been adapted into efficient vision models. Mamba, unlike Vision Transformers, achieves linear complexity for token interactions through a recurrent hidden state process. This sequential processing is enhanced by a parallel scan algorithm, which reduces the computational time of recurrent steps from $L$ sequential steps to $log(L)$ parallel steps with respect to the number of input tokens ($L$). In this work, we propose Fast Vision Mamba (FastVim), that further reduces the computational time of the SSM block by reducing the number of recurrent steps in Vision Mamba models while still retaining model performance. By alternately pooling tokens along image dimensions across Mamba blocks, we obtain a 2$\times$ reduction in the number of parallel steps in SSM block. Our model offers up to $72.5\%$ speedup in inference speed compared to baseline Vision Mamba models on high resolution (2048$\times$2048) images. Our experiments demonstrate state-of-the-art performance with dramatically improved throughput in a range of tasks such as image classification, cell perturbation prediction, segmentation, and object detection. Code is made available at https://github.com/insitro/FastVim

Figures

Figures reproduced from arXiv: 2502.00594 by the authors.

Figure 1
Figure 1. FastVim accelerates Vim by mean pooling tokens across [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of FastVim: Input image tokens are fed to norm and expansion layers, then output x is transposed (T) every block for alternate pooling of rows and columns. Tokens are pooled post-Conv1D, processed by SSM, and decompressed before skip-connection (D in eq. 3). Note that the flattened tokens are reshaped into a 2D grid prior to the transpose and pooling layers, and are flattened again after these operations. I… view at source ↗
Figure 3
Figure 3. Stability Issue in Vim-B on ImageNet-1k. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Comparison of FLOPs (G) for FastVim, Vim, and ViT [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparison of Inference Throughput (it/s) for FastVim, [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Effect of scaling factor in finetuning performance from [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Comparison of inference processing time for only the [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Comparison of Inference Throughput (it/s) for FastVim, Vim, and ViT across different resolutions. Tested on H100 GPU with [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Comparison of Inference Throughput (it/s) for FastVim, Vim, and ViT across different resolutions. Tested with batch size of 128, [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Comparison of Inference Throughput (it/s) for FastVim, Vim, and ViT across different resolutions. Tested on H100 GPU with [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Comparison of Inference Throughput (it/s) for FastVim, Vim, and ViT across different resolutions. Tested on H100 GPU with [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 14
Figure 14. Figure 14: Illustration of flattened scanpath options available fol [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Illustration of flattened scanpath options available fol [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

94 extracted references · 43 canonical work pages

  1. [1]

    Hiervl: Learning hierarchical video- language embeddings

    Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hierarchical video- language embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23066–23078, 2023. 8

    2023

  2. [2]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 12

    arXiv 2021

  3. [3]

    Channel vision transformers: An image is worth c x 16 x 16 words

    Yujia Bao, Srinivasan Sivanandan, and Theofanis Karaletsos. Channel vision transformers: An image is worth c x 16 x 16 words. arXiv preprint arXiv:2309.16108, 2023. 2, 3, 4, 7, 14, 15

    arXiv 2023

  4. [4]

    Hypermae: Modulating implicit neural representations for mae training

    Varun Belagali, Lei Zhou, Xiang Li, and Dimitris Samaras. Hypermae: Modulating implicit neural representations for mae training. 2023. 4

    2023

  5. [5]

    Prefix sums and their applications

    Guy E Blelloch. Prefix sums and their applications. 1990. 1

    1990

  6. [6]

    To- ken merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 8

    arXiv 2022

  7. [7]

    High-performance large-scale image recognition without normalization

    Andy Brock, Soham De, Samuel L Smith, and Karen Si- monyan. High-performance large-scale image recognition without normalization. In International conference on ma- chine learning, pages 1059–1071. PMLR, 2021. 16

    2021

  8. [8]

    Jump cell painting dataset: morphological im- pact of 136,000 chemical and genetic perturbations.BioRxiv, pages 2023–03, 2023

    Srinivas Niranj Chandrasekaran, Jeanelle Ackerman, Eric Alix, D Michael Ando, John Arevalo, Melissa Bennion, Nicolas Boisseau, Adriana Borowa, Justin D Boyd, Laurent Brino, et al. Jump cell painting dataset: morphological im- pact of 136,000 chemical and genetic perturbations.BioRxiv, pages 2023–03, 2023. 2, 6

    2023

Show all 94 references
  1. [9]

    Towards a general-purpose foundation model for computational pathology

    Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 30(3):850–862,

  2. [10]

    Openmmlab semantic seg- mentation toolbox and benchmark, 2020

    MMSegmentation Contributors. Openmmlab semantic seg- mentation toolbox and benchmark, 2020. 18

    2020

  3. [11]

    Randaugment: Practical automated data augmen- tation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmen- tation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020. 12

    2020

  4. [12]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. 6, 16

    2023 arXiv

  5. [13]

    Transformers are ssms: General- ized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: General- ized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. 8

    2024 arXiv

  6. [14]

    Flashattention: Fast and memory-efficient exact at- tention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R´e. Flashattention: Fast and memory-efficient exact at- tention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. 6

    2022

  7. [15]

    Vpn++: Rethinking video-pose embeddings for understand- ing activities of daily living

    Srijan Das, Rui Dai, Di Yang, and Francois Bremond. Vpn++: Rethinking video-pose embeddings for understand- ing activities of daily living. IEEE Transactions on Pat- tern Analysis and Machine Intelligence , 44(12):9703–9717,

  8. [16]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 4, 5, 12

    2009

  9. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1

    2010 arXiv

  10. [18]

    Learned representation-guided diffusion models for large-image generation

    Alexandros Graikos, Srikar Yellapragada, Minh-Quan Le, Saarthak Kapse, Prateek Prasanna, Joel Saltz, and Dimitris Samaras. Learned representation-guided diffusion models for large-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  11. [19]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 1, 2

    2023 arXiv

  12. [20]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R ´e. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. 1, 2

    2021 arXiv

  13. [21]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R ´e. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing sys- tems, 34:572–585, 2021. 2

    2021

  14. [22]

    Mambavision: A hy- brid mamba-transformer vision backbone

    Ali Hatamizadeh and Jan Kautz. Mambavision: A hy- brid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083, 2024. 5, 8, 12

    2024 arXiv

  15. [23]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 7

    2017

  16. [24]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 2, 4, 6, 12

    2022

  17. [25]

    To- ken dropping for efficient bert pretraining

    Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. To- ken dropping for efficient bert pretraining. arXiv preprint arXiv:2203.13240, 2022. 8

    2022 arXiv

  18. [26]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Com- puter Vision–ECCV 2016: 14th European Conference, Am- sterdam, The Netherlands, October 11–14, 2016, Proceed- ings, Part IV 14, pages 646–661. Springer, 2016. 12

    2016

  19. [27]

    Localmamba: Visual state space model with windowed selective scan

    Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. arXiv preprint arXiv:2403.09338,

    arXiv

  20. [28]

    Attention-based deep multiple instance learning

    Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In Inter- national conference on machine learning, pages 2127–2136. PMLR, 2018. 4, 15

    2018

  21. [29]

    Object- centric diffusion for efficient video editing

    Kumara Kahatapitiya, Adil Karjauv, Davide Abati, Fatih Porikli, Yuki M Asano, and Amirhossein Habibian. Object- centric diffusion for efficient video editing. In European Conference on Computer Vision , pages 91–108. Springer,

  22. [30]

    Si-mil: Taming deep mil for self-interpretability in gigapixel histopathology

    Saarthak Kapse, Pushpak Pati, Srijan Das, Jingwei Zhang, Chao Chen, Maria Vakalopoulou, Joel Saltz, Dimitris Sama- ras, Rajarsi R Gupta, and Prateek Prasanna. Si-mil: Taming deep mil for self-interpretability in gigapixel histopathology. In Proceedings of the IEEE/CVF Conferen...

    2024

  23. [31]

    Vision transformers inference acceleration based on adaptive layer normalization

    Fekhr Eddine Keddous, Arcadi Llanza, Nadiya Shvai, and Amir Nakib. Vision transformers inference acceleration based on adaptive layer normalization. Neurocomputing, 610:128524, 2024. 16

    2024

  24. [32]

    Vitally consistent: Scaling biological representation learning for cell microscopy

    Kian Kenyon-Dean, Zitong Jerry Wang, John Urbanik, Kon- stantin Donhauser, Jason Hartford, Saber Saberian, Nil Sahin, Ihab Bendidi, Safiye Celik, Marta Fay, et al. Vitally consistent: Scaling biological representation learning for cell microscopy. arXiv preprint arXiv:2411.025...

    2024 arXiv

  25. [33]

    Videomamba: State space model for efficient video understanding

    Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In European Conference on Computer Vision, pages 237–255. Springer, 2025. 8

    2025

  26. [34]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. In European conference on computer vision , pages 280–296. Springer, 2022. 7

    2022

  27. [35]

    Not all patches are what you need: Expediting vision transformers via token reorganiza- tions

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganiza- tions. arXiv preprint arXiv:2202.07800, 2022. 8

    2022 arXiv

  28. [36]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024. 5

    2024 arXiv

  29. [37]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  30. [38]

    Map: Unleashing hybrid mamba- transformer vision backbone’s potential with masked au- toregressive pretraining

    Yunze Liu and Li Yi. Map: Unleashing hybrid mamba- transformer vision backbone’s potential with masked au- toregressive pretraining. arXiv preprint arXiv:2410.00871 ,

    arXiv

  31. [39]

    Vmamba: Visual state space model, 2024

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model, 2024. 1, 5, 6, 8, 12

    2024

  32. [40]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 1, 8, 12

    2021

  33. [41]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 11976–11986,

  34. [42]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 12

    2017 arXiv

  35. [43]

    Sgdr: Stochas- tic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochas- tic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. 12

    2016 arXiv

  36. [44]

    Vim4path: Self-supervised vi- sion mamba for histopathology images

    Ali Nasiri-Sarvi, Vincent Quoc-Huy Trinh, Hassan Rivaz, and Mahdi S Hosseini. Vim4path: Self-supervised vi- sion mamba for histopathology images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6894–6903, 2024. 8

    2024

  37. [45]

    An image is worth more than 16x16 patches: Exploring transformers on individual pixels

    Duy-Kien Nguyen, Mahmoud Assran, Unnat Jain, Martin R Oswald, Cees GM Snoek, and Xinlei Chen. An image is worth more than 16x16 patches: Exploring transformers on individual pixels. arXiv preprint arXiv:2406.09415, 2024. 7

    2024 arXiv

  38. [46]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 4

    2023 arXiv

  39. [47]

    Efficientvmamba: Atrous selective scan for light weight visual mamba

    Xiaohuan Pei, Tao Huang, and Chang Xu. Efficientvmamba: Atrous selective scan for light weight visual mamba. arXiv preprint arXiv:2403.09977, 2024. 8, 12

    2024 arXiv

  40. [48]

    Enhancing feature di- versity boosts channel-adaptive vision transformers

    Chau Pham and Bryan A Plummer. Enhancing feature di- versity boosts channel-adaptive vision transformers. arXiv preprint arXiv:2405.16419, 2024. 7

    2024 arXiv

  41. [49]

    Per- ceptual grouping in contrastive vision-language models

    Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Per- ceptual grouping in contrastive vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5571–5584, 2023. 15

    2023

  42. [50]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949,

  43. [52]

    Autoregressive pretraining with mamba in vision

    Sucheng Ren, Xianhang Li, Haoqin Tu, Feng Wang, Fangxun Shu, Lei Zhang, Jieru Mei, Linjie Yang, Peng Wang, Heng Wang, et al. Autoregressive pretraining with mamba in vision. arXiv preprint arXiv:2406.07537, 2024. 5

    2024 arXiv

  44. [53]

    Learn- ing to merge tokens in vision transformers

    Cedric Renggli, Andr ´e Susano Pinto, Neil Houlsby, Basil Mustafa, Joan Puigcerver, and Carlos Riquelme. Learn- ing to merge tokens in vision transformers. arXiv preprint arXiv:2202.12015, 2022. 8

    2022 arXiv

  45. [54]

    Tokenlearner: What can 8 learned tokens do for images and videos? arXiv preprint arXiv:2106.11297, 2021

    Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can 8 learned tokens do for images and videos? arXiv preprint arXiv:2106.11297, 2021. 8

    2021 arXiv

  46. [55]

    Groupmamba: Parameter-efficient and accurate group visual state space model

    Abdelrahman Shaker, Syed Talal Wasim, Salman Khan, Juergen Gall, and Fahad Shahbaz Khan. Groupmamba: Parameter-efficient and accurate group visual state space model. arXiv preprint arXiv:2407.13772, 2024. 5, 8

    2024 arXiv

  47. [56]

    Famba-v: Fast vision mamba with cross-layer token fusion

    Hui Shen, Zhongwei Wan, Xin Wang, and Mi Zhang. Famba-v: Fast vision mamba with cross-layer token fusion. arXiv preprint arXiv:2409.09808, 2024. 8

    2024 arXiv

  48. [57]

    Simplified state space layers for sequence modeling

    Jimmy TH Smith, Andrew Warrington, and Scott W Linder- man. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933, 2022. 1, 3 10

    2022 arXiv

  49. [58]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International conference on machine learning , pages 10347–10357. PMLR, 2021. 8

    2021

  50. [59]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International conference on machine learning , pages 10347–10357. PMLR, 2021. 12

    2021

  51. [60]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 1

    2017

  52. [61]

    Mamba-r: Vision mamba also needs registers

    Feng Wang, Jiahao Wang, Sucheng Ren, Guoyizhe Wei, Jieru Mei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Mamba-r: Vision mamba also needs registers. arXiv preprint arXiv:2405.14858, 2024. 12

    2024 arXiv

  53. [62]

    Unified perceptual parsing for scene understand- ing

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understand- ing. In Proceedings of the European conference on computer vision (ECCV), pages 418–434, 2018. 7

    2018

  54. [63]

    A whole-slide foundation model for digital pathology from real-world data

    Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier Gonz ´alez, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, pages 1–8, 2024. 8

    2024

  55. [64]

    Plainmamba: Improving non-hierarchical mamba in visual recognition

    Chenhongyi Yang, Zehui Chen, Miguel Espinosa, Linus Er- icsson, Zhenyu Wang, Jiaming Liu, and Elliot J Crowley. Plainmamba: Improving non-hierarchical mamba in visual recognition. arXiv preprint arXiv:2403.17695, 2024. 12

    2024 arXiv

  56. [65]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 6023–6032, 2019. 12

    2019

  57. [66]

    Exploring token pruning in vision state space models

    Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zi- chong Meng, Hangyu Zheng, Xuan Shen, Stratis Ioannidis, Wei Niu, Pu Zhao, et al. Exploring token pruning in vision state space models. arXiv preprint arXiv:2409.18962, 2024. 8, 14

    2024 arXiv

  58. [67]

    Rethinking token reduction for state space models

    Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang, Yi- fan Gong, Xuan Shen, Xue Lin, Pu Zhao, and Yanzhi Wang. Rethinking token reduction for state space models. arXiv preprint arXiv:2410.14725, 2024. 8

    2024 arXiv

  59. [68]

    mixup: Beyond empirical risk minimization

    Hongyi Zhang. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017. 12

    2017 arXiv

  60. [69]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633–641,

  61. [70]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 1, 3, 4, 5, 7, 12, 17 11 Fast Vision Mamba: Pooling Spatial ...

    2024 arXiv

  62. [71]

    We closely followed the pre- training (Table 9), fine-tuning (Table 10), and linear- probing (Table 11) settings from the Masked Autoen- coders [24] codebase

    Self-Supervised Learning: MAE (addi- tional) Implementation Details . We closely followed the pre- training (Table 9), fine-tuning (Table 10), and linear- probing (Table 11) settings from the Masked Autoen- coders [24] codebase. All MAE pretraining is done for 1600 epochs in t...

  63. [72]

    2) We applied a scaling factor of 1 − mask ratio (75% masking by default) during fine-tuning and linear probing when pooling tokens before the SSM block

    Since Vim contains twice the number of layers compared to ViTs, we decreased the layer-wise learning rate decay every two blocks, instead of every block as in the MAE codebase for ViT fine-tuning, to ensure adequate fine-tuning of the initial layers. 2) We applied a scaling fa...

  64. [73]

    mean pool in Fast- MaskVim

    Divide by number of columns vs. mean pool in Fast- MaskVim. In Table 12, we compare the performance of pre-training FastMaskVim using the default setting, where the sum of tokens in a row is divided by the number of columns, against mean pooling, where each row’s sum is divide...

  65. [74]

    In Table 13, we compare the performance of fine-tuning pre-trained FastMaskVim using alternate layer learning rate decay instead of per-layer decay as in the MAE codebase

    Finetuning with alternate layer lr decay. In Table 13, we compare the performance of fine-tuning pre-trained FastMaskVim using alternate layer learning rate decay instead of per-layer decay as in the MAE codebase. We observe that, since Vim typically contains twice the number ...

  66. [75]

    Apply- ing the scaling factor results in an improvement of 0.3% compared to the default mean pooling in fine-tuning without multiplying by the scaling factor

    Finetuning with scaling factor.In Table 14, we demon- strate the effect of using a scaling factor (0.25) in the fine-tune transfer of pre-trained FastMaskVim. Apply- ing the scaling factor results in an improvement of 0.3% compared to the default mean pooling in fine-tuning wi...

  67. [76]

    In Table 15, we compare the linear probing performance of Fast- MaskVim with and without the scaling factor (0.25)

    Linear probing with scaling factor. In Table 15, we compare the linear probing performance of Fast- MaskVim with and without the scaling factor (0.25). We observe a drastic difference in performance and note 13 that without the scaling factor, the model was unable to train due...

  68. [77]

    Additional ablations

  69. [78]

    In Table 16, we compare the performance of FastVim-S with a class token versus without a class token (default)

    Effect of using class token in FastVim. In Table 16, we compare the performance of FastVim-S with a class token versus without a class token (default). We observe that having a class token improves performance but leads to slower reshape-transpose, pooling, and repeat operatio...

  70. [79]

    In Table 17, we empir- ically demonstrate the performance of FastVim trained with different combinations of input normalization and post-SSM normalization

    The performance impact of different input norm and post-ssm norm combinations. In Table 17, we empir- ically demonstrate the performance of FastVim trained with different combinations of input normalization and post-SSM normalization. We found that using RMS Table 16. Effect o...

  71. [80]

    In Table 18, we explore whether in Fig

    Effect of decompression after the skip connection on models’ performance. In Table 18, we explore whether in Fig. 2, we can move the skip connection Dxt before repeating/decompressing the output to achieve even more speedup. However, we empirically found that it leads to a sig...

  72. [81]

    We followed the implementation details primarily from ChannelViT [3]

    JUMP-CP (additional) Implementation details. We followed the implementation details primarily from ChannelViT [3]. Specifically, we used a learning rate of 1 × 10−3, a batch size of 256, and trained the model for 100 epochs, including 10 warmup epochs. We set the drop path rat...

  73. [82]

    Channel- First with and without sorted HCS

    ChannelVim-S: Effect of Spatial-First vs. Channel- First with and without sorted HCS. In Table 19, we demonstrate the key configurations required to extend ChannelViT [3] to the Mamba-based encoder, termed ChannelVim. As explained in detail in Sec. 3.3, due to the sequential p...

  74. [83]

    We then explore the effect of different pooling methods, such as max pooling [49] and attention pooling [28], as detailed in Table 20 on the JUMP-CP dataset

    FastChannelVim-S: Effect of different pooling methods (mean, max, and attention pooling): In this study, we use average pooling of tokens to compress the tokens before the SSM scan. We then explore the effect of different pooling methods, such as max pooling [49] and attention...

  75. [84]

    Now, we preliminarily explore pooling along two dimen- Table 20

    FastChannelVim-S: Effect of Pooling across 2 dimen- sions: So far, we have explored pooling along only one spatial dimension, either across rows or columns. Now, we preliminarily explore pooling along two dimen- Table 20. FastChannelVim-S: Effect of different pooling methods (...

  76. [85]

    Additional Throughput analysis All throughput analysis is done on the H100 and involves inference throughput unless otherwise specified

  77. [86]

    Effect of Autocast. In Fig. 8, we compare the through- put of ViT-T, Vim-T, and our FastVim-T across different resolutions, both with and without the autocast func- tionality for Vim and FastVim, since a few parameters need to be in floating point (fp) 32 in Mamba. In contrast...

  78. [87]

    Throughput on A100. In Fig. 9, we compare the throughput of Vim and FastVim on both A100 and H100 GPUs. As shown, at a resolution of 1536, FastVim provides almost a 100% improvement on the A100 compared to a 70% speedup on the H100 over Vim. The likely reason for this discrepa...

  79. [88]

    Effect of LayerNorm post-SSM. In Fig. 10, we illustrate the effect of using LayerNorm post-SSM on throughput for both Vim and FastVim. It is evident that adding LayerNorm results in slower throughput but is essential for maintaining stability, as shown in Fig. 3. Unlike BatchN...

  80. [89]

    Throughput across model sizes. In Fig. 11, we display the throughput of Vim and FastVim across Tiny, Small, and Base-sized models with a batch size of 16. Across all model sizes, our method consistently provides a speedup in throughput compared to the Vim baseline

  81. [90]

    In Table 22, we demonstrate the throughput improvement in FastChannelVim compared to ChannelVim

    Throughput on per-channel modeling tasks. In Table 22, we demonstrate the throughput improvement in FastChannelVim compared to ChannelVim. With a longer token sequence (patch size 8), FastChannelVim delivers a speedup of 62.3% over ChannelVim without any drop in accuracy (see ...

  82. [91]

    Here, we calcu- late the processing time for Forward SSM + Backward SSM in only one block (see Fig

    Dissecting SSM processing time. Here, we calcu- late the processing time for Forward SSM + Backward SSM in only one block (see Fig. 2) for Vim-T versus FastVim-T. The SSM time include the parameter projec- tion ( B, C, ∆) for selective scan, the SSM scan time, and the skip con...

  83. [92]

    We employed the AdamW optimizer with a weight decay of 0.01

    Semantic Segmentation implementation details In line with Vim [70] and LocalVim [27], we used a batch size of 16 and an input size of 512x512. We employed the AdamW optimizer with a weight decay of 0.01. A Poly learning rate schedule was used, decaying over 160K iter- ations, ...

  84. [93]

    We employed the AdamW optimizer with a weight decay of 0.05, with a total batch size of 64

    Object Detection and Instance Segmenta- tion implementation details Following the code from LocalVim [27], we utilize the neck architecture from ViTDet and train Cascade Mask R-CNN as the detector. We employed the AdamW optimizer with a weight decay of 0.05, with a total batch...

  85. [94]

    2), we apply mean pooling to the tokens before performing the SSM scan

    Kernel details In FastVim (refer to Fig. 2), we apply mean pooling to the tokens before performing the SSM scan. Consequently, this operation must be repeated before integrating with the skip connection (D in Eq. 3). When implementing this in Py- Torch, we utilize the repeat i...

  86. [95]

    Model configurations for FastVim Model Layers Embedding dim

    Model configurations Table 25. Model configurations for FastVim Model Layers Embedding dim. Tiny 24 192 Small 24 384 Base 24 768 Large 48 1024 Huge 64 1280 18 (a) Autocast as False (b) Autocast as True Figure 8. Comparison of Inference Throughput (it/s) for FastVim, Vim, and V...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.