Pith. sign in

REVIEW 4 major objections 5 minor 30 references

Surface Vision Mamba: Leveraging Bidirectional State Space Model for Efficient Spherical Manifold Representation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Mamba-based backbone for spherical brain surfaces claims transformer-level accuracy with 4.8x faster inference and 91.7% lower memory than a surface vision transformer.

desk verdict A real efficiency contribution with an overstated accuracy claim; the paper deserves peer review but needs qualification. read the letter →

arxiv 2501.14679 v5 pith:M2CTLP6M submitted 2025-01-24 cs.CV cs.AI

Rongzhao He , Weihao Zheng , Leilei Zhao , Ying Wang , Dalin Zhu , Dan Wu , Bin Hu This is my paper · ORCID
classification cs.CVcs.AI
keywords surfacevisionmambastatespacemodelsphericalmanifoldcorticalanalysisicospherepatchingneurodevelopmentalphenotypepredictionbrainageefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the Vision Mamba architecture, an attention-free state space model with bidirectional scanning, can be transplanted to spherical surfaces and serve as an efficient backbone for cortex analysis. The method, Surface Vision Mamba (SiM), turns a subdivided icosphere into a sequence of triangular patches, feeds both hemispheres plus a shared class token through bidirectional Mamba blocks, and regresses neurodevelopmental phenotypes from cortical morphometry. On neonatal brain-age prediction, fine-tuned SiM-Small/3 reaches MAE 0.56, edging out the Surface Vision Transformer's 0.60, while at the finer Ico-4 partitioning SiM is reported 4.8 times faster and uses 91.7% less GPU memory than SiT. The paper also reports gains on 18-month language and motor outcome prediction and on a replication dataset, and it uses sensitivity analysis to locate cortical regions that drive age prediction. A sympathetic reader would care because it offers a path to higher-resolution spherical surface analysis without the quadratic cost of attention.

What carries the argument

The load-bearing object is the Surface Vision Mamba block, a bidirectional state space model with a selective scan mechanism adapted from Vision Mamba. The spherical data is first turned into a consumable sequence by subdividing an icosphere into triangular patches, flattening each patch into a token, and inserting a class token between the left and right hemisphere token streams, with positional embeddings added before the bidirectional scan. The selective scan compresses relevant context into a hidden state, making runtime and memory scale sub-quadratically with sequence length, so finer icosphere orders, which produce longer token sequences, remain feasible where attention-based SiT runs out of memory.

What would settle it

Rerun the reported Ico-4 batch-inference benchmark on the released code: the claim stands only if SiM reproduces roughly 4.8x faster inference and 91.7% lower GPU memory than SiT on the same A100 hardware, and if fine-tuned SiM-Small/3 keeps MAE 0.56 on the held-out Subset 1 test set.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a bidirectional selective state space model, originally built for 1D visual tokens, is a drop-in, domain-agnostic backbone for genus-zero surfaces. SiM represents the cortical surface by subdividing an icosphere into triangular patches, flattening each patch into a token, concatenating left-hemisphere tokens, a learned class token, and right-hemisphere tokens, and adding positional embeddings before passing the sequence through bidirectional Mamba blocks. This formulation lets sequence length grow with icosphere subdivision order, and the paper's experiments show that with ImageNet-pretrained Vim initialisation, SiM-Small/3 predicts postmenstrual age with MAE 0.56 versus 0.60 for SiT-Small/3, while Ico-4 batch inference is 4.8x faster and consumes 91.7% less GPU memory than SiT. The paper is explicit that from-scratch training markedly worsens results and that autoregressive pretraining helped only marginally, so the headline advantage is tied to the fine-tuning route.

Load-bearing premise

The headline results rest on the assumption that ImageNet-pretrained Vision Mamba weights, learned from natural 2D images, transfer productively to sequences of flattened triangular patches of neonatal cortical surfaces.

Editorial extensions

If this is right

  • At fine grid resolutions such as Ico-4, SiM can process surface data that makes attention-based SiT-Base run out of memory, so the practical resolution of cortical surface analysis can increase for a fixed GPU budget.
  • Because SiM is domain-agnostic, the same icosphere-patching recipe should transfer to other genus-zero surfaces beyond the neonatal cortex.
  • The reported sensitivity maps give a concrete way to ask which cortical regions and which morphometric features, such as curvature, sulcal depth, thickness, and myelination, drive age prediction, which is directly useful for studying preterm birth effects.
  • If the efficiency gap holds, batch inference over large neonatal cohorts becomes feasible on a single 40G A100 device, lowering the barrier for clinical-scale application.
  • The paper's autoregressive pretraining results suggest that self-supervised pretraining on small medical surface datasets does not yet deliver the gains seen in natural-image vision, so data-efficient pretraining remains an open bottleneck for this class of models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The reported accuracy advantage is largely a statement about transfer from ImageNet-pretrained Vim weights rather than about the spherical geometry itself; a fair test of the architecture would compare all models under matched pretraining on surface data, and the same fine-tuning recipe may also lift SiT.
  • Editorial inference: The efficiency gain should grow with icosphere order, since SiM's cost is near-linear in sequence length while SiT's attention is quadratic, so at Ico-5 or finer the gap could be even larger than the Ico-4 numbers, though accuracy at such resolutions is untested here.
  • Editorial inference: The bidirectional scan order is only one of many possible traversals of the sphere; a scan ordered along cortical geodesics or by functional parcels might capture long-range structure better, a testable design choice the paper leaves open.
  • Editorial inference: The patching scheme's success suggests that any smooth signal on a genus-zero surface could be analyzed this way, making geoscience, cosmology, and omnidirectional vision problems with spherical data natural next testbeds, provided the sequence construction preserves enough local structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Surface Vision Mamba (SiM), a Vision Mamba adaptation for genus-zero spherical surfaces, representing cortical data as sequences of triangular patches from subdivided icospheres and processing them with a bidirectional state space model. The authors evaluate SiM on neonatal postmenstrual age prediction, 18-month language and motor outcome prediction, and a replication dataset, comparing against GDL and attention-based baselines under three training strategies (from scratch, ImageNet fine-tuning, and autoregressive pretraining). The headline claims are that SiM outperforms both attention- and GDL-based methods and delivers 4.8 times faster inference with 91.7% lower GPU memory than the Surface Vision Transformer at Ico-4.

Significance. An efficient, domain-agnostic backbone for cortical surface data would be a useful contribution, and the paper includes a credible engineering effort: the code is released, hyperparameters are provided in the appendix, and the sensitivity analysis offers an interpretability angle that is uncommon in this literature. The central accuracy claim, however, is only supported in the ImageNet-fine-tuned setting and is not supported by statistical testing. The efficiency claim is plausible but is reported from a single configuration without measurement details. If the authors condition their claims appropriately and add the missing ablation, the work would be a solid empirical contribution.

major comments (4)
  1. [Abstract and Section 4.2, Table 4] The unqualified claim that 'SiM outperforms both attention- and GDL-based methods' is contradicted by the Supervised (scratch) column of Table 4, where every SiM variant has higher MAE than the corresponding SiT variant; for example, SiM-Small/3 has MAE 1.09±0.84 versus SiT-Small/3 at 0.60±0.51. The advantage appears only in the Fine-tuning column, so the abstract and conclusion must either restrict the claim to ImageNet-fine-tuned models or provide a substantive explanation for the scratch results.
  2. [Section 3.5 and Appendix A.1, Table 4] The fine-tuning comparison is confounded with pretraining: SiM is initialized from Vim weights pretrained on ImageNet, while the GDL baselines are trained from scratch and the SiT fine-tuning baseline is not described as using comparably pretrained weights. Without an ablation that gives SiT the same pretraining advantage, or trains SiM from scratch with an equally strong recipe, the fine-tuned accuracy gap (0.56 vs 0.60 for Small/3) cannot be attributed to the SiM architecture, which is the paper's central architectural claim.
  3. [Section 4.2, Table 4] The reported MAE improvements are within overlapping standard deviations (e.g., SiM-Small/3 0.56±0.50 vs SiT-Small/3 0.60±0.43) and no significance test, confidence interval, or per-subject paired analysis is reported. With a held-out test set of roughly 10% of 408 subjects, the observed differences may be noise; please provide uncertainty quantification for the main comparisons.
  4. [Section 4.3, Figure 3] The 4.8 times speedup and 91.7% memory reduction are presented as headline results in the abstract, but Figure 3 reports only the tiny model and does not provide error bars, batch size, input dimensions, or a precise measurement protocol beyond '40G A100 device.' The efficiency claim should be reported with the exact experimental conditions and ideally across model sizes and multiple runs.
minor comments (5)
  1. [Table 2] The column headers 'First, Second, Third, Fourth, Fifth' should be explicitly mapped to Ico-1 through Ico-5; the main text interchangeably uses 'Ico-2' and 'second-order' without defining the correspondence.
  2. [Tables 5–7] The hyperparameter tables group columns by 'Scratch, Fine-tuning, AR Pretraining, AR Fine-tuning' with T/S/B variants, but it is not stated whether the same schedules apply to the SiT baselines; please clarify which entries pertain to SiT and which to SiM.
  3. [Section 3.4] The text says the paper evaluates 'first- to third-order icosphere' patching methods, while Table 2 lists configurations up to fifth order; please reconcile this discrepancy and state explicitly which icosphere orders are used in which experiment.
  4. [Section 3.3] The class token is concatenated between the left and right hemisphere tokens, but the regression head that consumes the final class token is not described; please specify the output head and loss in the main text.
  5. [Appendix B.2, Figure 5] The paired t-test result is reported only as p<0.01; please include the test statistic and degrees of freedom.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SiM's claims rest on external ImageNet initialization and empirical comparisons; the only self-citation (HRINet) is a baseline, not a load-bearing premise.

full rationale

I walked the claimed derivation chain. The paper does not derive its target result from itself: SiM is an empirical adaptation of the externally published Vision Mamba architecture, applied to spherical cortical surfaces via deterministic icosphere patching and standard SSM equations (Eqs. 1-5). No fitted parameter is renamed as a prediction; the efficiency numbers (4.8x faster, 91.7% lower memory) and accuracy numbers (e.g., SiM-S/3 MAE 0.56 vs SiT-S/3 0.60) are measured against external baselines including SiT and GDL methods. The ImageNet-pretrained Vim weights are an external initialization, not fit to the dHCP target data, so the fine-tuned advantage is not forced by construction. The autoregressive pretraining is self-supervised on the same dataset, but the paper honestly reports its limited benefit ('the benefits of autoregressive pretraining were limited, likely due to constraints of small samples'), and the evaluation uses held-out test splits; this is standard empirical practice, not circular reasoning. The only author-overlapping citation is HRINet [Zhao et al., 2024], which appears as a comparison baseline and nowhere justifies the central claim. The paper itself discloses that scratch-trained SiM underperforms SiT in Table 4, and the abstract's unqualified superiority claim is arguably overstated relative to that table; however, overstatement is a correctness/validity concern, not circularity. No uniqueness theorem is imported from the authors, no ansatz is smuggled in via a self-citation, and no known result is merely renamed. Hence no circular step reaches the threshold required for a nonzero score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and therefore rests on domain assumptions rather than analytic axioms. No free parameters are fit to derive the result; all model hyperparameters are reported. The main load-bearing premises are that a linear bidirectional scan over flattened icosphere patches captures spherical long-range dependencies, that ImageNet-pretrained Vim weights transfer to cortical metrics, and that the Ico-6 resampling preserves the information needed for prediction. No new entities are introduced.

assumptions (4)
  • domain assumption Bidirectional 1D Mamba scans over a flattened list of icosphere patches can capture long-range dependencies on the sphere.
    Entire architecture; the paper does not justify that a single linear patch ordering plus reverse scan is sufficient for spherical topology (Section 3.3, Figure 2).
  • domain assumption ImageNet-pretrained Vim weights transfer to neonatal cortical surface metrics.
    Best results use ImageNet fine-tuning (Section 3.5, Table 4); scratch training degrades noticeably, so this transfer is load-bearing.
  • domain assumption Resampling the 32,492-vertex dHCP template to Ico-6 via barycentric interpolation preserves the morphometric features and surface topology enough for prediction.
    Section 3.1, used for all experiments; no validation of resampling fidelity.
  • standard math ZOH discretization of SSM equations is valid for the discrete input sequence.
    Preliminaries Section 3.2, standard background from prior SSM literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Surface Vision Mamba: Leveraging Bidirectional State Space Model for Efficient Spherical Manifold Representation." pith.science (2026). https://pith.science/paper/M2CTLP6M

@misc{pith2026250114679,
  author       = {Pith},
  title        = {Pith review of: Surface Vision Mamba: Leveraging Bidirectional State Space Model for Efficient Spherical Manifold Representation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M2CTLP6M}},
  note         = {Machine review of arXiv:2501.14679}
}
read the original abstract

Attention-based methods have demonstrated exceptional performance in modelling long-range dependencies on spherical cortical surfaces, surpassing traditional Geometric Deep Learning (GDL) models. However, their extensive inference time and high memory demands pose challenges for application to large datasets with limited computing resources. Inspired by the state space model in computer vision, we introduce the attention-free Vision Mamba (Vim) to spherical surfaces, presenting a domain-agnostic architecture for analyzing data on spherical manifolds. Our method achieves surface patching by representing spherical data as a sequence of triangular patches derived from a subdivided icosphere. The proposed Surface Vision Mamba (SiM) is evaluated on multiple neurodevelopmental phenotype regression tasks using cortical surface metrics from neonatal brains. Experimental results demonstrate that SiM outperforms both attention- and GDL-based methods, delivering 4.8 times faster inference and achieving 91.7% lower memory consumption compared to the Surface Vision Transformer (SiT) under the Ico-4 grid partitioning. Sensitivity analysis further underscores the potential of SiM to identify subtle cognitive developmental patterns. The code is available at https://github.com/Rongzhao-He/surface-vision-mamba.

Figures

Figures reproduced from arXiv: 2501.14679 by the authors.

Figure 1
Figure 1. Representative icosahedron discretized spherical surfaces [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed Surface Vision Mamba (SiM) architecture. The cortical data from the left and right hemispheres are [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 1
Figure 1. To evaluate the impact of patching methods, we [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: Comparison of PMA prediction performance and efficiency between SiT and our SiM. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 4
Figure 4. Figure 4: Spatial distribution of informative vertices for PMA pre [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The significant difference between predicted and chrono [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 14 canonical work pages

  1. [1]

    Longformer: The long-document trans- former

    [Beltagy et al., 2020] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150,

    arXiv 2020

  2. [4]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    [Dao, 2023] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

    arXiv 2023

  3. [7]

    Benchmarking geometric deep learning for cortical segmentation and neurodevel- opmental phenotype prediction

    [Fawaz et al., 2021] Abdulah Fawaz, Logan ZJ Williams, Amir Alansary, Cher Bass, Karthik Gopinath, Mariana da Silva, Simon Dahan, Chris Adamson, Bonnie Alexan- der, Deanne Thompson, et al. Benchmarking geometric deep learning for cortical segmentation and neurodevel- opmental phenotype prediction. bioRxiv, pages 2021–12,

    work page 2021

  4. [9]

    Mamba: Linear- time sequence modeling with selective state spaces

    [Gu and Dao, 2023] Albert Gu and Tri Dao. Mamba: Linear- time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    arXiv 2023

  5. [10]

    Hippo: Recurrent memory with optimal polynomial projections

    [Gu et al., 2020] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R ´e. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474–1487,

    2020

  6. [11]

    Efficiently modeling long sequences with structured state spaces

    [Gu et al., 2021] Albert Gu, Karan Goel, and Christopher R´e. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396,

    arXiv 2021

  7. [12]

    Vision gnn: An image is worth graph of nodes

    [Han et al., 2022] Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision gnn: An image is worth graph of nodes. Advances in neural information processing systems, 35:8291–8303,

    work page 2022

  8. [13]

    Hyperattention: Long-context attention in near- linear time

    [Han et al., 2023] Insu Han, Rajesh Jayaram, Amin Kar- basi, Vahab Mirrokni, David P Woodruff, and Amir Zandieh. Hyperattention: Long-context attention in near- linear time. arXiv preprint arXiv:2310.05869,

    arXiv 2023

Show all 30 references
  1. [14]

    Demystify mamba in vision: A linear attention perspective

    [Han et al., 2024] Dongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. arXiv preprint arXiv:2405.16605,

    2024 arXiv

  2. [17]

    Vmamba: Visual state space model,

    [Liu et al., 2024] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jian- bin Jiao, and Yunfan Liu. Vmamba: Visual state space model,

    2024

  3. [18]

    Recurrent neural networks

    [Medsker et al., 2001] Larry R Medsker, Lakhmi Jain, et al. Recurrent neural networks. Design and Applications , 5(64-67):2,

    2001

  4. [22]

    Sensitivity analysis for im- portance assessment

    [Saltelli, 2002] Andrea Saltelli. Sensitivity analysis for im- portance assessment. Risk analysis, 22(3):579–590,

    2002

  5. [24]

    Attention is all you need

    [Vaswani, 2017] A Vaswani. Attention is all you need. Ad- vances in Neural Information Processing Systems ,

    2017

  6. [25]

    Geometric deep learning for post- menstrual age prediction based on the neonatal white mat- ter cortical surface

    [V osyliuset al., 2020] Vitalis V osylius, Andy Wang, Cem- lyn Waters, Alexey Zakharov, Francis Ward, Loic Le Fol- goc, John Cupitt, Antonios Makropoulos, Andreas Schuh, Daniel Rueckert, et al. Geometric deep learning for post- menstrual age prediction based on the neonatal wh...

    2020

  7. [26]

    Linformer: Self-attention with linear complexity

    [Wang et al., 2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768,

    2020 arXiv

  8. [27]

    Mam- baout: Do we really need mamba for vision? arXiv preprint arXiv:2405.07992,

    [Yu and Wang, 2024] Weihao Yu and Xinchao Wang. Mam- baout: Do we really need mamba for vision? arXiv preprint arXiv:2405.07992,

    2024 arXiv

  9. [28]

    Spherical u-net on cortical surfaces: methods and applications

    [Zhao et al., 2019] Fenqiang Zhao, Shunren Xia, Zheng- wang Wu, Dingna Duan, Li Wang, Weili Lin, John H Gilmore, Dinggang Shen, and Gang Li. Spherical u-net on cortical surfaces: methods and applications. In Infor- mation Processing in Medical Imaging: 26th International Confe...

    2019

  10. [29]

    An attention-based hemispheric relation inference network for perinatal brain age predic- tion

    [Zhao et al., 2024] Leilei Zhao, Dalin Zhu, Xiaomin Wang, Xia Liu, Tongtong Li, Boyang Wang, Zhijun Yao, Wei- hao Zheng, and Bin Hu. An attention-based hemispheric relation inference network for perinatal brain age predic- tion. IEEE Journal of Biomedical and Health Informatics ,

    2024

  11. [30]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    [Zhu et al., 2024] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417,

    2024 arXiv

  12. [2001]

    Geometric deep learning on graphs and manifolds using mixture model cnns

    [Monti et al., 2017] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE conference on computer vision and pattern recogniti...

    2017

  13. [2002]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    [Shah et al., 2024] Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08608,

    2024 arXiv

  14. [2017]

    Blockwise self- attention for long document understanding

    [Qiu et al., 2019] Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self- attention for long document understanding. arXiv preprint arXiv:1911.02972,

    2019 arXiv

  15. [2018]

    An image is worth 16x16 words: Transformers for image recognition at scale

    [Dosovitskiy, 2020] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,

    2020 arXiv

  16. [2019]

    Autoregres- sive pretraining with mamba in vision

    [Ren et al., 2024] Sucheng Ren, Xianhang Li, Haoqin Tu, Feng Wang, Fangxun Shu, Lei Zhang, Jieru Mei, Lin- jie Yang, Peng Wang, Heng Wang, et al. Autoregres- sive pretraining with mamba in vision. arXiv preprint arXiv:2406.07537,

    2024 arXiv

  17. [2020]

    Surface vision transformers: Attention-based modelling applied to cortical analysis

    [Dahan et al., 2022] Simon Dahan, Abdulah Fawaz, Lo- gan ZJ Williams, Chunhui Yang, Timothy S Coalson, Matthew F Glasser, A David Edwards, Daniel Rueck- ert, and Emma C Robinson. Surface vision transformers: Attention-based modelling applied to cortical analysis. In Internatio...

    2022

  18. [2021]

    Gpt-3: Its nature, scope, limits, and conse- quences

    [Floridi and Chiriatti, 2020] Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and conse- quences. Minds and Machines, 30:681–694,

    2020

  19. [2022]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    [Dao et al., 2022] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems , 35:16344–16359,

    2022

  20. [2023]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    [Devlin, 2018] Jacob Devlin. Bert: Pre-training of deep bidi- rectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    2018 arXiv

  21. [2024]

    Agent attention: On the integration of soft- max and linear attention

    [Han et al., 2025] Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang. Agent attention: On the integration of soft- max and linear attention. In European Conference on Computer Vision, pages 124–140. Springer,

    2025

  22. [2025]

    Swin transformer: Hierarchical vision transformer using shifted windows

    [Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF in- ternational conference on computer vision , pages 10012– 10022,

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.