Pith. sign in

REVIEW 3 major objections 5 minor 35 references

A Study on Context Length and Efficient Transformers for Biomedical Image Analysis

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that replacing self-attention with attention-free operators like Hyena and MambaVision maintains accuracy while cutting training time by over 80 percent on long-context biomedical imaging tasks, and that smaller patches…

desk verdict A useful empirical benchmark with a real framing problem: the patch-size gains are solid, but attributing them to context length is undercut by the paper's own window-size control. read the letter →

arxiv 2501.00619 v1 pith:G24B3CT2 submitted 2024-12-31 cs.CV cs.AIcs.LG

Sarah M. Hooper , Hui Xue This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords biomedicalimagingcontextlengthself-attentionefficienttransformersHyenaMambaVisionTransformerSwin
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether biomedical image analysis benefits from longer transformer context, and whether recent attention-free operators can deliver that benefit cheaply. Across six datasets spanning 2D and 3D segmentation, denoising, and classification, the authors find that shrinking the ViT patch size, which lengthens the token sequence, consistently improves performance, most strongly for pixel-level prediction. Increasing Swin's attention window size, by contrast, yields only minor gains. The paper then shows that replacing self-attention with Hyena or MambaVision maintains or slightly improves accuracy while speeding up training by over 80 percent at long context lengths, and fits configurations that self-attention cannot run on the available hardware.

What carries the argument

The paper's central object is the context length of a vision transformer, the number of image tokens processed together, and the two levers used to change it: patch size in ViT (smaller patches give more tokens) and attention window size in Swin (larger windows attend over more tokens at once). The argument that efficiency can be gained without losing accuracy is carried by two attention replacements: Hyena, which uses long convolutions with gating and scales as O(n $log^{2}$ n), and MambaVision, a selective state-space operator that scans tokens through a learnable hidden state. These operators are swapped into the same transformer blocks in place of self-attention, so any performance difference is attributed to the operator itself rather than to architecture changes. This comparison is what supports the claim that the operators replicate attention's accuracy while removing its quadratic cost.

What would settle it

Run a ViT on the same datasets with (a) fixed image resolution and variable patch count and (b) fixed patch count and variable image resolution; if performance tracks resolution rather than token count, the paper's conclusion that longer context drives the gains would be refuted, and the Swin window-size sweep already provides a control pointing in that direction.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that context length is a major lever for transformer performance in biomedical imaging, but only when it is increased by preserving image resolution through smaller patches; enlarging attention windows gives little benefit. As a practical corollary, the paper argues that attention-free operators, Hyena (built on long convolutions with subquadratic O(n $log^{2}$ n) scaling) and MambaVision (built on a selective state-space scan), can serve as drop-in replacements for self-attention in ViT and Swin backbones. In its experiments these operators match or exceed self-attention accuracy on segmentation, denoising, and classification, cut training time by up to roughly 80 percent at the longest contexts, and allow the use of patch sizes that exceed the memory available to self-attention. This is offered as evidence that efficient long-context processing, not quadratic attention itself, is what matters for high-resolution biomedical imaging.

Load-bearing premise

The paper's headline claim rests on treating smaller patch size as a pure way to lengthen context, but smaller patches also preserve more image detail; since larger attention windows, which lengthen context without changing resolution, show only minor gains, the patch-size results may be driven by resolution rather than by context length.

Editorial extensions

If this is right

  • Smaller patch sizes improve performance across segmentation, denoising, and classification, with the largest gains on pixel-level tasks, so future backbones for medical imaging should prioritize resolution-preserving tokenization.
  • Hyena and MambaVision can replace self-attention in ViT and Swin with little or no accuracy loss, which would let laboratories train high-resolution models that currently exceed GPU memory.
  • At long context lengths these operators speed up training by over 80 percent relative to self-attention, making them a practical choice for clinical-scale workloads.
  • The Swin results suggest that local windows plus hierarchical merging already capture enough context for many tasks, so effort spent enlarging windows is less valuable than effort spent preserving resolution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run is to decouple resolution from context length directly, for instance by keeping patch count fixed while varying input resolution; the Swin window results already hint that resolution, not token count, is the active ingredient.
  • If the resolution interpretation is right, then the efficiency gains of Hyena and MambaVision become even more useful: they let models afford resolution-preserving tokenization that quadratic attention simply cannot fit in memory.
  • The same operator swap is likely to transfer to other dense prediction tasks such as instance segmentation or image restoration, where fine detail matters more than global reasoning.
  • The classification gap for Swin with attention-free operators is likely attributable to the removed shift operation, which suggests that an efficient shifted-window implementation for Hyena and MambaVision would close most of that gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents an empirical study of how transformer context length affects performance on biomedical image analysis, using six curated datasets spanning 2D/3D data and segmentation, denoising, and classification tasks. The authors vary ViT patch size and Swin attention window size as two ways to change context length, and they evaluate Hyena and MambaVision as drop-in replacements for self-attention in ViT/Swin backbones. They report that smaller patches strongly improve performance, especially for pixel-level tasks; that attention window size has only minor effects in Swin; and that Hyena and MambaVision achieve comparable performance to attention while providing large training-speedups at long context lengths and enabling configurations that self-attention cannot fit in memory.

Significance. If the efficiency results hold, the paper provides a practically useful benchmark for biomedical imaging: it demonstrates that Hyena and MambaVision can substitute for self-attention with modest accuracy changes and large training-speed gains at long context lengths, and that smaller-patch tokenization improves ViT/Swin performance across several tasks. Strengths include the diversity of tasks and modalities, bootstrapped confidence intervals, the reporting of parameter counts and learning rates, and appendix control experiments such as the Swin shift-removal study. The main interpretive limitation is that the central claim—that there is a strong relationship between context length and performance—is not supported by the paper's own controlled manipulation, because the patch-size sweep confounds context length with input resolution; the Discussion itself attributes the findings to preserving high-resolution information. This reduces the strength of the abstract's headline claim but does not eliminate the value of the empirical measurements.

major comments (3)
  1. [Section 3.3, Figures 4 and 5; Abstract and Section 5] The claim of a "strong relationship between context length and performance" is not supported by the paper's own window-size control experiment. Varying the ViT patch size changes both the number of tokens (context length) and the spatial resolution of each token, so the two effects are never separated. The Swin attention-window sweep, which holds resolution and the patch embedding fixed and isolates context length, shows only minor impacts for most tasks in Figure 5, with CMR denoising the only clear gain. The Discussion in Section 5 instead attributes the patch-size results to "preserving high-resolution information," which is a different mechanism from the abstract's framing. Please reframe the abstract and introduction to claim that token granularity, rather than context length per se, drives the improvements, or add an experiment that varies context length while holding resolution fixed (for example, by padding token sequences or by testing larger input images at a fixed window size).
  2. [Section 3.3, Tables 5-7] The ViT patch-size comparison is additionally confounded by parameter-count changes and by per-configuration learning-rate tuning. Table 7 shows, for example, the Abdominal CT attention backbone having 33.9M parameters at patch 32 but 23.2M at patch 16, and Tables 5-6 show learning rates varying across patch sizes for the same operator and task (e.g., CMR ViT with attention uses 1e-3 at patch 4 but 1e-2 at patches 8, 16, and 32). Because both the model size and the optimizer setting vary, the observed performance differences cannot be attributed solely to context length or even to resolution. Please quantify the effect of the parameter-count drift, hold the learning rate fixed across the sweep as a sensitivity check, or explicitly justify that per-configuration tuning does not change the qualitative conclusions.
  3. [Section 4.3, Tables 3-4 and Figures 6-9] The reported "over 80% speedup" at ViT patch size 4 is presented without specifying which configurations are included in the average. For several tasks, self-attention cannot run at patch 4 (indicated by X in Figure 4), so the average may be computed over a subset of tasks where both operators run, or it may compare against attention at a different patch size. Please state explicitly which tasks contribute to each average in Tables 3-4, whether self-attention was feasible at that configuration, and note in the main text that the timings are for the backbone only with batch size one; this determines how the headline efficiency claim should be interpreted.
minor comments (5)
  1. [Figure 5 caption] The caption says "We visualize performance for each task, operator, and patch size" but the x-axis is the attention window size, not the patch size; please correct the caption to avoid misleading readers.
  2. [Table 7, Embolism row] The text "MaMambaVisionmba" appears to be a typo and should read "MambaVision".
  3. [Appendix B.2.1, Table 9] The text states that smaller Swin patches improved performance on all six tasks, but the Microscopy row shows identical SSIM values (0.60 with overlapping confidence intervals) for patch 4 and patch 2; please correct the sentence to reflect the actual pattern, noting that five of six tasks improved.
  4. [Section 1] The sentence "these operators can achieve comparable performance to self-attention while improving efficiency by over 80%" is broader than the evidence; the 80% figure applies only to the longest-context ViT configuration (patch 4) and is based on backbone-only timings, so the sentence should be qualified with those conditions.
  5. [Section 3.3, first paragraph] The phrase "how many tokens are processed concurrently" is imprecise for Swin, where attention is computed within each local window rather than over the whole image; consider phrasing such as "the number of tokens attended to within a window."

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: an empirical benchmark whose central claim is confounded by resolution, not reduced to its inputs by construction.

full rationale

This paper is an empirical evaluation, not a derivation. It curates six datasets, trains ViT and Swin with varying patch and window sizes, and compares self-attention against Hyena and MambaVision. No parameter is fitted to a target metric and then reported as a prediction of that same metric; learning rates are tuned per configuration, which is standard practice and does not construct the reported Dice, SSIM, or AUROC values. The paper cites MambaVision, Mamba, and Hyena from other groups and uses their public implementations; none of the load-bearing claims about efficiency or performance rest on a self-citation chain. The abstract's 'strong relationship between context length and performance' is an empirical generalization, not a definitional equivalence. The paper's own Section 2 notes that 'image resolution and context length are closely linked,' and the Swin window-size sweep in Section 4.2 shows only minor effects, so the patch-size results may be largely a resolution effect. That is a validity and confounding concern, not circularity: the reported numbers are measurements of real held-out performance, and the claim does not equal its input by construction. Under the provided rubric, an honest non-finding is appropriate: score 0 with no circular steps.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new theoretical entities are introduced; all model components come from prior work. The ledger's parameters and assumptions are experimental design choices that the central empirical claims rest on.

free parameters (4)
  • learning rate per experiment = 1e-5 to 1e-2 depending on config (Tables 5-6)
    Tuned per architecture, operator, patch/window size by validation performance; affects all performance comparisons.
  • CMR denoising SNR reduction ratio = uniform in [1, 40]
    Synthetic MRI noise level chosen by the authors for the private dataset; drives denoising difficulty and the window-size effect observed for CMR.
  • denoising loss weights = equal weights (stated as 'sum of' MSE, Charbonnier, Gaussian)
    No weights or sensitivity analysis given; choice affects denoising performance numbers.
  • CT window settings and crop sizes = abdominal: level 50/width 400, 64 slices; embolism: level 100/width 700, 64 slices
    Preprocessing choices that change input content and task difficulty; not swept.
assumptions (5)
  • standard math Self-attention complexity is O(n^2) (Keles et al., 2023)
    Used in Section 3.1 to motivate efficiency evaluation; accepted from cited literature.
  • domain assumption Six curated datasets are representative of biomedical imaging tasks
    Generalization of conclusions beyond these datasets is assumed; dataset sizes are small per the paper's own limitations.
  • domain assumption Removing Swin's shift for Hyena/MambaVision gives a fair operator comparison
    The paper tests the shift's impact in Appendix B.2.2 and finds classification gaps, so the assumption is acknowledged but still baked into main results.
  • domain assumption Checkpointing by minimum validation loss and 250 epochs yields converged, comparable models
    No convergence analysis; assumes training budget is sufficient for all configurations.
  • ad hoc to paper The synthetic MRI noise model represents realistic cardiac MR denoising
    The private CMR dataset is degraded with a uniform SNR ratio; the realism of this model is not validated against real low-SNR acquisitions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Study on Context Length and Efficient Transformers for Biomedical Image Analysis." pith.science (2026). https://pith.science/paper/G24B3CT2

@misc{pith2026250100619,
  author       = {Pith},
  title        = {Pith review of: A Study on Context Length and Efficient Transformers for Biomedical Image Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G24B3CT2}},
  note         = {Machine review of arXiv:2501.00619}
}
read the original abstract

Biomedical imaging modalities often produce high-resolution, multi-dimensional images that pose computational challenges for deep neural networks. These computational challenges are compounded when training transformers due to the self-attention operator, which scales quadratically with context length. Recent developments in long-context models have potential to alleviate these difficulties and enable more efficient application of transformers to large biomedical images, although a systematic evaluation on this topic is lacking. In this study, we investigate the impact of context length on biomedical image analysis and we evaluate the performance of recently proposed long-context models. We first curate a suite of biomedical imaging datasets, including 2D and 3D data for segmentation, denoising, and classification tasks. We then analyze the impact of context length on network performance using the Vision Transformer and Swin Transformer by varying patch size and attention window size. Our findings reveal a strong relationship between context length and performance, particularly for pixel-level prediction tasks. Finally, we show that recent long-context models demonstrate significant improvements in efficiency while maintaining comparable performance, though we highlight where gaps remain. This work underscores the potential and challenges of using long-context models in biomedical imaging.

Figures

Figures reproduced from arXiv: 2501.00619 by the authors.

Figure 1
Figure 1. Visualization of how context length changes with patch size and attention window size. When using ViT, we use smaller patches to tokenize the input image, resulting in longer context lengths. When using Swin, we use larger windows of attention, resulting in longer context lengths. cal imaging applications benefit from longer context, and if so, what are efficient and effective approaches for training long-context mo… view at source ↗
Figure 2
Figure 2. Attention and alternative operators. Left, we show a standard transformer block. Right, we show the operators we evaluate in the transformer blocks: self-attention, Hyena, and MambaVision. The computational complexity of self-attention is O(n 2 ) (Keles et al., 2023), meaning using self￾attention with longer sequences results in quadratic increases to memory and computation. Alternatives to Attention. Many alternati… view at source ↗
Figure 3
Figure 3. Task visualization. We visualize a network input and ground truth output for each task. Starting from the upper left and moving clockwise: retinal vessel segmentation, microscopy denoising, pneumothorax classification, pulmonary embolism classification, CMR denoising, and abdominal CT organ segmentation. in NLP. Swin restricts attention to local windows, then shifts and merges these windows. By stacking multiple Swi… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: ViT performance. We visualize performance for each task, operator, and patch size with 95% confidence intervals. An X on the x-axis indicates the patch size exceeded our hardware capacity. achieving improved performance across the four pixel-level prediction tasks with…
Figure 5
Figure 5. Figure 5: Swin performance. We visualize performance for each task, operator, and patch size with 95% confidence intervals. An X on the x-axis indicates that the window size exceeded our hardware capacity [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: ViT timing. We visualize timing for a forward and backward pass for each task, operator, and patch size. An X on the x-axis indicates that the patch size exceeded our hardware capacity. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Swin timing. We visualize timing for a forward and backward pass for each task, operator, and patch size. An X on the x-axis indicates that the window size exceeded our hardware capacity [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: ViT maximum memory allocated. We visualize maximum memory allocated for each task, op￾erator, and patch size. An X on the x-axis indicates that the patch size exceeded our hardware capacity. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Swin maximum memory allocated. We visualize maximum memory allocated for each task, oper￾ator, and patch size. An X on the x-axis indicates that the window size exceeded our hardware capacity. ETR, or UPerNet task heads). We plot the maximum memory allocated for each d…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 7 canonical work pages

  1. [4]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 ,

    arXiv 1904

  2. [6]

    We required a minimum batch size of two to fit on the GPU to enable batch normalization layers

    We set batch size to maximize GPU memory. We required a minimum batch size of two to fit on the GPU to enable batch normalization layers. A.2. Data Preprocessing For the retinal vessel segmentation dataset (Jin et al., 2022), we directly used the public data with no addi- tional preprocessing. When training the Swin mod- els, we resized the images to 1024...

    work page 2022

  3. [7]

    A survey on long text modeling with trans- formers

    Zican Dong, Tianyi Tang, Lunyi Li, and Wayne Xin Zhao. A survey on long text modeling with trans- formers. arXiv preprint arXiv:2302.14502 ,

    arXiv

  4. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias 10 A Study on Context Length and Efficient Transformers for Biomedical Image Analysis Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint...

    arXiv 2010

  5. [9]

    Hun- gry hungry hippos: Towards language model- ing with state space models

    Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher R´ e. Hun- gry hungry hippos: Towards language model- ing with state space models. arXiv preprint arXiv:2212.14052,

    arXiv

  6. [10]

    Mamba: Linear-time se- quence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time se- quence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    arXiv

  7. [11]

    Ef- ficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R´ e. Ef- ficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021a. Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R´ e. Combin- ing recurrent, convolutional, and continuous-time models with linear state space layers. Advances in n...

    arXiv

  8. [12]

    Mambavision: A hybrid mamba-transformer vision backbone

    Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083,

    arXiv

Show all 35 references
  1. [13]

    Exploring long-sequence masked autoencoders

    Ronghang Hu, Shoubhik Debnath, Saining Xie, and Xinlei Chen. Exploring long-sequence masked autoencoders. arXiv preprint arXiv:2210.07224 ,

    arXiv

  2. [14]

    Advancing trans- former architecture in long-context large language models: A comprehensive survey

    Yunpeng Huang, Jingwei Xu, Zixu Jiang, Junyu Lai, Zenan Li, Yuan Yao, Taolue Chen, Lijuan Yang, Zhou Xin, and Xiaoxing Ma. Advancing trans- former architecture in long-context large language models: A comprehensive survey. arXiv preprint arXiv:2311.12351,

    arXiv

  3. [15]

    Optimizing vision transformer per- formance with customizable parameters

    E Ibrahimovic. Optimizing vision transformer per- formance with customizable parameters. In 2023 46th MIPRO ICT and Electronics Convention (MIPRO), pages 1721–1726. IEEE,

    2023

  4. [16]

    U-mamba: Enhanc- ing long-range dependency for biomedical image segmentation

    Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhanc- ing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722 ,

    arXiv

  5. [17]

    Mm1: Methods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Faucon- nier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611,

    arXiv

  6. [18]

    Deepstack: Deeply stacking visual tokens is sur- prisingly simple and effective for lmms

    Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deepstack: Deeply stacking visual tokens is sur- prisingly simple and effective for lmms. arXiv preprint arXiv:2406.04334,

    arXiv

  7. [19]

    An image is worth more than 16x16 patches: Exploring transformers on individual pixels

    Duy-Kien Nguyen, Mahmoud Assran, Unnat Jain, Martin R Oswald, Cees GM Snoek, and Xinlei Chen. An image is worth more than 16x16 patches: Exploring transformers on individual pixels. arXiv preprint arXiv:2406.09415,

    arXiv

  8. [20]

    The what, why, and how of context length extension techniques in large language models–a detailed survey

    Saurav Pawar, SM Tonmoy, SM Zaman, Vinija Jain, Aman Chadha, and Amitava Das. The what, why, and how of context length extension techniques in large language models–a detailed survey. arXiv preprint arXiv:2401.07872,

    arXiv

  9. [21]

    Efficientv- mamba: Atrous selective scan for light weight vi- sual mamba

    Xiaohuan Pei, Tao Huang, and Chang Xu. Efficientv- mamba: Atrous selective scan for light weight vi- sual mamba. arXiv preprint arXiv:2403.09977 ,

    arXiv

  10. [22]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Al- balak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048 ,

    arXiv

  11. [24]

    Retentive network: A successor to trans- former for large language models

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to trans- former for large language models. arXiv preprint arXiv:2307.08621,

    arXiv

  12. [26]

    Preliminary study on patch sizes in vision transformers (vit) for covid-19 and diseased lungs classification

    Joel CM Than, Pun Liang Thon, Omar Mohd Ri- jal, Rosminah M Kassim, Ashari Yunus, Norl- iza M Noor, and Patrick Then. Preliminary study on patch sizes in vision transformers (vit) for covid-19 and diseased lungs classification. In 2021 IEEE National Biomedical Engineering Conf...

    2021

  13. [27]

    Attention is all you need

    Ashish Vaswani. Attention is all you need. arXiv preprint arXiv:1706.03762,

    arXiv

  14. [28]

    Mamba-unet: Unet-like pure visual mamba for medical image segmentation

    Ziyang Wang, Jian-Qing Zheng, Yichi Zhang, Ge Cui, and Lei Li. Mamba-unet: Unet-like pure visual mamba for medical image segmentation. arXiv preprint arXiv:2402.05079,

    arXiv

  15. [29]

    Segmamba: Long-range sequential mod- eling mamba for 3d medical image segmentation

    Zhaohu Xing, Tian Ye, Yijun Yang, Guang Liu, and Lei Zhu. Segmamba: Long-range sequential mod- eling mamba for 3d medical image segmentation. arXiv preprint arXiv:2401.13560 ,

    arXiv

  16. [30]

    W2s: microscopy data with joint denoising and super- resolution for widefield to sim mapping

    Ruofan Zhou, Majed El Helou, Daniel Sage, Thierry Laroche, Arne Seitz, and Sabine S¨ usstrunk. W2s: microscopy data with joint denoising and super- resolution for widefield to sim mapping. In Com- puter Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, ...

    2020

  17. [31]

    Vi- sion mamba: Efficient visual representation learn- ing with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xin- long Wang, Wenyu Liu, and Xinggang Wang. Vi- sion mamba: Efficient visual representation learn- ing with bidirectional state space model. arXiv preprint arXiv:2401.09417,

    arXiv

  18. [35]

    95% confidence intervals are reported in parentheses, com- puted by bootstrapping over the test set

    For seg- mentation, we report Dice; for denoising, we report SSIM; and for classification, we report AUROC. 95% confidence intervals are reported in parentheses, com- puted by bootstrapping over the test set. Table 9: Effect of patch size on Swin performance (95% confidence in...

    2021

  19. [400]

    For the microscopy denoising dataset (Zhou et al., 2020), we treated each of the three supplied channels in the public dataset as different images

    We resized each axial image using linear interpolation to 256 × 256 and center cropped to 64 axial slices. For the microscopy denoising dataset (Zhou et al., 2020), we treated each of the three supplied channels in the public dataset as different images. We selected a single f...

    2020

  20. [700]

    We cropped around the lung region then resized each axial slice to 256 × 256 and center cropped the axial slices to 64 slices, ensuring the embolism was captured in the cropped region. A.3. Model Implementation We used the ViT and Swin implementations from Monai (Cardoso et al...

    2022

  21. [2017]

    Efficient transformers: A survey

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. CoRR, abs/2009.06732,

    2009 arXiv

  22. [2019]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sar- los, Peter Hawkins, Jared Davis, Afroz Mohiud- din, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794,

    2009 arXiv

  23. [2020]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vi- jay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08608,

    arXiv

  24. [2021]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

    arXiv

  25. [2022]

    Vim-unet: Vi- sion mamba for biomedical segmentation

    Anwai Archit and Constantin Pape. Vim-unet: Vi- sion mamba for biomedical segmentation. arXiv preprint arXiv:2404.07705,

    arXiv

  26. [2023]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Co- han. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 ,

    2004 arXiv

  27. [2024]

    Zoology: Measuring and improving recall in efficient language models

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher R´ e. Zoology: Measuring and improving recall in efficient language models. arXiv:2312.04927,

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.