Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Rectifying Magnitude Neglect in Linear Attention

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that restoring the query magnitude to linear attention recovers softmax-like focus at linear complexity, improving accuracy across vision, language, speech, and generation.

desk verdict New offset-normalized linear attention that works well empirically, but the concentration proof is only valid under an unquantified positivity condition that cannot hold for unbounded scaling. read the letter →

arxiv 2507.00698 v3 pith:VHZIJ4XY submitted 2025-07-01 cs.CV

Qihang Fan , Huaibo Huang , Yuang Ai , Ran He This is my paper · ORCID
classification cs.CV
keywords linearattentionquerymagnitudesoftmaxscoredistributionvisiontransformermagnitude-awarenormalizationefficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Softmax attention gets sharper as the query vector grows, because larger magnitudes make already-favored keys win even harder. Standard linear attention cannot do this: its normalization cancels the magnitude of the transformed query, so the score distribution stays smooth no matter how the query scales. The paper identifies this “magnitude neglect” as a key cause of linear attention’s underperformance and proposes Magnitude-Aware Linear Attention (MALA), which rewrites each score as $\beta\phi(Q)\phi(K)^\top - \gamma$ with $\beta$ and $\gamma$ fixed by normalization. MALA makes attention concentrate like softmax as the query magnitude grows, but at a gentler fractional rate, and the paper reports gains over both softmax and prior linear attention on image classification, detection, segmentation, NLP, speech, and generation.

What carries the argument

The load-bearing object is the affine attention score $$\text{Attn}(Q_i,K_j) = \$\beta$\,\$\varphi$(Q_i)\$\varphi$(K_j)^\top - \gamma,$$ with $$\$\beta$ = 1 + \frac{1}{\$\varphi$(Q_i)\sum_{m=1}^N \$\varphi$(K_m)^\top},\qquad \gamma = \frac{\$\varphi$(Q_i)\sum_{m=1}^N \$\varphi$(K_m)^\top}{N},$$ chosen so the $N$ scores for query $i$ sum to $1$ by addition rather than division. Under a magnitude scaling $a>1$ of $\phi(Q_i)$, $\beta$ and $\gamma$ update as $\beta_{\mathrm{new}}=(\beta+a-1)/a$ and $\gamma_{\mathrm{new}}=a\gamma$, so the ratio between a high-scoring key and a lower-scoring key becomes $$\frac{A_m - \frac{a\$\beta$}{a+\$\beta$-1}\gamma}{A_n - \frac{a\$\beta$}{a+\$\beta$-1}\gamma},$$ which the paper proves is larger than the unscaled ratio whenever all scores are positive. This closed-form update is what lets MALA mimic softmax attention’s sharpening trend while changing the ratio only fractionally rather than exponentially.

What would settle it

Scan a trained MALA model’s attention scores over a large validation set for any $\beta\phi(Q_i)\phi(K_j)^\top - \gamma \le 0$; finding one would falsify the positivity premise used in the proof of $p_m>p$. A more direct test is to fix a query, scale its magnitude by $a>1$, and verify that the ratio of its two highest positive scores strictly increases as the paper’s Equation (12) predicts.

Watch

Extended reading notes

Core claim

The paper’s central claim is that linear attention’s performance gap to softmax comes not from the kernel approximation per se but from discarding a single scalar: the query’s magnitude. In softmax attention, scaling the query by $a>1$ raises the score ratio between two keys to the power $a$, so attention concentrates sharply on already-favored keys. In linear attention, the normalization cancels $\|\phi(Q)\|$ exactly, so the distribution is fixed. MALA re-introduces the magnitude with an affine normalization: each score is $\beta\phi(Q_i)\phi(K_j)^\top-\gamma$, with $\beta$ and $\gamma$ chosen so scores sum to 1. When $\phi(Q_i)$ is scaled by $a$, the new $\beta$ and $\gamma$ are $(\beta+a-1)/a$ and $a\gamma$, and the ratio of two positive scores grows with $a$, along a fractional curve rather than an exponential one. The paper argues this reproduces softmax’s variation trend while staying more balanced, and attributes its empirical improvements across seven task families to that property.

Load-bearing premise

The proof that MALA’s attention gets spikier as the query magnitude grows assumes every attention score stays positive; if any score becomes negative or zero, the monotonicity argument no longer applies, and the paper offers only an empirical statement that such scores did not appear.

Editorial extensions

If this is right

  • In the paper’s ablations, MALA beats both the softmax baseline and every listed linear-attention baseline while keeping the same parameter counts and FLOPs.
  • The attention maps become locally concentrated like softmax’s but without collapsing to a few tokens, so MALA recovers local perception without a separate convolution branch.
  • The operator still computes as $\beta\phi(Q)(\sum_j \phi(K_j)^\top V_j) - \gamma \sum_j V_j$, so it retains linear complexity in token count and never forms the quadratic attention matrix.
  • The paper reports that this single mechanism transfers across image classification, detection, segmentation, NLP, speech recognition, and image generation.
  • Ablations show that removing either $\beta$ or $\gamma$, or learning them, collapses accuracy, identifying the closed-form rescaling and offset as the operative components.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit is that the fractional-growth rate provides a dial for attention sharpness: replacing the fixed $a\beta/(a+\beta-1)$ factor with a learned temperature would let a model tune spikiness continuously, which is testable but not explored.
  • The positivity assumption could be made unconditional by clipping scores or re-parameterizing $\gamma$; this variant is not examined in the paper and would turn the concentration proof into a guarantee.
  • Because MALA retains the associative linear-time computation, it should transplant into autoregressive and long-context settings, but the paper only reports small-scale NLP and speech results, so that reach is an inference rather than a demonstrated claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that linear attention ignores query magnitude, which makes its attention distribution too smooth compared with softmax attention, and proposes Magnitude-Aware Linear Attention (MALA). MALA adds a scale-dependent boost β and an offset γ so that attention scores still sum to one while the ratio between two keys' scores grows when the query is scaled, but grows fractionally rather than exponentially. The authors prove this ratio growth in the appendix under a positivity condition, assert that negative/zero scores never occur in their experiments, and report strong empirical results across image classification, detection, segmentation, NLP, speech recognition, and image generation. The central theoretical claim is the monotone-concentration property in Eq. (12), whose proof depends on an unverified and, for unbounded scaling, unsustainable positivity assumption.

Significance. If the positivity gap is closed, this is a useful and simple contribution: a parameter-free modification of linear attention that preserves O(N) complexity and shows consistent gains over softmax attention in direct ablations (Table 9: 75.1 vs 72.2 on DeiT-T; 85.3 vs 83.0 on Swin-S). The paper also includes extensive benchmarks across seven task families, a kernel-function ablation, and an ablation isolating the β and γ terms. However, the theoretical explanation for MALA's behavior currently rests on a positivity condition that is asserted empirically but not measured, and that cannot hold for unbounded magnitude scaling; therefore the mechanism claimed in Eqs. (10)-(12) is not yet established. The significance is conditional on repairing this proof or adding a nonnegative variant.

major comments (3)
  1. [Appendix A, Eq. (22); Section 3.3, Eq. (12)] The proof that p_m > p requires A_n/γ > aβ/(a+β−1) for the scaled query, i.e., all attention scores must remain positive along the scaling path. The manuscript justifies this with the statement in Section 3.3 that negative/zero scores have "actual frequency of occurrence equal zero," but no measurement protocol is given, no bound on the scale factor a is provided, and the assertion is made only for image classification, object detection, and segmentation, not for the NLP, speech, or generation experiments. Moreover, the condition cannot hold for all a in a nonuniform score vector: writing r_j = ϕ(Q_i)ϕ(K_j)^T and S = Σ r_j, the score after scaling ϕ(Q_i) by a is s_j(a) = a(r_j − S/N) + r_j/S. For any key with r_j < S/N, s_j(a) becomes negative for all sufficiently large a, and since not all r_j can exceed their mean unless they are all equal, unbounded magnitude scaling inevitably produces negative scores; the denominator in Eq. (22) then crosses zero. A concrete check is that for r_j < S/N, positivity requires a < (r_j/S)/(S/N − r_j). Consequently, the monotone-concentration claim of Eq. (12) is unproved over the magnitude range claimed, and the "fractional growth" advantage over Softmax Attention is not established. Please either (i) provide a formal or empirical bound on the range of a encountered at every layer, head, and token, with a specified verification protocol and results covering all reported task families, or (ii) modify Eq. (8) to guarantee nonnegative attention scores by construction. This gap is load-bearing because the paper's theoretical explanation of MALA's behavior is exactly this concentration property.
  2. [Section 3.3, Eq. (12) and Fig. 2] Even if p_m > p were proved, the paper's conclusion that "attention becomes more concentrated on keys that originally received higher attention" does not follow from the ratio of two keys alone; one must show that all scores above the mean increase while all scores below the mean decrease. In this specific construction that stronger property is in fact true, since s_j(a) = a(r_j − S/N) + r_j/S has derivative r_j − S/N with respect to a, independent of positivity. I recommend replacing the ratio-based argument with this direct per-score monotonicity argument, which is simpler and does not require positivity for the monotonicity itself. However, positivity is still needed to ensure that the resulting scores form a valid nonnegative attention distribution, so this repair does not remove the need to address the first major comment.
  3. [Section 4.8, Table 9] The direct comparisons to softmax attention are only reported at DeiT-T and Swin-T/Swin-S scales. The main tables compare MA ViT to published backbones with different architectures and training schedules, so the broad statement in Section 5 that MALA "achieves superior performance" over softmax attention is not fully supported by a same-architecture softmax baseline at the larger scales. A same-architecture softmax counterpart for at least the base or large setting would make the claim more rigorous. This is secondary to the positivity issue but affects the strength of the conclusion.
minor comments (5)
  1. [Eq. (6)] The expression "= p s" appears to be a typographical error for p^a; please correct the notation and use a consistent variable for the scaling factor throughout Section 3.2.
  2. [Table 1] The arrow notation "Softmax−→Linear" in Table 1 is unexplained; please specify that replacing Q by Q/||Q|| in softmax attention removes magnitude information and makes the distribution resemble linear attention.
  3. [References] Reference [16] is malformed (the title and author list are merged into one line), and references [39] and [40] appear to be the same paper; please consolidate and fix the bibliography.
  4. [Table 6] The benchmark name "LMB" is likely a typo for "LAMBADA"; if so, please spell out the benchmark name for clarity.
  5. [Section 3.3, Fig. 2 and Fig. 4] The visualizations of attention distributions are qualitative; adding a quantitative measure such as entropy or a Gini coefficient would strengthen the "more balanced" claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the magnitude-concentration property is derived from MALA's explicit beta/gamma construction, not fitted; self-citations are non-load-bearing.

full rationale

None of the paper's load-bearing derivations reduce to their inputs. Section 3.3 defines MALA via Eqs. (8)-(9) with beta and gamma chosen so that attention scores sum to 1; the concentration claim in Eq. (12) and its appendix proof (Eqs. (15)-(22)) are explicit algebraic consequences of that definition under a stated positivity condition. Beta and gamma are not learned or fitted to downstream results; the ablation in Table 11 even shows that replacing them with learnable parameters hurts performance. The empirical results in Tables 2-8 are measured against external baselines and tasks, so the reported gains are not constructed from the theory. The self-citations (RA VLT baseline [14], architecture components [10]-[13], diffusion training recipes [1,2]) are used as baselines or experimental settings, not as evidence for the magnitude-concentration property, and none is load-bearing. The only caveat is that the proof of p_m > p assumes all attention scores remain positive, as the appendix states: 'Since we only consider the positive attention scores, we have A_n/gamma > a*beta/(a+beta-1) > 1.' The paper asserts this holds empirically ('their actual frequency of occurrence is equal zero') rather than proving a bound. This is a correctness or domain-of-validity limitation, not a circularity, because the positivity condition is not the conclusion being derived and no fitted parameter is renamed as a prediction. Overall, the central derivation is self-contained.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the choice of a positive kernel, on the derived β/γ re-normalization (no fitted parameters), and on the empirical positivity of attention scores. No invented entities appear.

assumptions (3)
  • domain assumption The kernel function φ(.) = ELU(.)+1 is used as the linear attention kernel and is assumed adequate for the approximation.
    Used in Eq. (2) and Section 4.8. The concentration property is proven for any positive kernel, but experiments use this specific kernel, and the ablation suggests performance is not sensitive to the kernel.
  • domain assumption All attention scores under MALA are assumed to be positive in the proof of the monotonicity property.
    Section 3.3 and Appendix A require A_n > γ·c for the derivative argument; the paper asserts no negative scores occur in practice, but this is an unproved empirical claim.
  • domain assumption The standard linear attention formulation with a fixed kernel (no learned mixture) is assumed to be an adequate approximation to softmax.
    Standard assumption in linear attention literature, introduced in Eqs. (1)-(2). The paper does not question this backdrop.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rectifying Magnitude Neglect in Linear Attention." pith.science (2026). https://pith.science/paper/VHZIJ4XY

@misc{pith2026250700698,
  author       = {Pith},
  title        = {Pith review of: Rectifying Magnitude Neglect in Linear Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHZIJ4XY}},
  note         = {Machine review of arXiv:2507.00698}
}
read the original abstract

As the core operator of Transformers, Softmax Attention exhibits excellent global modeling capabilities. However, its quadratic complexity limits its applicability to vision tasks. In contrast, Linear Attention shares a similar formulation with Softmax Attention while achieving linear complexity, enabling efficient global information modeling. Nevertheless, Linear Attention suffers from a significant performance degradation compared to standard Softmax Attention. In this paper, we analyze the underlying causes of this issue based on the formulation of Linear Attention. We find that, unlike Softmax Attention, Linear Attention entirely disregards the magnitude information of the Query. This prevents the attention score distribution from dynamically adapting as the Query scales. As a result, despite its structural similarity to Softmax Attention, Linear Attention exhibits a significantly different attention score distribution. Based on this observation, we propose Magnitude-Aware Linear Attention (MALA), which modifies the computation of Linear Attention to fully incorporate the Query's magnitude. This adjustment allows MALA to generate an attention score distribution that closely resembles Softmax Attention while exhibiting a more well-balanced structure. We evaluate the effectiveness of MALA on multiple tasks, including image classification, object detection, instance segmentation, semantic segmentation, natural language processing, speech recognition, and image generation. Our MALA achieves strong results on all of these tasks. Code will be available at https://github.com/qhfan/MALA

Figures

Figures reproduced from arXiv: 2507.00698 by the authors.

Figure 1
Figure 1. Comparison between Softmax Attention and Linear At [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of attention score distributions across different mechanisms. As the magnitude of the Query ( [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Attention scores of different models. When [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization of attention scores on DeiT-T setting. Soft [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Comparison of general backbones’ inference speed on [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of general backbones’ inference speed on [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The distribution of attention scores from DeiT-T. Feature [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Norm$\times$Direction: Restoring the Missing Query Norm in Vision Linear Attention

    cs.LG 2025-06 conditional novelty 6.0 of 10

    NaLaFormer restores query-norm sensitivity in linear attention with a norm-aware power feature map and a cosine direction similarity that keeps attention scores non-negative.

Reference graph

Works this paper leans on

58 extracted references · 52 canonical work pages · cited by 1 Pith paper

  1. [1]

    Dico: Revitalizing convnets for scalable and efficient diffusion modeling.arXiv preprint arXiv:2505.11196, 2025

    Yuang Ai, Qihang Fan, Xuefeng Hu, Zhenheng Yang, Ran He, and Huaibo Huang. Dico: Revitalizing convnets for scalable and efficient diffusion modeling.arXiv preprint arXiv:2505.11196, 2025. 8

    arXiv 2025

  2. [2]

    Breaking complexity barriers: High-resolution im- age restoration with rank enhanced linear attention.arXiv preprint arXiv:2505.16157, 2025

    Yuang Ai, Huaibo Huang, Tao Wu, Qihang Fan, and Ran He. Breaking complexity barriers: High-resolution im- age restoration with rank enhanced linear attention.arXiv preprint arXiv:2505.16157, 2025. 8

    arXiv 2025

  3. [3]

    Hydra attention: Efficient at- tention with many heads, 2022

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient at- tention with many heads, 2022. 8

    work page 2022

  4. [4]

    Efficientvit: Lightweight multi-scale attention for high- resolution dense prediction

    Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high- resolution dense prediction. InICCV, pages 17302–17313,

    work page

  5. [5]

    Cascade r-cnn: Delving into high quality object detection

    Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. InCVPR, 2018. 6

    work page 2018

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, et al. Imagenet: A large-scale hierarchical image database. InCVPR, 2009. 5

    work page 2009

  7. [7]

    Davit: Dual attention vision transformers

    Mingyu Ding, Bin Xiao, Noel Codella, et al. Davit: Dual attention vision transformers. InECCV, 2022. 2

    work page 2022

  8. [8]

    Cswin transformer: A general vision transformer backbone with cross-shaped windows

    Xiaoyi Dong, Jianmin Bao, Dongdong Chen, et al. Cswin transformer: A general vision transformer backbone with cross-shaped windows. InCVPR, 2022. 1, 2, 5, 6, 7

    work page 2022

Show all 58 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021. 1, 2

    2021

  2. [10]

    Re- thinking local perception in lightweight vision transformer

    Qihang Fan, Huaibo Huang, Jiyang Guan, and Ran He. Re- thinking local perception in lightweight vision transformer. arXiv preprint arXiv:2303.17803, 2023. 2

    2023 arXiv

  3. [11]

    Lightweight vision transformer with bidirectional interac- tion

    Qihang Fan, Huaibo Huang, Xiaoqiang Zhou, and Ran He. Lightweight vision transformer with bidirectional interac- tion. InNeurIPS, 2023. 2, 5, 7

    2023

  4. [12]

    Semantic equitable clustering: A simple and effec- tive strategy for clustering vision tokens.arXiv preprint arXiv:2405.13337, 2024

    Qihang Fan, Huaibo Huang, Mingrui Chen, and Ran He. Semantic equitable clustering: A simple and effec- tive strategy for clustering vision tokens.arXiv preprint arXiv:2405.13337, 2024. 5

    2024 arXiv

  5. [13]

    Rmt: Retentive networks meet vision trans- formers

    Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, and Ran He. Rmt: Retentive networks meet vision trans- formers. InCVPR, 2024. 1, 5, 6, 7

    2024

  6. [14]

    Breaking the low- rank dilemma of linear attention

    Qihang Fan, Huaibo Huang, and Ran He. Breaking the low- rank dilemma of linear attention. InCVPR, 2025. 5

    2025

  7. [15]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023. 7

    2023 arXiv

  8. [16]

    Sucheng ren, xingyi yang, songhua liu, xinchao wang

    SG-Former: Self guided Transformer with Evolving To- ken Reallocation. Sucheng ren, xingyi yang, songhua liu, xinchao wang. InICCV, 2023. 5, 7

    2023

  9. [17]

    Conformer: Convolution-augmented transformer for speech recognition

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Par- mar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zheng- dong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmented transformer for speech recognition. InINTERSPEECH, 2020. 7

    2020

  10. [18]

    Cmt: Convolutional neural networks meet vision transformers

    Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolutional neural networks meet vision transformers. InCVPR, 2022. 1, 2, 6, 7

    2022

  11. [19]

    Visual attention network.arXiv preprint arXiv:2202.09741, 2022

    Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network.arXiv preprint arXiv:2202.09741, 2022. 7

    2022 arXiv

  12. [20]

    Flatten transformer: Vision transformer using fo- cused linear attention

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using fo- cused linear attention. InICCV, 2023. 1, 2, 3, 5, 6, 7, 8

    2023

  13. [21]

    Bridging the divide: Reconsidering softmax and linear attention

    Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song, and Gao Huang. Bridging the divide: Reconsidering softmax and linear attention. In NeurIPS, 2024. 2, 3, 5, 6, 8

    2024

  14. [22]

    Demystify mamba in vision: A linear attention perspective

    Dongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yi- fan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. InNeurIPS, 2024. 3, 5, 6, 7

    2024

  15. [23]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In CVPR, 2023. 2, 5, 6

    2023

  16. [24]

    Global context vision transformers

    Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. InICML, 2023. 5, 6

    2023

  17. [25]

    Girshick

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross B. Girshick. Mask r-cnn. InICCV, 2017. 6

    2017

  18. [26]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, and Zhuang Liu. Deep networks with stochastic depth. InECCV, 2016. 5

    2016

  19. [27]

    Vision transformer with super token sampling

    Huaibo Huang, Xiaoqiang Zhou, Jie Cao, Ran He, and Tie- niu Tan. Vision transformer with super token sampling. In CVPR, 2023. 5, 7

    2023

  20. [28]

    Learning correlation structures for vision trans- formers

    Manjin Kim, Paul Hongsuck Seo, Cordelia Schmid, and Minsu Cho. Learning correlation structures for vision trans- formers. InCVPR, 2024. 2

    2024

  21. [29]

    Panoptic feature pyramid networks

    Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Doll´ar. Panoptic feature pyramid networks. InCVPR, 2019. 6

    2019

  22. [30]

    Mpvit: Multi-path vision transformer for dense pre- diction

    Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense pre- diction. InCVPR, 2022. 7

    2022

  23. [31]

    Uniformer: Unified transformer for efficient spatiotemporal representation learning, 2022

    Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning, 2022. 2, 6

    2022

  24. [32]

    Siyuan Li, Zedong Wang, Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, and Stan Z. Li. Moganet: Multi-order gated aggregation network. InICLR,

  25. [33]

    Girshick, and Kaim- ing He andPiotr Doll´ar

    Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, and Kaim- ing He andPiotr Doll´ar. Focal loss for dense object detection. InICCV, 2017. 6

    2017

  26. [34]

    Scale-aware modulation meet transformer

    Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lian- wen Jin. Scale-aware modulation meet transformer. InICCV,

  27. [35]

    Dynamic group transformer: A general vision transformer backbone with dynamic group attention

    Kai Liu, Tianyi Wu, Cong Liu, and Guodong Guo. Dynamic group transformer: A general vision transformer backbone with dynamic group attention. InIJCAI, 2022. 2

    2022

  28. [36]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. InNeurIPS, 2024. 5, 6, 7

    2024

  29. [37]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 1, 2, 6, 8

    2021

  30. [38]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, et al. A convnet for the 2020s. InCVPR, 2022. 5, 6

    2022

  31. [39]

    Softmax-free linear transformers

    Jiachen Lu, Li Zhang, Junge Zhang, Xiatian Zhu, Jianfeng Feng, and Tao Xiang. Softmax-free linear transformers. IJCV, 2024. 3, 5

    2024

  32. [40]

    Softmax-free linear transformers

    Jiachen Lu, Li Zhang, Junge Zhang, Xiatian Zhu, Jianfeng Feng, and Tao Xiang. Softmax-free linear transformers. IJCV, 2024. 7

    2024

  33. [41]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, 2023. 8

    2023

  34. [42]

    The devil in linear transformer

    Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Ling- peng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. InEMNLP, 2022. 2, 3

    2022

  35. [43]

    Hornet: Efficient high-order spatial interactions with recursive gated convolutions

    Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Lam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. In NeurIPS, 2022. 6

    2022

  36. [44]

    Efficient attention: Attention with linear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. InWACV, 2021. 3, 8

    2021

  37. [45]

    Multi-scale vmamba: Hierarchy in hierarchy visual state space model

    Yuheng Shi, Minjing Dong, and Chang Xu. Multi-scale vmamba: Hierarchy in hierarchy visual state space model. InNeurIPS, 2024. 5, 7

    2024

  38. [46]

    Inception transformer

    Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xin- chao Wang, and Shuicheng Y AN. Inception transformer. In NeurIPS, 2022. 2, 6

    2022

  39. [47]

    Reten- tive network: A successor to Transformer for large language models.ArXiv, abs/2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Reten- tive network: A successor to Transformer for large language models.ArXiv, abs/2307.08621, 2023. 7

    2023 arXiv

  40. [48]

    Dic: Rethinking conv3x3 de- signs in diffusion models

    Yuchuan Tian, Jing Han, Chengcheng Wang, Yuchen Liang, Chao Xu, and Hanting Chen. Dic: Rethinking conv3x3 de- signs in diffusion models. InCVPR, 2025. 8

    2025

  41. [49]

    Train- ing data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, et al. Train- ing data-efficient image transformers & distillation through attention. InICML, 2021. 5, 8

    2021

  42. [50]

    Atten- tion is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. Atten- tion is all you need. InNeurIPS, 2017. 1, 2, 7

    2017

  43. [51]

    Pyra- mid vision transformer: A versatile backbone for dense pre- diction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyra- mid vision transformer: A versatile backbone for dense pre- diction without convolutions. InICCV, 2021. 1, 2, 6

    2021

  44. [52]

    Pvtv2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):1–10, 2022. 1, 2, 6, 7

    2022

  45. [53]

    Internimage: Exploring large-scale vi- sion foundation models with deformable convolutions

    Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vi- sion foundation models with deformable convolutions. In CVPR, 2023. 5, 6, 7

    2023

  46. [54]

    Unified perceptual parsing for scene understand- ing

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understand- ing. InECCV, 2018. 6

    2018

  47. [55]

    Gated linear attention transformers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. InICML, 2024. 7

    2024

  48. [56]

    Castling-vit: Compressing self-attention via switching to- wards linear-angular attention during vision transformer in- ference

    Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, and Yingyan Lin. Castling-vit: Compressing self-attention via switching to- wards linear-angular attention during vision transformer in- ference. InCVPR, 2023. 8

    2023

  49. [57]

    Biformer: Vision transformer with bi-level routing attention

    Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson Lau. Biformer: Vision transformer with bi-level routing attention. InCVPR, 2023. 2, 5, 6

    2023

  50. [58]

    Dig: Scal- able and efficient diffusion models with gated linear atten- tion

    Lianghui Zhu, Zilong Huang, Bencheng Liao, Jun Hao Liew, Hanshu Yan, Jiashi Feng, and Xinggang Wang. Dig: Scal- able and efficient diffusion models with gated linear atten- tion. InCVPR, 2025. 8

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.