PolaFormer: Polarity-aware Linear Attention for Vision Transformers

Dongmei Jiang; Weikang Meng; Xin Li; Yadan Luo; Zheng Zhang

REVIEW 3 major objections 6 minor 1 cited by

PolaFormer: Polarity-aware Linear Attention for Vision Transformers

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PolaFormer shows that linear attention can recover the information lost when softmax is replaced: explicitly modeling same-signed and opposite-signed query-key interactions and rescaling each channel with a learnable power function…

desk verdict Solid empirical linear-attention paper with a real mechanism, but the stated entropy-reduction theorem is false as written and needs to be fixed or demoted. read the letter →

arxiv 2501.15061 v2 pith:5R4YVDEO submitted 2025-01-25 cs.CV cs.AI

Weikang Meng , Yadan Luo , Xin Li , Dongmei Jiang , Zheng Zhang This is my paper · ORCID

classification cs.CVcs.AI

keywords polarity-awareattentionlinearvisiontransformersentropyreductionlearnablepowerfunctionkernelizedLongRangeArenaImageNetclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that linear attention falls short of softmax attention for two correctable reasons: it discards all interactions involving negative query or key coordinates, and it produces flatter, higher-entropy attention distributions. PolaFormer answers both by decomposing queries and keys into positive and negative parts, computing same-signed and opposite-signed similarities in separate streams with learnable mixing weights, and rescaling each channel with a learnable power function. The paper claims a theorem that any element-wise function with positive first and second derivatives strictly reduces the entropy of the attention distribution, which justifies the power-function sharpening. On ImageNet-1K, COCO, ADE20K, and the Long Range Arena, PolaFormer improves over linear-attention baselines by up to 4.6% while keeping complexity linear in sequence length.

What carries the argument

The machinery is a polarity-aware attention that runs two parallel linear-attention streams: one for same-signed pairs $(q^+, k^+)$ and $(q^-, k^-)$, and one for opposite-signed pairs $(q^+, k^-)$ and $(q^-, k^+)$, with value vectors split along the channel dimension and the two output halves gated by learnable sign-aware coefficient matrices $G_s$ and $G_o$. The second component is the positive-sequence entropy (PSE) measure, which defines the entropy of a non-negative sequence and is used to prove the theorem; the chosen entropy-reducing function is a per-channel power function $x^p$ with learnable exponent $p = 1 + \alpha\sigma(w)$, sharpening strong similarities relative to weak ones. A depthwise convolution is added to counteract the low-rank tendency of the softmax kernel, which the paper argues causes degenerate solutions when the value space is compact.

What would settle it

Search numerically over pairs of positive sequences $(a, b)$ and increasing convex functions $g$ with $g'(x)>0$ and $g''(x)>0$—for example $g(x) = x^2 + 1$ applied to $(1, 0.1)$—and check whether $\mathrm{PSE}(g(a), g(b))$ is always strictly less than $\mathrm{PSE}(a, b)$; a single counterexample pair where the transformed entropy is not lower would falsify Theorem 1 and the paper's stated theoretical basis for the power function.

Watch

Extended reading notes

Core claim

The central claim is that the information loss in linear attention is not inevitable: writing $q = q^+ - q^-$ and $k = k^+ - k^-$ shows that the full dot product consists of same-signed minus opposite-signed terms, and a learnable sign-aware mixing of the two streams—rather than a direct subtraction—can preserve negative-value interactions while keeping attention weights non-negative and training stable. To restore spikiness, the paper introduces a channel-wise learnable power function $g(x_i) = x_i^{p_i}$ with $p_i = 1 + \alpha\sigma(w_i)$ applied to the ReLU-transformed positive and negative parts, and proves that element-wise functions with $g'>0$ and $g''>0$ reduce the positive-sequence entropy of the resulting attention distribution. Inserted into DeiT, Swin, PVT, and PVTv2 backbones, the module reports consistent gains: up to +3.7% top-1 accuracy on ImageNet-1K, up to +4.6% on COCO detection metrics, and up to +2.6% mIoU on ADE20K, while its LRA variants average up to 60.72% with substantially lower peak memory than softmax attention.

Load-bearing premise

The load-bearing premise is that any increasing strictly convex element-wise function maps every positive ratio $a/b$ to a strictly larger ratio $f(a)/f(b)$, which is what the entropy-reduction proof needs; if the ratio-widening property fails for some functions in the class, the theorem's guarantee collapses.

Editorial extensions

If this is right

Linear attention can encode negative-value interactions without sacrificing the O(N) per-head complexity, because both the same-signed and opposite-signed streams use the same kernelized associative trick.
A learnable per-channel power exponent gives the model a differentiable, data-dependent way to sharpen attention maps, separating strong from weak query-key responses.
Dropping the module into existing backbones (DeiT, Swin, PVT, PVTv2) improves ImageNet-1K top-1 accuracy by +0.3% to +3.7% at comparable FLOPs.
On dense prediction, fine-tuning PolaFormer backbones improves COCO detection by up to +4.6% AP and ADE20K segmentation by up to +2.6% mIoU, consistent with the claim that fine-grained attention benefits from lower entropy.
On the Long Range Arena, the best PolaFormer variant averages 60.72% with roughly 0.22x the peak memory of softmax attention, indicating the benefits carry over to long-sequence tasks outside vision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

This suggests the sign decomposition could transfer to linear attention in language and audio modeling, where negative query-key correlations may carry different semantics than in vision.
If the entropy-reduction theorem turns out to hold only for power functions rather than the full claimed class, the learnable exponent still stands on its own as a sharpening layer for any kernel attention; the empirical gains would not require the general theorem.
The learned exponents p per channel offer a diagnostic: channels that converge near p=1 are already sufficiently sharp, while channels with large p are where linear attention over-smooths most, which could guide architectural capacity allocation.
The use of a convolution specifically to raise attention rank suggests rank deficiency, not just spikiness, is a bottleneck for linear attention, and the same rank-repair idea could improve other sub-quadratic attention designs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: PolaFormer is worth reading for the mechanism, not for the theory. The polarity split plus per-channel learnable power function gives consistent gains across ImageNet, COCO, ADE20K, and LRA, and the ablations are honest enough that I believe the effect is real. The entropy-reduction theorem, on the other hand, is false as stated.

What's new and good: decomposing q and k into positive and negative parts and treating same-signed and opposite-signed interactions separately is a clean idea that prior linear-attention work didn't do. The learnable exponent per channel is a natural extension of FLatten's fixed norm. The empirical work is broad: three vision tasks plus LRA, multiple backbones, and the gains over FLatten and the base models are consistent, in the 0.3–4.6% range. The efficiency numbers (Table 6) also look reasonable. Code is public. I'd trust the experiments.

Where it gets soft: Theorem 1. The proof in Appendix A.1 defines f(⟨x,y⟩)=⟨g(x),g(y)⟩, but no such univariate f exists: two different y's with the same inner product against x can give different transformed inner products. Lemma 1's Jensen step is reversed—for convex g you get E[g(q)g(k)] ≥ g(E q)g(E k), not ≤. Lemma 2 needs f(0)=0 to make d>c, and even the paper's own power function fails it: take x=(1,1,1), y1=(10,10,10), y2=(15,1,1), g(t)=t². The PSE goes from about 0.654 to 0.683, contradicting Eq. (8). So the theorem is not just missing rigor; it's false. The method may still work—and apparently does—but the theoretical motivation needs to be rewritten as an observation or conjecture, not a proven guarantee.

Also minor: Section 4.1 says the split introduces 'no additional learnable parameters,' then immediately introduces learnable Gs and Go. And Gs, Go are sized R^{N×d/2}, meaning their parameter count grows with sequence length; that deserves a clarification, even if it's a per-token scale.

Who this is for: anyone working on efficient attention who wants a working engineering recipe. It deserves a serious referee, but with a hard ask: fix or demote the theory, and clarify the parameter dependence. I'd cite it for the mechanism and the empirical comparison, not for the proof.

Referee Report

3 major / 6 minor

Summary. The paper proposes PolaFormer, a linear-attention mechanism that separately models same-signed and opposite-signed query-key interactions through a polarity-aware decomposition, mixes these streams with learnable sign-aware coefficient matrices, and applies a channel-wise learnable power function to restore the 'spiky' low-entropy behavior of softmax attention. The authors provide a theoretical claim (Theorem 1) that element-wise functions with positive first and second derivatives reduce the entropy of the attention distribution, and they support this with experiments on ImageNet-1K, COCO, ADE20K, and Long Range Arena, reporting gains up to 4.6% while preserving linear complexity in sequence length.

Significance. If the empirical results are reproducible, PolaFormer is a potentially useful linear-attention variant with broad, consistent improvements across classification, detection, segmentation, and LRA, and the released code is a concrete strength. However, the paper's headline theoretical guarantee, Theorem 1, is false as stated, and the proof in Appendix A.1 contains several invalid steps. Since the learnable power function is explicitly motivated by this theorem, the theoretical foundation for the 'spikiness' component is not established. The significance of the work therefore currently rests on the empirical evaluation, which is extensive but does not by itself support the theoretical claims made in the abstract and Section 4.2.

major comments (3)

[§4.2, Appendix A.1, Eq. (8)] Theorem 1 is false as stated, and the proof is invalid. For x=(1,1,1), y1=(10,10,10), y2=(15,1,1), and g(t)=t^2 (which satisfies g'>0 and g''>0), the original scores are ⟨x,y1⟩=30 and ⟨x,y2⟩=17, giving PSE≈0.654 nats; the transformed scores are ⟨g(x),g(y1)⟩=300 and ⟨g(x),g(y2)⟩=227, giving PSE≈0.683 nats, which violates Eq. (8) for the paper's own power function. The proof breaks down because (i) Lemma 1 defines f(⟨x,y⟩):=⟨g(x),g(y)⟩, but no such univariate function exists for d>1: with x=(1,1), y=(1,0), and y'=(0.5,0.5), both have ⟨x,y⟩=1 but ⟨g(x),g(y)⟩=1 and ⟨g(x),g(y')⟩=0.5 for g(t)=t²; (ii) the Jensen step in Eq. (15) has the inequality reversed for convex g (since E[g(X)]≥g(E[X])), and the conclusion that f is 'concave ... having a positive second derivative' is self-contradictory; and (iii) Lemma 2's key claim that d:=f(a)/f(b)>c:=a/b is not implied by f'>0 and f''>0 when f(0)>0, e.g., f(t)=t²+1000, a=10, b=9 gives d≈1.018<c≈1.111. The main-text theorem also omits the 'dimensions are independently distributed' condition that is inserted in the appendix statement, and that probabilistic assumption does not justify a deterministic inequality on fixed vectors.
[§4.2, Eq. (9)] Because the entropy-reduction claim is false for the very power function adopted in Eq. (9), the statement that Theorem 1 'provides insights into why commonly used feature maps such as ReLU or ELU+1 fail to reduce entropy' and the design of learnable exponents p are not supported by the stated theory. The counterexample in the previous comment uses exactly g(t)=t² with p=2, showing that the power function can increase PSE. The empirical success of the power function may still be valid, but the theoretical justification advanced in the abstract and in Section 4.2 for the 'loss of attention spikeness' motivation is not established. This is load-bearing because the choice of g(x;p) rests entirely on Theorem 1.
[§1 and §4.1, Eq. (6)] The text says the polarity-aware mixing handles 'both types of interactions without introducing additional learnable parameters' (Introduction and Section 4.1), but immediately defines G_s ∈ R^{N×d/2} and G_o ∈ R^{N×d/2} as 'two learnable polarity-aware coefficients matrices applied with element-wise multiplication'. This is an internal contradiction that must be resolved. Additionally, the R^{N×d/2} notation suggests parameters that depend on the sequence length N, which would be atypical and inconsistent with the complexity analysis in Eq. (10); if these are intended as per-channel scalars, the notation should be corrected to R^{d/2} or an equivalent broadcast shape.

minor comments (6)

[Appendix A.1, Lemma 2] In the proof of Lemma 2, the line 'H1 = h(c) > H2 = h(c)' appears to be a typo for 'H2 = h(d)' with d>c; as written, the inequality is between identical arguments. Also, the claim that the argument 'extendable to N > 2' is asserted without proof.
[§5.4, Table 3] The checkmark pattern in Table 3 is ambiguous; the text refers to 'row 1 and row 4' for DCN versus DWC, but the columns are not clearly aligned with the headers. Please reformat with explicit row labels and unambiguous column markers.
[§5.1, third paragraph] The sentence 'For instance, in Table 1, our DeiT-T-PolaFormer surpasses other DeiT variants from 0.5% to 6.3%' is followed by a discussion of PVT-T results; clarify which model is being compared and to which baselines.
[Eq. (10)] The phrase 'the complexity complexity of PolaFormer' should be 'the computational complexity of PolaFormer'.
[Appendix A.2] The phrase 'The training interation is set to 40000' should be 'The training iteration is set to 40000'.
[§3.2, Eq. (2)] The kernel notation SM(q,k)=exp(q_i k_j^⊤) mixes vector and scalar notation; please use ⟨q,k⟩ or q^⊤k consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PolaFormer's empirical gains come from external benchmarks with learned parameters, and its theoretical theorem is a mathematical claim, not a fitted or self-referential prediction.

full rationale

PolaFormer's central empirical claims are validated on external benchmarks (ImageNet-1K, COCO, ADE20K, LRA) against independent baselines, and all learnable quantities (the power exponents p and the polarity coefficients Gs, Go) are trained and then evaluated on held-out data. No fitted parameter is renamed as a prediction, and no experimental result is asserted to follow by construction from the training objective. The entropy-reduction Theorem 1 is presented as a mathematical statement with a proof in Appendix A.1; its assumptions (g'>0 and g''>0) do not include the conclusion, and the theorem is not derived from any fitted value or from the paper's own experimental outputs. The subsequent choice of a learnable power function is explicitly described as a simplicity-oriented design decision made after the theorem, not as the theorem's input. The reference list contains no load-bearing self-citation by the present authors, and no uniqueness theorem or prior work by the same group is invoked to force the model design. The Appendix's Lemma 1 and Lemma 2 contain mathematical errors, and the claim that 'it is easy to prove that d > c > 1' is false in general, but an invalid or unproven mathematical argument is a correctness risk, not circularity, because the theorem is not equivalent to its own inputs. Accordingly, no step in the claimed derivation chain reduces to its own assumptions or to a self-citation chain, and the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central method depends on learnable parameters p, Gs, Go, and alpha, and on several assumptions (distribution independence, low-rank SM, convex function ratio monotonicity) that are either standard domain assumptions or unproven. No new physical entities are introduced.

free parameters (3)

Learnable power exponent p = learned; p = 1 + alpha sigmoid(w), alpha in {3,5,7}
Channel-wise exponent used to rescale attention responses; fitted to task data (Eq. 9).
Sign-aware coefficient matrices Gs and Go = learned
Element-wise scaling for same-signed and opposite-signed streams; shape N x d/2 per layer, contradicting the text's claim of no extra parameters.
Scaling factor alpha = 3, 5, 7 (chosen per task)
Hyperparameter controlling the upper bound of the power exponent; sensitivity within 2% reported.

assumptions (5)

standard math Query and key vectors can be decomposed into positive and negative parts q = q+ - q-, k = k+ - k-.
True by definition; used in Eq. (4).
domain assumption The softmax kernel SM(q,k)=exp(qk^T) can be approximated as a difference of products of feature maps of positive/negative parts (Eq. 5).
This approximation is heuristic; the paper abandons the subtractive form due to potential negative weights.
ad hoc to paper Dimensions of q and k are independently distributed (Lemma 1), enabling the Jensen inequality step.
This distributional assumption is introduced only in the proof and is not validated on image data.
ad hoc to paper Any increasing convex function f maps each ratio a/b to a larger ratio f(a)/f(b) (Lemma 2).
False in general unless f(0)=0; the paper's g is x^p (g(0)=0), but the lemma is stated for arbitrary f with f''>0, so the proof is invalid.
domain assumption Softmax attention maps are inherently low-rank, causing degenerate value vectors (citing Verma 2021).
Used to justify adding convolutions to increase rank.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PolaFormer: Polarity-aware Linear Attention for Vision Transformers." pith.science (2026). https://pith.science/paper/5R4YVDEO

@misc{pith2026250115061,
  author       = {Pith},
  title        = {Pith review of: PolaFormer: Polarity-aware Linear Attention for Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5R4YVDEO}},
  note         = {Machine review of arXiv:2501.15061}
}

read the original abstract

Linear attention has emerged as a promising alternative to softmax-based attention, leveraging kernelized feature maps to reduce complexity from quadratic to linear in sequence length. However, the non-negative constraint on feature maps and the relaxed exponential function used in approximation lead to significant information loss compared to the original query-key dot products, resulting in less discriminative attention maps with higher entropy. To address the missing interactions driven by negative values in query-key pairs, we propose a polarity-aware linear attention mechanism that explicitly models both same-signed and opposite-signed query-key interactions, ensuring comprehensive coverage of relational information. Furthermore, to restore the spiky properties of attention maps, we provide a theoretical analysis proving the existence of a class of element-wise functions (with positive first and second derivatives) that can reduce entropy in the attention distribution. For simplicity, and recognizing the distinct contributions of each dimension, we employ a learnable power function for rescaling, allowing strong and weak attention signals to be effectively separated. Extensive experiments demonstrate that the proposed PolaFormer improves performance on various vision tasks, enhancing both expressiveness and efficiency by up to 4.6%.

Figures

Figures reproduced from arXiv: 2501.15061 by the authors.

**Figure 1.** Attention weight visualization. Unlike prior linear attention approaches ((Katharopoulos et al., 2020) the 3rd and (Han et al., 2023a) 4th plots) that generate uniform responses, the proposed PolaFormer captures a more accurate query-key interaction with lower entropy, closely resembling softmax while maintaining linear complexity. 1 INTRODUCTION Transformers have demonstrated remarkable success across a broad range… view at source ↗

**Figure 2.** The overall framework of PolaFormer. Our framework explicitly separates query-key pairs based on their polarity into two distinct streams, with scaled outputs controlled by the learnable sign-aware matrices Gs and Go for same-signed and opposite-signed components, respectively. A channel-wise power function with the learnable exponent p is employed to learn the rescaling process, capturing the sharpness characterist… view at source ↗

**Figure 3.** Visualizations of weights in Gs and Go . where [·, ·] denotes concatenation operation. Gs ∈ R N× d 2 and Go ∈ R N× d 2 are two learnable polarity-aware coefficients matrices applied with element-wise multiplication, which are expected to learn the complementary relationship between same-signed and oppositesigned values. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Efficiency analysis with Accuracy vs. FLOPs and Accuracy vs. Runtime curves on the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Visualization of different attention probability distributions [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

**Figure 6.** Figure 6: Visualization of the attention maps. A.5 VISUALIZATION OF ATTENTION PROBABILITY DISTRIBUTION’S ENTROPY We compute the entropy of standard self-attention, linear attention (Katharopoulos et al., 2020) and PolaFormer. Additionally, we visualize the distribution of one ro…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention
cs.CV 2025-10 conditional novelty 5.0 of 10

A pretrained FLUX diffusion model is adapted with local-window attention plus low-resolution global guidance, allowing 4K text-to-image generation from 1K-only training data at about 2x lower cost.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages · cited by 1 Pith paper

[1]

Hydra attention: Efficient attention with many heads

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In Computer Vision - ECCV 2022 Workshops - Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pp. 35–49. Springer,

work page 2022
[3]

We evaluate the PolaFormer based on the official implementation of Skyformer (Chen et al., 2021)

Long Range Arena. We evaluate the PolaFormer based on the official implementation of Skyformer (Chen et al., 2021). For Listops and Text Classification, we set batch size to 32 with1e − 4 learning rate. For Pathfinder, we set batch size to 128 with 5e − 4 learning rate. For Image Classification, we set batch size to 256 with 1e − 4 learning rate. For Retr...

work page 2021
[4]

The initialization strategies tested included Kaiming uniform, zero initialization, normal distribution (N (0, 1)), uniform distribution (U (0, 1)), and constant ones. The results are summarized in the table below: Init Comparison Kaiming Uniform Zeros Normal(0,1) Uniform(0,1) Ones Acc 73.06 72.17 74.30 74.40 70.70 17 Published as a conference paper at IC...

work page 2025
[8]

Particular object retrieval with integral max-pooling of CNN activations

12 Published as a conference paper at ICLR 2025 Giorgos Tolias, Ronan Sicre, and Herv´e J´egou. Particular object retrieval with integral max-pooling of CNN activations. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings,

work page 2025
[10]

Revisiting Linformer with a modified self-attention with linear complexity

URL https://arxiv.org/abs/2101.10277. Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 548–558. IEEE,

work page Pith review arXiv
[11]

vheat: Building vision models upon heat conduction

Zhaozhi Wang, Yue Liu, Yunfan Liu, Hongtian Yu, Yaowei Wang, Qixiang Ye, and Yunjie Tian. vheat: Building vision models upon heat conduction. CoRR, abs/2405.16555,

arXiv
[13]

Castling-vit: Compressing self-attention via switching towards linear- angular attention at vision transformer inference

Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, and Yingyan Celine Lin. Castling-vit: Compressing self-attention via switching towards linear- angular attention at vision transformer inference. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 ,...

work page 2023
[15]

The source code is available in the supplementary material for reference

14 Published as a conference paper at ICLR 2025 A A PPENDIX This Appendix provides proof and supporting lemma for Theorem 1, followed by implementation details for various vision tasks. The source code is available in the supplementary material for reference. • Proof. A.1: The mathematical proof and supporting lemmas of Theorem 1 • Implementation Details....

work page 2025

Show all 18 references

[16]

Consider the element-wise function g for pairs of (x, y) with dimension d: g(x) = (g(x1),

Proof. Consider the element-wise function g for pairs of (x, y) with dimension d: g(x) = (g(x1), . . . , g(xd)) g(y) = (g(y1), . . . , g(yd)) (13) Then, the inner-product between g(x) and g(y) is given by, ⟨g(x), g(y)⟩ = dX i=1 g(xi)g(yi). (14) Because of the independence acro...

2025
[2011]

Nikita Nangia and Samuel R. Bowman. Listops: A diagnostic dataset for latent tree learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics (NAACL), pp. 92–99. Association for Computational Linguistics,

2018
[2014]

Girshick, Kaiming He, and Piotr Doll ´ar

Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Doll ´ar. Focal loss for dense object detection. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 2999–3007. IEEE Computer Society,

2017
[2018]

Vmamba: Visual state space model

Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. CoRR, abs/2401.10166,

arXiv
[2019]

The basic learning rate is set to 1e−3 for 1024 batch size

to train all of our models for 400 epochs, including 20 epochs for linear warm-up. The basic learning rate is set to 1e−3 for 1024 batch size. Additionally, we use a weight decay of5e−2. The training framework is developed on the top of the official Swin Transformer implementa...

2025
[2020]

On the computational complexity of self-attention

11 Published as a conference paper at ICLR 2025 Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. On the computational complexity of self-attention. In 34th International Conference on Algorithmic Learning Theory, volume 201, pp. 597–619,

2025
[2021]

Flatten transformer: Vision transformer using focused linear attention

Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. InIEEE International Conference on Computer Vision (ICCV), pp. 5938–5948, 2023a. Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Shiji Son...

2017 arXiv
[2022]

Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition

Han Cai, Chuang Gan, and Song Han. Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition. arXiv preprint arXiv:2205.14756,

arXiv
[2023]

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on im- agenet. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montre...

2021 arXiv
[2024]

Cvt: Introducing convolutions to vision transformers

Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pp. 22–31. IEEE,

2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Hydra attention: Efficient attention with many heads

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. In Computer Vision - ECCV 2022 Workshops - Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pp. 35–49. Springer,

work page 2022

[2] [3]

We evaluate the PolaFormer based on the official implementation of Skyformer (Chen et al., 2021)

Long Range Arena. We evaluate the PolaFormer based on the official implementation of Skyformer (Chen et al., 2021). For Listops and Text Classification, we set batch size to 32 with1e − 4 learning rate. For Pathfinder, we set batch size to 128 with 5e − 4 learning rate. For Image Classification, we set batch size to 256 with 1e − 4 learning rate. For Retr...

work page 2021

[3] [4]

The initialization strategies tested included Kaiming uniform, zero initialization, normal distribution (N (0, 1)), uniform distribution (U (0, 1)), and constant ones. The results are summarized in the table below: Init Comparison Kaiming Uniform Zeros Normal(0,1) Uniform(0,1) Ones Acc 73.06 72.17 74.30 74.40 70.70 17 Published as a conference paper at IC...

work page 2025

[4] [8]

Particular object retrieval with integral max-pooling of CNN activations

12 Published as a conference paper at ICLR 2025 Giorgos Tolias, Ronan Sicre, and Herv´e J´egou. Particular object retrieval with integral max-pooling of CNN activations. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings,

work page 2025

[5] [10]

Revisiting Linformer with a modified self-attention with linear complexity

URL https://arxiv.org/abs/2101.10277. Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 548–558. IEEE,

work page Pith review arXiv

[6] [11]

vheat: Building vision models upon heat conduction

Zhaozhi Wang, Yue Liu, Yunfan Liu, Hongtian Yu, Yaowei Wang, Qixiang Ye, and Yunjie Tian. vheat: Building vision models upon heat conduction. CoRR, abs/2405.16555,

arXiv

[7] [13]

Castling-vit: Compressing self-attention via switching towards linear- angular attention at vision transformer inference

Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, and Yingyan Celine Lin. Castling-vit: Compressing self-attention via switching towards linear- angular attention at vision transformer inference. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 ,...

work page 2023

[8] [15]

The source code is available in the supplementary material for reference

14 Published as a conference paper at ICLR 2025 A A PPENDIX This Appendix provides proof and supporting lemma for Theorem 1, followed by implementation details for various vision tasks. The source code is available in the supplementary material for reference. • Proof. A.1: The mathematical proof and supporting lemmas of Theorem 1 • Implementation Details....

work page 2025