Pith. sign in

REVIEW 3 major objections 5 minor 30 references

Norm$\times$Direction: Restoring the Missing Query Norm in Vision Linear Attention

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NaLaFormer claims that linear attention's true defect is a cancelled query norm, and that restoring it with a norm-aware power map plus cosine direction similarity brings linear attention to state-of-the-art levels across vision and…

desk verdict The query-norm mechanism is not inert—the stress-test misread the exponent notation—but the appendix proof has a real false inequality, and the evaluation omits the closest norm-aware baseline. read the letter →

arxiv 2506.21137 v3 pith:FGYLKCNK submitted 2025-06-26 cs.LG

Weikang Meng , Yadan Luo , Liangyu Huo , Yingjian Li , Yaowei Wang , Xin Li , Zheng Zhang This is my paper · ORCID
classification cs.LG
keywords linearattentionquerynormentropynorm-directiondecompositioncosinesimilarityvisiontransformerspikinesscomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Linear attention exchanges softmax attention's quadratic cost for a kernel that should be just as expressive, but it consistently underperforms. This paper argues that a hidden culprit is the query norm: in softmax attention the norm of each query survives inside the exponential and acts as a dynamic temperature that sharpens the attention distribution, while linear attention's normalization cancels that norm, leaving the model 'query-norm unaware' and its attention maps too smooth. To fix this, NaLaFormer re-injects the query norm through a norm-aware power feature map and uses a cosine-based direction similarity that keeps scores non-negative without discarding negative inner-product contributions. The paper reports that this restores the norm-entropy correlation and sets new state-of-the-art linear attention results, including 82.9% on Swin-T, 84.3% on ImageNet-1K at 26M parameters, 61.2% average on the Long Range Arena, and up to a 92.3% peak-memory reduction in super-resolution. If correct, the query norm is the missing control that can make linear attention competitive with softmax across modalities.

What carries the argument

The load-bearing object is the Norm×Direction (ND) decomposition x = ||x||_p · dir(x), which splits each query and key into magnitude (norm) and orientation (direction). The paper uses it to expose the asymmetry: in standard linear attention the factor ||φ(q_t)|| cancels in the attention fraction, whereas in softmax attention ||q_t|| survives inside exp(·). The remedy is a query-norm-aware feature map φq(q)=dir(q)f(||q||), with f(||q||)=λ(τ+tanh(||q||)), which makes the sharpness of the attention distribution depend on the query norm, plus a cosine direction similarity φc that maps each scalar direction component to (|x|cos x, |x|sin x), whose inner-product sum equals Σ_i |dir(q)_i||dir(k)_i|cos(dir(q)_i − dir(k)_i). After a tanh-based scaling into [−π/4, π/4], that angular difference stays within a range where the cosine is non-negative, guaranteeing non-negativity while preserving sign information that ReLU and 1+ELU would delete.

What would settle it

Take the vector (0.8, 0.8): its ℓ2 norm is about 1.13, larger than its maximum component 0.8, while the appendix's proof relies on the opposite inequality. Re-running the entropy argument with the correct inequality would show whether the monotone norm-entropy claim survives or needs extra conditions. Empirically, one can sample a trained softmax attention head and measure the entropy of attention rows against query norms: if rows with larger query norms are not systematically spikier, the paper's motivating correlation is not present.

Watch

Extended reading notes

Core claim

The central claim is that mainstream linear attention breaks a correlation that softmax attention exploits: the negative relationship between the norm of a query and the entropy of its attention distribution. Writing attention in norm×direction form shows that in linear attention the query norm cancels between numerator and denominator, so only key norms influence the output, whereas in softmax attention the query norm survives inside the exponential and can act as a scaling temperature. NaLaFormer restores this property by mapping each query to dir(q)·f(||q||) with f(||q||)=λ(τ+tanh(||q||)), making the effective kernel exponent depend on the query's norm, and by mapping direction components through a cosine similarity that keeps all dimensional contributions non-negative while preserving the sign information that ReLU and 1+ELU throw away. The paper presents this as the reason its model regains spikiness, and as the explanation for its reported gains across image classification, detection, segmentation, super-resolution, diffusion, language modeling, and the Long Range Arena.

Load-bearing premise

The proof that a larger query norm monotonically reduces attention entropy relies on a norm inequality asserted in Appendix A.1; the stated direction appears to be wrong, since for the norms used there the norm of a vector is at least its largest component, not at most it. If that step does not hold, the paper's formal motivation for the norm-aware power feature map is not established by its own proof.

Editorial extensions

If this is right

  • Linear attention can match or beat softmax attention on vision benchmarks while keeping linear sequence complexity.
  • The query norm becomes a principled, adjustable sharpness control that carries over to language modeling and long-sequence tasks.
  • Non-negativity can be enforced without discarding sign information, which should help other kernel-based attention designs.
  • Token-intensive tasks such as super-resolution become tractable, with up to a 92.3% peak-memory reduction and a 36.4% latency cut on inputs with over 70K tokens.
  • The 61.2% average on the Long Range Arena suggests the mechanism helps in very long sequences where quadratic softmax attention is impractical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the norm-entropy mechanism is the real driver, other linear attention variants could gain from norm injection without redesigning their kernels, and the entropy-versus-norm diagnostic could become a standard design tool for kernel maps.
  • The paper's correlation evidence comes from sampled attention computations; a cleaner test would be to measure whether the restored negative entropy-norm correlation persists after full training, layer by layer, and whether test accuracy tracks it.
  • The cosine direction map suggests that angular and directional decompositions of the inner product could be explored in higher dimensions or complex-valued feature spaces, potentially connecting to rotation-based position encodings.
  • One implicit implication is that fixed power-function sharpening methods may be solving the wrong half of the problem: norm-awareness, not just steepness, is what matches softmax's dynamic temperature behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies the cancellation of the query norm in standard linear attention as a cause of its limited expressiveness, and proposes a norm-direction (ND) decomposition of queries and keys. The proposed NaLaFormer uses a query-norm-aware power feature map—where the exponent depends on the query norm—plus a cosine-based direction similarity to enforce non-negativity while preserving sign information. The method is evaluated across image classification, detection, segmentation, super-resolution, diffusion, language modeling, and LRA, reporting state-of-the-art results for linear attention and substantial efficiency gains (e.g., 92.3% peak-memory reduction in super-resolution). The paper also provides component ablations and the code.

Significance. If the empirical results hold, NaLaFormer is a simple and practical linear attention mechanism that narrows the gap to softmax attention while retaining linear complexity. The reported gains over strong baselines (84.3% ImageNet-1K at 26M params, 61.2% LRA average, consistent improvements in detection/segmentation) are supported by a broad benchmark suite and component ablations. However, the theoretical motivation in Appendix A.1 contains a false Lp-norm inequality that invalidates the proof of Theorem A.2, so the paper's claimed 'mathematical framework' is not sound as written. The empirical contribution remains promising, but the theory must be corrected or explicitly weakened to a heuristic.

major comments (3)
  1. [Appendix A.1, Theorem A.2 proof] The proof uses the inequality '||x||_p <= x_max for p>1', which is false: for a finite vector x, ||x||_p >= x_max for all p. This inequality is essential for bounding the PSE term and for taking the limit c->infinity. With the inequality reversed, the bound does not follow and the existence of c0 is not established. The theorem itself may be true (larger query norm sharpens softmax attention), but the proof as written is invalid. Please correct the proof or explicitly reclassify the claim as an empirical/plausibility statement.
  2. [Eq. (5) and Eq. (12)] The definition φq(q) = dir(q)f(||q||) is ambiguous: if read as scalar multiplication, the factor f(||q||) would cancel in the normalized attention output of Eq. (14), making the norm-awareness mechanism mathematically inert. Based on Eq. (12) and Fig. 2, the intended definition is an elementwise power of the direction vector, which does not cancel. This distinction is central to the paper's claim. Please use unambiguous notation (e.g., explicit braces around the exponent) in Eq. (5), the figure, and the text, and state explicitly that f(||q||) is applied elementwise as an exponent.
  3. [Appendix A.1, final paragraph] The argument that the proposed power feature map 'compensates' for the softmax query-norm/entropy property relies on 'Lemma 2 in PolaFormer' without stating the lemma or verifying that the proposed map satisfies its assumptions. The sentence 'the composite function of the element-wise feature map with first and second derivative is concave, thus the feature map we proposed... can compensate' is an assertion, not a derivation. Please state Lemma 2 explicitly and provide a concrete check for the specific feature map in Eq. (5)/(12), or clearly label this as an intuition.
minor comments (5)
  1. [Sec. 4.7 / Table 7] The text says NaLaFormer attains 'higher training throughput' but Table 7 shows THRavg = 827.7, which is lower than PolaFormer (915.6), Nyström (1007.7), and Linformer (918.8). Please clarify which comparison the claim refers to.
  2. [Table 14] The column labeled 'IMAGE(CV)' reports values around 40–44, which are inconsistent with ImageNet top-1 accuracies reported elsewhere in the paper. Please specify the task, dataset, and metric used for this ablation.
  3. [Sec. 3.2 / Eq. (6)] The tanh-based scaling of direction components into [-π/4, π/4] is described after Eq. (6) but is not reflected in the formula. Please reconcile the notation, e.g., by defining the input to φc as the scaled direction component.
  4. [Eq. (10)] The formula SM_ours(q,k) = Σ_i cos(ϕ(q)_i − ϕ(k)_i) does not match the actual computation in Eq. (7) and Eq. (12), which involves |dir|·cos and |dir|·sin terms. Please align the conceptual formula with the implemented feature maps.
  5. [Abstract / Table 1] The abstract claims 'up to a 7.5% accuracy gain' on ImageNet-1K; in Table 1 the largest margin is against MetaLA, whose FLOPs are listed as '-' and whose parameter count is 6M versus NaLaFormer-T's 15M. Please specify the comparison group precisely or soften the statement.

Circularity Check

1 steps flagged · score 4.0 of 10

Theoretical motivation imports Lemma 2 from the authors' own PolaFormer, but no experimental prediction reduces by construction; central benchmarks are external.

  1. self citation load bearing [Appendix A.1, final paragraph (Entropy Analysis)]
    "Considering the Lemma 2 in PolaFormer (Meng et al., 2025), the composite function of the element-wise feature map with first and second derivative is concave, thus the feature map we proposed, power function with the exponent greater than 1 as well as changing with query norm can compensate for the property in softmax attention where the query norm influences PSE."

    The paper's central theoretical claim—that the norm-aware power feature map φq(q)=dir(q)^{f(||q||)} restores the query-norm/entropy correlation—is not established by the manuscript's own derivations. The decisive step delegates the needed property to 'Lemma 2 in PolaFormer (Meng et al., 2025)', the authors' own prior work, without stating or proving the lemma. Since the authors of PolaFormer overlap with the present paper, the justification chain for the core mechanism passes through a self-citation rather than through the equations of this paper. The empirical SOTA numbers remain externally benchmarked and the in-paper ablations are independent, so the circularity is partial and confined to the theoretical motivation.

full rationale

Most of the derivation chain is self-contained: the ND decomposition in Eqs. (1) and (4) is an algebraic identity; Theorem A.3 derives the query-norm-unawareness bound in-text; and the headline results (ImageNet-1K, COCO, ADE20K, LRA, SR) are compared against external baselines with in-paper ablations (Tables 8 and 11) attributing gains component-wise. The one load-bearing self-citation is Appendix A.1's closing step, where the claim that a power function with norm-dependent exponent can compensate for softmax's query-norm/PSE property rests on 'Lemma 2 in PolaFormer (Meng et al., 2025)' without proof or independent verification in this paper. This makes the theoretical motivation partially dependent on the authors' own prior work, but it does not make any reported accuracy, mIoU, PSNR, or LRA number reduce to a fitted value by construction. Two non-circular weaknesses should be noted for correctness rather than circularity: the proof of Theorem A.2 relies on the false inequality ||x||_p ≤ x_max (the correct statement is ||x||_p ≥ x_max), and the notation in Eqs. (5)/(12) might be misread as a scalar multiplier f(||q||) that cancels in Eq. (14); it is actually an elementwise exponent on direction components, so the norm-awareness mechanism is not inert by construction. Score 4 reflects one load-bearing self-citation while the central empirical claims retain independent content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on one new feature-map construction and two tuned hyperparameters. There are no new physical entities. Several definitions and a lemma are borrowed from the authors' earlier PolaFormer work, which is acceptable but weakens the independent grounding of the theoretical motivation.

free parameters (2)
  • lambda (λ) = 3 (selected in Table 14)
    Controls the norm-aware exponent f(x)=λ(τ+tanh x) and the key power k^λ; tuned on retrieval and image validation sets.
  • tau (τ) = 0.5 (selected in Table 14)
    Offset in f(x)=λ(τ+tanh x); chosen to keep the exponent bounded and positive, with Table 14 showing sensitivity across tasks.
assumptions (4)
  • domain assumption Positive Sequence Entropy (PSE) is the right measure of attention spikiness
    Introduced in the authors' prior PolaFormer (Meng et al., 2025) and used without independent justification to connect entropy to attention quality.
  • ad hoc to paper Lemma 2 from PolaFormer on concavity of composite feature maps
    Invoked at the end of Appendix A.1 to assert that the proposed power feature map 'can compensate' for softmax's query-norm entropy control; the lemma is from the authors' own prior paper and is not reproved here.
  • standard math Norm Equivalence Theorem
    Used in Definition 2.1 to ignore p-norm distinctions; a standard finite-dimensional result.
  • domain assumption Lower attention entropy improves task accuracy
    The paper assumes that sharpening attention (reducing PSE) is beneficial for accuracy; this underpins the design goal but is not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Norm$\times$Direction: Restoring the Missing Query Norm in Vision Linear Attention." pith.science (2026). https://pith.science/paper/FGYLKCNK

@misc{pith2026250621137,
  author       = {Pith},
  title        = {Pith review of: Norm$\times$Direction: Restoring the Missing Query Norm in Vision Linear Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGYLKCNK}},
  note         = {Machine review of arXiv:2506.21137}
}
abstract

Linear attention mitigates the quadratic complexity of softmax attention but suffers from a critical loss of expressiveness. We identify two primary causes: (1) The normalization operation cancels the query norm, which breaks the correlation between a query's norm and the spikiness (entropy) of the attention distribution as in softmax attention. (2) Standard techniques for enforcing non-negativity cause destructive information loss by nullifying valid inner-product interactions. To address these challenges, we introduce NaLaFormer, a novel linear attention mechanism built upon a norm$\times$direction (ND) decomposition of the query and key vectors. We leverage each component to solve a distinct problem: The query norm is injected into our kernel to create a query-norm-aware map that restores the attention distribution's spikiness. The direction vectors are processed by a geometric, cosine-based similarity metric that guarantees non-negativity while preserving the rich, fine-grained information of the inner product. We validate NaLaFormer through a comprehensive multi-modal evaluation, where it sets new state-of-the-art benchmarks for linear attention. Our model achieves up to a 7.5% accuracy gain on ImageNet-1K and a 4.7% mIoU improvement on ADE20K over comparable baselines. It demonstrates profound efficiency, reducing peak memory by a transformative 92.3% in token-intensive super-resolution tasks (70K+ tokens). NaLaFormer's versatility is further confirmed as it surpasses strong baselines like Mamba on common-sense reasoning and sets a new state-of-the-art on the Long Range Arena (LRA) benchmark. Code is available at https://github.com/ZacharyMeng/NaLaFormer .

Figures

Figures reproduced from arXiv: 2506.21137 by the authors.

Figure 1
Figure 1. Entropy-norm correlation in softmax attention. We plot the relationship between feature entropy and vector norms in a Swin Transformer sampled on ImageNet. The top row shows q-norms (x-axis) exhibit a strong negative correlation with attention entropy (y-axis). The bottom row shows that k-norms have no consistent effect. This observation suggests that the entropy diminishing in linear attention may stem from insuffi… view at source ↗
Figure 2
Figure 2. The NaLaFormer architecture and its core mechanisms. (a) The NaLaFormer block incorporates a simplified GLA and custom feature maps ϕq and ϕk. (b) Our norm-aware method (right) restores the negative query norm-entropy correlation lost in standard linear attention (left). (c) The cosine direction mechanism enforces non-negativity by decomposing similarity into norm and direction components, preventing information los… view at source ↗
Figure 3
Figure 3. We visualize the query norm-entropy relationship under three approaches: (1) Only preserve non-negativity with 1 + ELU operator (Katharopoulos et al., 2020). (2) Keep both non-negativity and spikiness with ReLU operator and power function as in FLatten (Han et al., 2023). (3) Our q-norm-aware approach shows a clear correlation between entropy and query norms. ot = X N i=1 ϕ(qt)ϕ(ki) ⊤vi PN j=1 ϕ(qt)ϕ(kj )⊤ = ϕ(qt) P… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Comparisons of element-wise dot product contributions for different non-negative strategies. The plots show qiki value for (1) the raw inputs, (2) our novel cosine-based approach, (3) ReLU activation, and (4) 1 + ELU activation. Our approach ensures all dimensional con…
Figure 5
Figure 5. Figure 5: Visualizations illustrating NaLaFormer’s semantic segmentation results on the CityScapes dataset (left) and NaLaSR and ESRT’s super-resolution results on the Urban100 benchmark (right). model with the ImageNet-1K pre-trained weight using mIoU as the evaluation metric, …
Figure 6
Figure 6. Figure 6: Efficiency analysis with Acc vs. FLOPs curves on the ImageNet-1K. 1K 2K 4K 6K 8K 16K Sequence Length 0 4 8 12 16 20 24 28 T G S (T o k e n s p er G P U S e c o n d)(× 1 0 3 ) OOM 2.5x Softmax RetNet Mamba GLA DeltaNet Gate DeltaNet NaLa+DN NaLa+GDN [PITH_FULL_IMAGE:fi…
Figure 8
Figure 8. Figure 8: Visualizations of the NaLaDiT-XL/2 with 256 × 256. 4.8. Ablation Study We conduct extensive ablations to validate NaLaFormer’s norm-aware mechanism, structural robustness, and hyper￾parameters, with exhaustive details deferred to Appendix A.3. Crucially, to address con…
Figure 9
Figure 9. Figure 9: Comparison of the visualizations among different models. Set14 Urban100 Set5 BSD100 Origin HR BICUBIC ESRT NaLaSR Urban100 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Visualizations of the NaLaSR in four different benchmarks comparing with ESRT under ×4 scale. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 23 canonical work pages

  1. [1]

    and Timofte, R

    Agustsson, E. and Timofte, R. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1122–1131,

    work page 2017

  2. [2]

    For a linear attention with nonlinear feature map, such as Log-Normal Attention (Nahshan et al., 2024), FLatten (Han et al., 2023), Efficientvit (Cai et al.,

    Therefore, we have the following derivations: 16 Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attention If the feature map is a linear transformation,i.e.,Φ(x m) =ϕ(q)ϕ(k m)⊤ andΦ(cx) =cΦ(x), such asReLU(·), we have, S= NX m=1 Φ(xm) PSElinear(x) = log(S)− NX i=1 Φ(xi) S log(Φ(xi)) PSElinear(cx) = log(c) + log(S)− NX i=1 cΦ(x i) c·S (l...

    work page 2024

  3. [4]

    CMT: convolutional neural networks meet vision transformers

    Guo, J., Han, K., Wu, H., Tang, Y ., Chen, X., Wang, Y ., and Xu, C. CMT: convolutional neural networks meet vision transformers. InProc. IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 12165–12175, 2022a. 10 Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attention Guo, M., Lu, C., Hou, Q., Liu, Z., Cheng, M., a...

    arXiv

  4. [8]

    Retentive network: A successor to trans- former for large language models.CoRR, abs/2307.08621,

    Sun, Y ., Dong, L., Huang, S., Ma, S., Xia, Y ., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to trans- former for large language models.CoRR, abs/2307.08621,

    arXiv

  5. [10]

    Building Vision Models upon Heat Conduction

    Wang, Z., Liu, Y ., Liu, Y ., Yu, H., Wang, Y ., Ye, Q., and Tian, Y . vheat: Building vision models upon heat conduction. CoRR, abs/2405.16555,

    work page Pith review arXiv

  6. [11]

    Multi-scale representations by varying window attention for semantic segmentation

    Yan, H., Wu, M., and Zhang, C. Multi-scale representations by varying window attention for semantic segmentation. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024,

    work page 2024

  7. [12]

    Yang, S., Wang, B., Shen, Y ., Panda, R., and Kim, Y

    URL https://github.com/ fla-org/flash-linear-attention. Yang, S., Wang, B., Shen, Y ., Panda, R., and Kim, Y . Gated linear attention transformers with hardware-efficient train- ing. InProc. International Conference on Machine Learn- ing (ICML), 2024a. Yang, S., Wang, B., Zhang, Y ., Shen, Y ., and Kim, Y . Par- allelizing linear transformers with the del...

    arXiv

  8. [13]

    Cluster-wise spatio-temporal masking for efficient video-language pretraining.CoRR, abs/2603.22953,

    Zhuang, W., Huang, Y ., Meng, W., Li, X., Liu, M., Hong, X., Wang, Y ., and Zuo, W. Cluster-wise spatio-temporal masking for efficient video-language pretraining.CoRR, abs/2603.22953,

    work page arXiv

Show all 30 references
  1. [14]

    We investigate the probability distribution generated from one single query vector and a series of key vectors with PSE, analyzing how PSE(x) varying with query norm with softmax

    to connect the softmax self-attention with PSE(·). We investigate the probability distribution generated from one single query vector and a series of key vectors with PSE, analyzing how PSE(x) varying with query norm with softmax. We first give the definition of PSE as followi...

    2020

  2. [15]

    and PolaFormer (Meng et al., 2025), they all have c1ϕ(q)≤ϕ(cq)≤c 2ϕ(q), and for ReLU(·)feature map,c 1 =c

    2025

  3. [17]

    and PoLaFormer (Meng et al., 2025), they all have c1ϕ(q)≤ϕ(cq)≤c 2ϕ(q) (and for ReLU(·)feature map,c 1 =c 2), thus, we have: For clearity, we suppose the original positive sequence is normalized,i.e., PN m=1 Φ(xm) = PN m=1 ϕ(q)ϕ(km)⊤ = 1, then, under the assumptionc 1ϕ(q)≤ϕ(cq...

    2025

  4. [18]

    When mapping each d(x), we set f(x) = π 4 tanh(x) to make the cosine function only inhibits the directions with opposite signals

    to serve as the positional encoding. When mapping each d(x), we set f(x) = π 4 tanh(x) to make the cosine function only inhibits the directions with opposite signals. Object Detection and Segmentation.We further conducted comprehensive experiments on the object detection task ...

    2014

  5. [19]

    The model are trained under both 1× (12 epochs) and 3× (36 epochs)

    project. The model are trained under both 1× (12 epochs) and 3× (36 epochs). We use the AdamW optimizer with 0.0001 learning rate, 0.0001 weight decay and “step” policy. Semantic Segmentation.We conduct the semantic segmentation of ADE20K dataset (Zhou et al., 2019). This wide...

    2019

  6. [20]

    and SiT (Ma et al., 2024), we adopt the diffusion transformer S/2 architecture as the main experimental setting. We integrate NaLaFormer into both DiT and SiT by replacing the original attention module while keeping the remaining architectures and training configurations uncha...

    2024

  7. [22]

    28M 4.4G 81.2 Hydra Attn (Bolya et al., 2022)29M 4.5G 80.7 Efficient Attn (Shen et al., 2021)29M 4.5G 81.0 Linear Angular (You et al., 2023)29M 4.5G 79.4 Enhanced Attn (Cai et al., 2023)29M 4.5G 81.8 FLatten Attn (Han et al.,

    2022

  8. [23]

    29M 4.5G 82.1 Agent Attn (Han et al., 2024c) 29M 4.5G 82.6 InLine Attn (Han et al., 2024a) 30M 4.5G 82.4 PolaFormer (Meng et al., 2025)29M 4.5G82.6 NaLaFormer 29M 4.8G 82.9 Table 13.Ablation studies of vision models with NaLaFormer-XT. W.O. RoPE CPE Layerscales Swish Ours-XT A...

    2025

  9. [24]

    employs shifted window mechanisms to balance local feature extraction with global context modeling while maintaining linear complexity. These advancements have established Transformer-based architectures as foundamental frameworks for visual tasks, effectively bridging the met...

    2024

  10. [25]

    Subsequent variants adopt this Softmax-free paradigm with diverse kernel functions, including ReLU (Han et al., 2023; Cai et al., 2023), 1+ELU (Katharopoulos et al.,

    to O(N) . Subsequent variants adopt this Softmax-free paradigm with diverse kernel functions, including ReLU (Han et al., 2023; Cai et al., 2023), 1+ELU (Katharopoulos et al.,

    2023

  11. [26]

    Furthermore, to enhance position awareness, Cosformer (Qin et al.,

    and SiLU (Yang et al., 2024b; MiniMax et al., 2025). Furthermore, to enhance position awareness, Cosformer (Qin et al.,

    2025

  12. [27]

    Inline (Han et al., 2024a) provides an important insight by proving that the softmax function is injective in most cases, whereas linear attention is not

    combines SiLU kernels with a gate mechanism, achieving scalability up to 456B parameters (MiniMax et al., 2025). Inline (Han et al., 2024a) provides an important insight by proving that the softmax function is injective in most cases, whereas linear attention is not. By modify...

    2025

  13. [28]

    Existing kernel-based linear attention mechanisms generally suffer from performance degradation compared to standard softmax attention

    constructs a lightweight recurrent-form linear attention by defining the optimal linear approximation conditions of the softmax attention map, however, when applied to encoder architectures, such as ViT models or bidirectional attention, the model performance becomes sensitive...

    2024

  14. [29]

    LAT” denotes the inference latency and “MEM

    Linear 43.2 64.1 46.4 28.0 46.4 57.9 NaLaFormer-S Linear 47.2 68.0 50.7 29.0 51.3 63.3 Table 18.Full table of LRA: Throughput and Peak Memory of various models. A denotes the accuracy, T denotes the throughput of each model and M denotes the peak memory cost. Softmax Kernelize...

    2011

  15. [2017]

    and Gimelshein, N

    Milakov, M. and Gimelshein, N. Online normalizer calcula- tion for softmax.CoRR, abs/1805.02867,

    arXiv

  16. [2018]

    Minimax-01: Scal- ing foundation models with lightning attention.CoRR, abs/2501.08313,

    MiniMax, Li, A., Gong, B., and et al. Minimax-01: Scal- ing foundation models with lightning attention.CoRR, abs/2501.08313,

    arXiv

  17. [2020]

    All downstream tasks are conducted based onlm-evaluation-harness

    and WinoGrande (Sakaguchi et al., 2020). All downstream tasks are conducted based onlm-evaluation-harness. We test throughput of the baseline models on a single A6000 GPU. 19 Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attention A.3. Ablation Study Impact...

    2020

  18. [2021]

    Touvron, H., Lavril, T., and Gautier Izacard, e. a. Llama: Open and efficient foundation language models.CoRR, abs/2302.13971,

    arXiv

  19. [2022]

    LAT” denotes the inference latency and “MEM

    ×4 195ms 3.0G 188ms 7.0G 79ms 2.2G 195ms 69G NaLaSR ×4 159ms 2.3G 147ms 2.9G 72ms 2.1G 124ms 5.3G - SAVE ×4 18.5% 23.3% 21.8% 58.6% 8.9% 4.5% 36.4% 92.3% Table 20.Quantitative comparison and efficiency analysis on benchmark datasets for image super-resolution. The best results...

    2020

  20. [2023]

    Think you have solved question answering? try arc, the AI2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457,

    arXiv

  21. [2024]

    Rectifying magnitude neglect in linear attention.CoRR, abs/2507.00698, 2025a

    Fan, Q., Huang, H., Ai, Y ., and He, R. Rectifying magnitude neglect in linear attention.CoRR, abs/2507.00698, 2025a. Fan, Q., Huang, H., and He, R. Breaking the low-rank dilemma of linear attention. InProc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp...

    arXiv

  22. [2025]

    Mirrorla: Reflecting feature map for vision linear attention.CoRR, abs/2602.04346,

    Meng, W., Huo, L., Luo, Y ., Wang, Y ., Li, Y ., and Zhang, Z. Mirrorla: Reflecting feature map for vision linear attention.CoRR, abs/2602.04346,

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.