REVIEW 6 cited by
The Devil in Linear Transformer
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Linear transformers aim to reduce the quadratic space-time complexity of vanilla transformers. However, they usually suffer from degraded performances on various tasks and corpus. In this paper, we examine existing kernel-based linear transformers and identify two key issues that lead to such performance gaps: 1) unbounded gradients in the attention computation adversely impact the convergence of linear transformer models; 2) attention dilution which trivially distributes attention scores over long sequences while neglecting neighbouring structures. To address these issues, we first identify that the scaling of attention matrices is the devil in unbounded gradients, which turns out unnecessary in linear attention as we show theoretically and empirically. To this end, we propose a new linear attention that replaces the scaling operation with a normalization to stabilize gradients. For the issue of attention dilution, we leverage a diagonal attention to confine attention to only neighbouring tokens in early layers. Benefiting from the stable gradients and improved attention, our new linear transformer model, transNormer, demonstrates superior performance on text classification and language modeling tasks, as well as on the challenging Long-Range Arena benchmark, surpassing vanilla transformer and existing linear variants by a clear margin while being significantly more space-time efficient. The code is available at https://github.com/OpenNLPLab/Transnormer .
Forward citations
Cited by 6 Pith papers
-
DOSA: A Tree-Guided, Self-Regressive Framework for Long Document Structure Analysis
DOSA's tree-guided self-regressive chunking improves multi-page document hierarchy parsing by up to 4 F1 and 19 TEDS over prior methods on DocHieNet.
-
MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs
A metacognition-driven self-evolution framework with activation steering reduces LLM jailbreak success rates on implicit domain risks from 57.8% to 4.6%.
-
StegaFFD: Privacy-Preserving Face Forgery Detection via Fine-Grained Steganographic Domain Lifting
StegaFFD hides face images inside innocent cover photos and performs face-forgery detection directly on the stego image, preserving accuracy while hiding the face.
-
Elucidating the Design Space of Decay in Linear Attention
A controlled study of decay in linear attention finds median decay near 0.8 works best, vector decay generally beats scalar decay, and RoPE/TPE give little benefit for models with sub-unity decay.
-
Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries
Transolver++ scales a physics-state attention model to 2.5 million mesh points via local adaptive temperatures, Gumbel-Softmax slice assignment, and AllReduce communication of physics states.
-
Position: The Future of Bayesian Prediction Is Prior-Fitted
PFNs, which amortize Bayesian inference by training on datasets sampled from a prior, are likely to supersede MCMC and variational inference for most prediction tasks, the authors argue.
Discussion (0). Continue with ORCID to comment.