Scaling Attention to Very Long Sequences in Linear Time with Wavelet-Enhanced Random Spectral Attention (WERSA)

Vincenzo Dentamaro

REVIEW 3 major objections 4 minor 22 references

Scaling Attention to Very Long Sequences in Linear Time with Wavelet-Enhanced Random Spectral Attention (WERSA)

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Wavelet-enhanced random spectral attention (WERSA) approximates standard attention in linear time with bounded error and reports best accuracy on long-context benchmarks including 128k-token inputs.

desk verdict WERSA is a plausible linear-attention recipe with decent empirical coverage, but its central approximation theorem reuses Performer's softmax guarantee for ReLU features, so the theoretical core doesn't hold as written. read the letter →

arxiv 2507.08637 v1 pith:5NE2ASQF submitted 2025-07-11 cs.LG cs.AIcs.CL

Vincenzo Dentamaro This is my paper · ORCID

classification cs.LGcs.AIcs.CL

keywords linearattentionwavelettransformHaarrandomfeaturessoftmaxkernelapproximationlong-contexttransformersmulti-resolutionanalysiscontent-adaptivefiltering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces WERSA, an attention mechanism that replaces the quadratic token-pair matrix of standard attention with a three-stage pipeline: a Haar wavelet decomposition of queries and keys, input-dependent filtering of the wavelet coefficients, and a random-feature approximation of the softmax kernel. The paper's central claim is that this pipeline runs in $O(n)$ time while preserving the behavior of standard multi-head attention: the WERSA Long-Context Approximation Theorem puts the Frobenius error below $\varepsilon\|V\|_F + O(2^{-\alpha L})$ with high probability, and the learned filters are claimed to converge to the optimal filter as sequence length grows. If the claim holds, transformer models could process sequences of hundreds of thousands or millions of tokens on a single GPU, which is exactly the regime where quadratic attention fails. The paper reports that WERSA achieves the best accuracy among seven attention mechanisms on five benchmarks, including the only successful results on 128k-token ArXiv sequences where standard attention and FlashAttention-2 run out of memory.

What carries the argument

The mechanism is built from four pieces working in sequence. The Haar wavelet transform, a multi-resolution decomposition into pairwise averages and differences at successive scales, splits queries and keys into local detail coefficients and a global approximation. A gating network fed by the average query representation produces per-scale filter coefficients $F_i = \sigma(g(Q'_{\mathrm{avg},h}))_i \cdot \omega_i$, with trainable weights $\omega_i$ controlling each scale's importance; these filters are applied to the wavelet coefficients before inverse reconstruction. The filtered signals are then passed through the random feature map $\phi(x) = \mathrm{ReLU}(xR/\beta)$ with $R \in \mathbb{R}^{d \times m}$ and trainable bandwidth $\beta$, which linearizes attention via $\mathrm{softmax}(QK^T)V \approx \phi(Q)(\phi(K)^T V)/(\phi(Q)\phi(K)^T \mathbf{1} + \varepsilon)$. The theorem's error bound splits into the random-feature term $\varepsilon\|V\|_F$, controlled by $m$, and the wavelet truncation term $O(2^{-\alpha L})$, controlled by the number of decomposition levels.

What would settle it

Compute a Monte Carlo estimate of $\mathbb{E}[\phi(q)^T \phi(k)]$ for $\phi(x) = \mathrm{ReLU}(xR/\beta)$ with Gaussian $R$ and compare it with the softmax kernel $\exp(q^T k / \sqrt{d})$ over queries and keys drawn from a trained WERSA model; if the relative bias is not below $\varepsilon$ for $m = 1024$, the theorem's lower bound on $m$ cannot produce the promised $\varepsilon\|V\|_F$ error term.

Watch

Extended reading notes

Core claim

The central discovery is that wavelet multi-resolution analysis and random spectral features can be combined into a single linear-time attention mechanism whose output is provably close to standard attention. Concretely, the paper proves that with $m$ random features, learned scale filters, and $L$ wavelet levels, $\|\mathrm{WERSA}(Q,K,V) - \mathrm{Attention}(Q,K,V)\|_F \leq \varepsilon\|V\|_F + O(2^{-\alpha L})$ holds with probability at least $1-\delta$, and the total complexity is $O(nd(d+m+L))$, which is $O(n)$ for fixed $d,m,L$. The same theorem states that the adaptive filter $\hat{F}$ converges to the optimal filter $F^*$ at rate $O(\sqrt{d \log n / n})$. Empirically, WERSA reports the highest accuracy on all tested datasets and is the only method among those evaluated that both fits in memory and performs well on 128k-token inputs, while using roughly a quarter of the FLOPS and a fifth of the training time of vanilla attention on ArXiv classification.

Load-bearing premise

The proof assumes that the ReLU random feature map $\phi(x) = \mathrm{ReLU}(xR/\beta)$ is an unbiased approximation of the softmax kernel, a property established for positive features specially constructed for softmax, not for ReLU features; the paper supplies no proof that this map satisfies the needed identity.

Editorial extensions

If this is right

Long inputs of hundreds of thousands of tokens become trainable on one GPU, because the mechanism avoids the quadratic attention matrix and the memory out-of-memory errors it causes.
With $m \geq (16/\varepsilon^2) \log(2n^2/\delta)$ random features, WERSA's output stays within $\varepsilon\|V\|_F + O(2^{-\alpha L})$ of standard multi-head attention with probability at least $1-\delta$, so fidelity is controllable by choosing $\varepsilon$.
The learned adaptive filters converge to the optimal filter at rate $O(\sqrt{d \log n / n})$ as sequence length grows, meaning longer sequences improve the quality of the input-dependent filtering.
Across IMDB, CIFAR-10/100, ListOps, and ArXiv, WERSA reports the best accuracy among seven attention mechanisms while cutting training time by up to 81% and FLOPS by 73.4% on ArXiv.
On ArXiv-128k, where standard attention and FlashAttention-2 run out of memory, WERSA reports the best accuracy (79.1%) and AUC (0.979) among methods that fit, at half the training time of Waveformer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the theorem's premise would swap the ReLU random feature map for the positive softmax features used in the cited kernel-approximation work; if WERSA's accuracy holds, the wavelet filtering, not the specific feature map, is carrying the gain, and the unbiasedness gap in the proof is closed.
The theorem's error bound treats wavelet truncation and random features as independent, but in practice the adaptive filter is learned from the same data, so the two error terms may interact; a testable extension is to measure empirical Frobenius error against both terms on held-out sequences with increasing $n$ and $L$.
Varying the wavelet level $L$ would map a concrete trade-off: deeper decompositions reduce the $O(2^{-\alpha L})$ truncation error but raise cost toward $O(n \log n)$, so long-document benchmarks could reveal the practical optimum.
Inspecting the learned scale weights $\omega_i$ for different input classes would test the claimed mechanism of content-adaptive filtering, namely that the model suppresses noisy high-frequency scales and enhances low-frequency global patterns when the input calls for it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: the paper is an architecture paper, not a theory paper. The WERSA combination—Haar wavelet decomposition, learned gating, random feature projection—is a reasonable new entry in the linear-attention zoo, and the empirical section is broader than many of its peers: five tasks, seven attention baselines, ablations. The ArXiv-128k result is the most interesting bit: WERSA matches Waveformer on accuracy and halves its wall-clock time while quadratic methods OOM.

That said, the headline theorem should not have been the headline. The random-feature step uses phi(x)=ReLU(xR/beta) and cites Choromanski et al. for the softmax approximation. That citation doesn't apply: Performer's unbiased features are positive exponentials, and ReLU inner products give a different kernel. No derivation is provided to show the ReLU map approximates softmax with O(1/sqrt(m)) error, so the approximation guarantee has no support. The wavelet error term O(2^{-alpha L}) is asserted from Mallat rather than derived for the actual filtered reconstruction, and the 'optimality of adaptive filters' part is a generic Rademacher bound with F* defined as the minimizer of the same error they want to bound—circular.

The empirical claims are more subdued than the abstract suggests. The accuracy gains are small: on IMDB, WERSA is 0.8901 vs 0.8892 for standard; on CIFAR-100 it is statistically tied with Waveformer. The big win is on ArXiv-128k for speed, not accuracy. The ablation study is useful and shows each component contributes, but that doesn't rescue the theory.

Who is this for? Someone looking for a practical linear-attention variant to try, especially on long sequences, could get value from the architecture and benchmark setup. But the paper as written overstates its theoretical grounding. The fix is either to use actual positive random features and prove the kernel bound, or to drop the theorem and present WERSA as an empirical study with ablations.

On balance, I'd send it to review—there is enough real engineering here that a referee could help the author reposition it—but the current version needs major revision before acceptance. I wouldn't cite it until the code is released and the theory is fixed.

Referee Report

3 major / 4 minor

Summary. The paper proposes WERSA, a linear-time attention mechanism that combines Haar wavelet decomposition, input-dependent filtering, and ReLU-based random feature projections to approximate standard attention. It claims a three-part theorem: (1) an approximation guarantee of the form ||WERSA(Q,K,V) - Attention(Q,K,V)||_F <= epsilon||V||_F + O(2^{-alpha L}), (2) linear computational complexity, and (3) convergence of learned adaptive filters to an optimal filter at rate O(sqrt(d log n / n)). The paper then reports experiments on IMDB, CIFAR-10/100, ListOps, ArXiv, and ArXiv-128k, comparing WERSA against MHA, FlashAttention-2, FNet, Linformer, Performer, and Waveformer, with claim of best accuracy and large speedups.

Significance. If the approximation theorem were correct, WERSA would be a notable contribution to efficient attention, offering a provably linear-time mechanism with input-adaptive multi-scale filtering. The paper includes a reasonably broad empirical benchmark on a single GPU and a component-wise ablation study, which are useful contributions. However, the central theoretical guarantees are not established: the random-feature argument relies on an inapplicable reference, the wavelet error bound is asserted without the necessary regularity assumptions, and the filter-optimality claim is a generic generalization bound that does not validate the WERSA architecture. The empirical results are suggestive but cannot substitute for the missing theory, and the paper explicitly markets the theorem as its main contribution. The manuscript is not ready for publication in its current form.

major comments (3)

[Section 3.3, Appendix B Part 1, Eqs. (5)-(7)] The proof asserts that phi(x)=ReLU(xR/beta) is an unbiased or near-unbiased approximation of the softmax kernel, citing Choromanski et al. [2020]. Performer's softmax random features are positive maps of the form exp(omega^T x - ||x||^2/2)/sqrt(m); their unbiasedness uses the Gaussian integral identity for exponentials. For ReLU features with Gaussian R, E[ReLU(q^T R/beta) ReLU(k^T R/beta)] is a piecewise-polynomial kernel that is not exp(q^T k / sqrt(d)) for any choice of beta, and the paper provides no alternative derivation. The stated bound E||phi(q)^T phi(k) - kappa(q,k)|| <= C/sqrt(m) is therefore unsupported. Since the theorem's Frobenius guarantee is decomposed into this random-feature term plus a wavelet term, Part 1 of the theorem is not established.
[Appendix B, Part 1, Wavelet Error] The bound ||Q - Q_F||_F <= C_Q 2^{-alpha L}, cited to Mallat [1999], is asserted without stating the regularity class of Q and K. For arbitrary bounded sequences, Haar approximation error does not necessarily decay exponentially with the number of levels; classical wavelet approximation rates require smoothness assumptions such as Besov or Sobolev regularity. The theorem states the bound for arbitrary Q,K in R^{n x d}, so the O(2^{-alpha L}) term is not a consequence of the stated hypotheses. Because this term is an explicit part of the advertised error guarantee, the wavelet half of the proof is also unsupported.
[Section 3.3, Part 3 and Appendix B, Part 3] The 'Optimality of Adaptive Filters' claim is circular and does not establish a property of WERSA. F* is defined as the minimizer of exactly the approximation error the paper wants to bound, and the displayed rate O(sqrt(d log n / n)) is a standard Rademacher-style generalization bound for empirical risk minimization over a fixed hypothesis class. No argument shows that WERSA's g network can represent or approach F*, and no distribution over inputs is specified for the 'n' in the bound. Consequently this part does not substantiate the claim of near-optimal input-dependent filtering.

minor comments (4)

[Table 1, ArXiv-128k rows] Standard and FlashAttn-2 are listed as OOM for accuracy, AUC, and time but have numeric FLOPS values (9330.0 and 9200.0); this is contradictory and should be clarified, since FLOPS should also be unavailable if the run failed.
[Section 3.3, theorem statement] The phrase 'to produce a limiting value as a function approaches infinity' is not meaningful; the intended limit should be stated as n -> infinity, and the dependence of alpha on 'smooth conditions' should be formalized or removed.
[Section 3.2 and Algorithm 2] The complexity analysis claims 'Unified Filtered Processing: O(nd) ... reducing the multiplier of L', but Algorithm 2's FilteredIDWT loops over L coefficient scales; the claimed O(n) complexity is only valid when L is treated as a constant, and the text should distinguish the asymptotic statement from the actual implementation.
[Section 5 and Table 1] Several reported advantages over the next-best method are within one standard deviation (e.g., CIFAR-100 and IMDB accuracy); the text should avoid claiming uniform superiority without significance testing or explicit error-bar comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: WERSA's derivation chains rely on external results and a generic generalization bound; the identified gaps are soundness issues, not reductions to the paper's own inputs.

full rationale

I walked the derivation chain in Section 3.3 and Appendix B. The approximation theorem decomposes the error into a random-feature term and a wavelet-truncation term. The random-feature term is imported from Choromanski et al. 2020, and the wavelet term from Mallat 1999 and Gao-Pavel 2017; these are external mathematical results, not self-citations, and the paper's use of them is not a case where an output quantity is defined as the fitted value of an input quantity. The known problem with the random-feature step is that Performer's proof applies to positive random features, whereas WERSA uses phi(x)=ReLU(xR/beta); this makes the claimed bound E||phi(q)^T phi(k)-kappa(q,k)|| <= C/sqrt(m) unsupported. That is a correctness or evidence gap, not a circular reduction: no equation in the paper makes WERSA's output equal to the random-feature assumption by construction. The adaptive-filter optimality claim (Part 3) does define F* as the minimizer of the same norm appearing in the theorem, and then asserts ||hatF-F*|| <= O(sqrt(d log n/n)) via Rademacher complexity. However, the paper never shows that the learned g network's filter hatF is the empirical minimizer of that attention-error objective, so the step is a non-sequitur rather than a definitional equivalence; it is not a fitted parameter being renamed as a prediction. The empirical sections compare WERSA against external baselines (MHA, FlashAttention-2, FNet, Linformer, Performer, Waveformer) on standard benchmarks, so the empirical claims are externally falsifiable and do not reduce to the theorem's assumptions. Overall, the paper's central derivation is incomplete and contains serious theoretical gaps, but I do not find a circular step in which a prediction or first-principles result is equivalent to its inputs by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central claims rest on several unverified assumptions: the validity of ReLU random features for softmax approximation, the applicability of wavelet truncation bounds to learned filters, and the smoothness of Q and K. The hyperparameters L, m, and beta are chosen without sensitivity analysis. The 'optimal filter' claim is a generic learning-theoretic statement rather than a property of the specific architecture. No new physical entities are introduced.

free parameters (4)

Wavelet levels L = 2
Set to 2 in all experiments; the paper does not justify this choice or show sensitivity.
Random feature dimension m = 1024
Fixed to 1024; the theorem gives a lower bound but no upper bound or tuning procedure.
Bandwidth parameter beta = 1.0
Fixed to 1.0 in experiments; described as trainable but not updated in practice.
Scale weights omega_i = learned
Learnable per-scale parameters; central to the method but their initialization and regularization are not specified.

assumptions (5)

domain assumption ReLU random features provide a valid approximation of the softmax kernel
Invoked in Eq. (6)-(7) and Appendix B; not true under standard random feature theory and not proven here.
domain assumption Wavelet approximation error decays as O(2^{-alpha L}) for filtered coefficients
Appendix B Part 1 borrows this from Mallat [1999], but the learned filter F is not a projection operator, so the bound does not follow.
domain assumption Smoothness of Q and K with decay rate alpha
The theorem assumes alpha > 0 depending on smoothness, but no verification on real data is provided.
domain assumption Input sequences padded to powers of two
The pseudocode pads to the next power of two for the Haar transform; this changes the effective sequence length and is not discussed in the main text.
standard math Rademacher complexity bound applies to the filter network g
Appendix B Part 3 uses a generic bound; it assumes g is an l2-regularized network, but the architecture is not specified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scaling Attention to Very Long Sequences in Linear Time with Wavelet-Enhanced Random Spectral Attention (WERSA)." pith.science (2026). https://pith.science/paper/5NE2ASQF

@misc{pith2026250708637,
  author       = {Pith},
  title        = {Pith review of: Scaling Attention to Very Long Sequences in Linear Time with Wavelet-Enhanced Random Spectral Attention (WERSA)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NE2ASQF}},
  note         = {Machine review of arXiv:2507.08637}
}

abstract

Transformer models are computationally costly on long sequences since regular attention has quadratic $O(n^2)$ time complexity. We introduce Wavelet-Enhanced Random Spectral Attention (WERSA), a novel mechanism of linear $O(n)$ time complexity that is pivotal to enable successful long-sequence processing without the performance trade-off. WERSA merges content-adaptive random spectral features together with multi-resolution Haar wavelets and learnable parameters to selectively attend to informative scales of data while preserving linear efficiency. Large-scale comparisons \textbf{on single GPU} and across various benchmarks (vision, NLP, hierarchical reasoning) and various attention mechanisms (like Multiheaded Attention, Flash-Attention-2, FNet, Linformer, Performer, Waveformer), reveal uniform advantages of WERSA. It achieves best accuracy in all tests. On ArXiv classification, WERSA improves accuracy over vanilla attention by 1.2\% (86.2\% vs 85.0\%) while cutting training time by 81\% (296s vs 1554s) and FLOPS by 73.4\% (26.2G vs 98.4G). Significantly, WERSA excels where vanilla and FlashAttention-2 fail: on ArXiv-128k's extremely lengthy sequences, it achieves best accuracy (79.1\%) and AUC (0.979) among viable methods, operating on data that gives Out-Of-Memory errors to quadratic methods while being \textbf{twice as fast} as Waveformer, its next-best competitor. By significantly reducing computational loads without compromising accuracy, WERSA makes possible more practical, more affordable, long-context models, in particular on low-resource hardware, for more sustainable and more scalable AI development.

Figures

Figures reproduced from arXiv: 2507.08637 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 7 canonical work pages

[2020]

T Brown, B Mann, N Ryder, et al

URL https://arxiv.org/pdf/2004.05150. T Brown, B Mann, N Ryder, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems ,

arXiv 2004
[1999]

Nikita Nangia and Samuel R. Bowman. Listops: A diagnostic dataset for latent tree learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 92–99,

work page 2018
[1]

FFT-Based Attention FFT-based attention mechanisms [Lee-Thorp et al., 2021, Gu et al., 2022] leverage Fast Fourier Transform’s efficiency to calculate attention in the frequency domain. The hypothesis is that time domain convolution can be equated to element-wise multiplication in the frequency domain: f ∗ g = F −1(F (f ) · F(g)), (12) where F and F −1 de...

work page 2021
[6]

On the use of arxiv as a dataset

Colin B Clement, Matthew Bierbaum, Kevin P O’Keeffe, and Alexander A Alemi. On the use of arxiv as a dataset. arXiv preprint arXiv:1905.00075,

arXiv 1905
[8]

T Dao, D

URL https://arxiv.org/pdf/1901.02860. T Dao, D. Y Fu, S Ermon, and A Rudra. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Sys- tems,

arXiv 1901
[9]

URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf . Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

arXiv 2022
[11]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,

arXiv 2010
[14]

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret

URL https://arxiv.org/pdf/2111.00396. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR,

arXiv

Show all 22 references

[16]

Mega: Moving average equipped gated attention

10 Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. arXiv preprint arXiv:2209.10655,

arXiv
[18]

Rwkv: Reinventing rnns for the transformer era

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048,

arXiv
[19]

Random feature attention

Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. arXiv preprint arXiv:2103.02143,

arXiv
[20]

Compressive transformers for long-range sequence modelling

Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507,

1911 arXiv
[22]

S Wang et al

URL https: //arxiv.org/abs/1706.03762. S Wang et al. Linformer: Self-attention with linear complexity,

arXiv
[23]

Y Xiong, Z Zeng, et al

URL https://arxiv.org/ pdf/2006.04768. Y Xiong, Z Zeng, et al. Nyströmformer: A nyström-based algorithm for approximating self- attention. In Proceedings of the AAAI Conference on Artificial Intelligence,

2006 arXiv
[24]

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin

URL https://proceedings.neurips.cc/paper_ files/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf. Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Internat...

2020
[25]

11 Appendix A: Background Self-Attention The self-attention mechanism in transformer networks [Vaswani et al., 2017] extracts interactions between each pair in a sequence. For a input sequence X ∈ Rn×d with n being the length and d being the feature space, self-attention first...

2017
[27]

Here, R ∈ Rd×m has independent and identically distributed Gaussian entries, E ϕ(q)T ϕ(k) − κ(q, k) ≤ C√m , where κ refers to the softmax kernel

for approximating dot-product kernels is extended by using ReLU features defined as ϕ(x) = ReLU(xR/β). Here, R ∈ Rd×m has independent and identically distributed Gaussian entries, E ϕ(q)T ϕ(k) − κ(q, k) ≤ C√m , where κ refers to the softmax kernel. Using McDiarmid’s inequality...

1999
[2019]

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al

URL https://arxiv.org/pdf/1904.10509. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794,

1904 arXiv
[2021]

Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar

URL https://arxiv.org/pdf/2105.03824. Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research, 24(388):1–26,

arXiv
[2022]

Peter L Bartlett and Shahar Mendelson

URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 960a172bc7fbf0177ccccbb411a7d800-Paper-Conference.pdf . Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov)...

2022
[2023]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

2019
[2024]

On the properties of the softmax function with application in game theory and reinforcement learning

Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805,

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2020]

T Brown, B Mann, N Ryder, et al

URL https://arxiv.org/pdf/2004.05150. T Brown, B Mann, N Ryder, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems ,

arXiv 2004

[2] [1999]

Nikita Nangia and Samuel R. Bowman. Listops: A diagnostic dataset for latent tree learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 92–99,

work page 2018

[3] [1]

FFT-Based Attention FFT-based attention mechanisms [Lee-Thorp et al., 2021, Gu et al., 2022] leverage Fast Fourier Transform’s efficiency to calculate attention in the frequency domain. The hypothesis is that time domain convolution can be equated to element-wise multiplication in the frequency domain: f ∗ g = F −1(F (f ) · F(g)), (12) where F and F −1 de...

work page 2021

[4] [6]

On the use of arxiv as a dataset

Colin B Clement, Matthew Bierbaum, Kevin P O’Keeffe, and Alexander A Alemi. On the use of arxiv as a dataset. arXiv preprint arXiv:1905.00075,

arXiv 1905

[5] [8]

T Dao, D

URL https://arxiv.org/pdf/1901.02860. T Dao, D. Y Fu, S Ermon, and A Rudra. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Sys- tems,

arXiv 1901

[6] [9]

URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf . Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

arXiv 2022

[7] [11]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,

arXiv 2010

[8] [14]

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret

URL https://arxiv.org/pdf/2111.00396. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR,

arXiv