Pith. sign in

REVIEW 3 major objections 6 minor 46 references

Scaling Context Requires Rethinking Attention

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing softmax with a p-th power inner product turns attention into a linear-cost layer with a tunable state, and the paper shows it beats both softmax and linear attention per FLOP on 64k-token text.

desk verdict A solid kernel-and-empirics paper with an honest limitations section; the WSFR balance principle is too under-tested to carry the framing, but the engineering and the loss-per-FLOP comparison are real. read the letter →

arxiv 2507.04239 v1 pith:OTPPFXCN submitted 2025-07-06 cs.LG cs.AI

Carles Gelada , Jacob Buckman , Sean Zhang , Txus Bach This is my paper
classification cs.LGcs.AI
keywords powerattentionlinearstateexpansionsymmetricembeddinglong-contexttrainingin-contextlearningweight-stateFLOPratioGPUkernels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-context training has three competing goals: a state large enough to use the context, compute that does not grow quadratically with context, and the ability to do in-context learning. The paper argues that neither exponential attention nor ordinary linear attention satisfies all three: the first spends too much state compute, the second too little, and windowed attention loses in-context learning. It introduces power attention, which replaces the softmax exponential with the p-th power of the inner product, making the layer a linear attention whose state size is tunable through p without adding parameters. In experiments on 65,536-token documents, power attention reaches lower heldout best-context loss per FLOP than both exponential and original linear attention, and its kernels run several times faster than Flash Attention at long contexts.

What carries the argument

The load-bearing identity is the symmetric power embedding: $\mathrm{SPOW}_p(x)$ lists the degree-$p$ monomials of $x$'s entries with multinomial coefficients so that $\langle \mathrm{SPOW}_p(q), \mathrm{SPOW}_p(k)\rangle = (q^\top k)^p$. This turns power attention into linear attention with a constant-size recurrent state of dimension $v\binom{d+p-1}{p}$, giving $p$ as a parameter-free control over state size and thus over the weight-state FLOP ratio. Around this identity the paper builds a gated chunked recurrence and GPU kernels that expand tiles of keys and queries on the fly, so the expanded $D$-dimensional objects are never materialized in memory.

What would settle it

Train the same RWKV architecture with power attention at $p$ values producing weight-state FLOP ratios roughly 1:1, 30:1, and 1:30 at equal total FLOPs and equal context, and compare heldout loss; if the unbalanced configurations match or beat the balanced one, the balance principle behind $p=2$ is refuted.

Watch

Extended reading notes

Core claim

The paper shows that $(q^\top k)^p$ attention can be written as plain linear attention with a state of size $v \binom{d+p-1}{p}$, by using the symmetric power embedding $\mathrm{SPOW}_p$, which satisfies $\langle \mathrm{SPOW}_p(q), \mathrm{SPOW}_p(k)\rangle = (q^\top k)^p$. Because $p$ is a hyperparameter and not a learned parameter, this gives a parameter-free dial for state size that can keep the weight-state FLOP ratio near 1:1 at long context. The experiments then show that a $p=2$ power attention RWKV model trained on 65,536-token batches dominates both exponential attention and original RWKV linear attention in loss per FLOP within the studied compute budget, while preserving in-context learning much better than windowed attention.

Load-bearing premise

The central premise is that compute-optimal sequence models should keep weight and state FLOPs roughly balanced; if that rule is wrong, the choice of $p=2$ and the interpretation of the long-context comparison lose their support.

Editorial extensions

If this is right

  • At 65,536-token contexts, $p=2$ power attention runs 8.6x faster than Flash Attention at head size 32 and 3.3x faster at head size 64.
  • Within the paper's FLOP budget, power attention obtains lower heldout best-context loss per FLOP than both exponential attention and RWKV's original linear attention.
  • Because $p$ adjusts state size independently of parameters, a single architecture can maintain a balanced weight-state FLOP ratio across model scales and context lengths.
  • Power attention's in-context learning curves are steeper than windowed attention with the same state size, and all four scaling axes tested steepen the curve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run is to fit optimal $p$ from a scaling law across FLOP budgets, possibly per layer, turning the balance principle into a quantitative recipe.
  • In domains where compute-optimal context is much longer than 64k (agent traces, chain-of-thought, audio, video), power attention's cost advantage over softmax attention should grow with context length.
  • The paper uses standard softmax normalization, requiring positive attention scores and limiting $p$ to even powers; replacing the normalization could admit odd $p$ and finer state-size control.
  • Because the open-source kernels lag Flash Attention in hardware utilization, reimplementing the fused expansion in lower-level kernels could turn the per-FLOP dominance into larger wall-clock dominance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that neither standard softmax transformers nor existing subquadratic/linear-attention architectures are well suited to training at long context lengths, and introduces "power attention," a linear-attention layer whose state size can be varied through a hyperparameter p by using the p-th power of the inner product as the attention kernel. The authors prove (Lemmas 4.1 and 4.2) that the p-th power kernel can be represented exactly as an inner product of an expanded feature map, either the full tensor power TPOW or the symmetric power SPOW, and they propose a hardware-aware tiled variant (TSPOW) with open-source Triton kernels. The paper introduces a "weight-state FLOP ratio" (WSFR) balance principle in Section 3.2, uses it to select p=2, and then reports experiments on LongCrawl64 showing that a p=2 power-attention RWKV model achieves lower heldout best-context loss per FLOP than both native RWKV linear attention and exponential attention when training on 64k-token sequences in the roughly 1e9 TeraFLOP regime, while also being faster than Flash Attention at that context length. The paper also reports that power attention shows better in-context learning per FLOP than windowed attention.

Significance. If the results hold, power attention is a practically useful linear-time attention layer with an adjustable state size, backed by a clean mathematical identity and an open-source kernel implementation. The proofs of Lemmas 4.1 and 4.2 are parameter-free and correct, and the per-FLOP empirical comparison is a strength, as is the open-sourced implementation and the explicit reporting of wall-clock speedups. The central empirical claim is internally consistent: in the stated FLOP regime at 64k context, p=2 power attention does achieve lower loss per FLOP than the two compared baselines. However, the paper's theoretical framing rests on the WSFR balance principle, and that principle is not convincingly validated by the evidence presented. The choice of p=2, which is the only instantiation tested in the long-context comparison, is justified entirely by that principle, so the gap between the paper's broad conclusions and the empirical support is the main weakness.

major comments (3)
  1. [§3.2, Figure 2, and Appendix D] The claim that the three configurations in Figure 2 are "nearly identical except for WSFR" is contradicted by the details given in Appendix D. The three runs differ simultaneously in depth (8 vs 24 vs 26 layers), width (512 vs 1024 vs 1280), context length (65536 vs 4096 vs 64), and batch size (32 vs 512 vs 32768). Loss differences across these runs therefore cannot be attributed to WSFR alone, so the experiment does not isolate the balance principle. This matters because the WSFR balance principle is the paper's stated reason for choosing p=2, and if it is unvalidated, the central design choice loses its theoretical grounding.
  2. [§5.3 and §4] The long-context comparison in Figure 9a tests only p=2 and does not sweep over p, even though the paper's central contribution is that "state size can be adjusted independently of parameters" (Section 4). Without a sweep over p, the empirical dominance of power attention over RWKV linear attention and exponential attention cannot be attributed to the state-size/WSFR mechanism; a p=2 model is one fixed point in the design space, and the claimed advantage of adjustable state size is not directly demonstrated.
  3. [§5.3, §6, and Abstract] The paper states in Section 5.3 that a context length of 65536 is far larger than compute-optimal in this setting and that, given sufficient training FLOPs, exponential attention would overtake power attention because of its larger state. These are important qualifications, but they appear only in the body; the Abstract and Section 6 state more broadly that power attention "dominates both exponential attention and linear attention at long-context training." This overstates the evidence, which supports a narrower claim about the specific FLOP regime and dataset. The authors should either strengthen the empirical evidence (e.g., a p-sweep and a compute-optimal context analysis) or qualify the claims in the Abstract and Conclusions.
minor comments (6)
  1. [§5.1] The text says "RKWV architecture" but the architecture name elsewhere is RWKV; this typo should be corrected.
  2. [Appendix D] There are several typos in this appendix: "resuls" should be "results," "architeture" should be "architecture," and the sentence "Here we we prove" in Appendix A should read "Here we prove."
  3. [Algorithm 2 (Appendix F.4)] The algorithm description uses "subtitles" where "subtiles" is meant; this should be fixed to avoid confusion.
  4. [Appendix C] The text references "Figure C" but the actual figure is labeled as part of Figure 10; the cross-reference should be corrected.
  5. [Table 5 (Appendix F.1) and Section 4] The implementation computes the power in log space using |S| (the absolute value of the attention score), whereas the theoretical identity in Lemma 4.1 and 4.2 is for (q·k)^p without absolute value. For even p these coincide, but the manuscript should explicitly state that the implementation targets even p or otherwise reconcile the absolute value with the stated theory.
  6. [References] The manuscript cites "Buckman and Gelada [a]" and "Buckman and Gelada [b]" as companion works, but these do not appear to be publicly available at the cited locations. The authors should provide arXiv identifiers or other accessible references, or at minimum flag these as "in preparation" in the reference list.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the load-bearing comparisons are external and the state-expansion identity is a proven mathematical fact, with the WSFR argument being a heuristic design principle rather than a fitted input.

full rationale

The derivation chain is not circular. Power attention is introduced in Eq. 7 as the p-th power of the inner product, and Lemmas 4.1 and 4.2 prove that this equals a linear attention with state expansion via TPOW/SPOW (Appendix B); this is a parameter-free identity, not a fit. The choice p=2 is motivated by the WSFR balance heuristic (Section 3.2), which rests on a doubling argument and the Figure 2 experiment; whatever the weaknesses of that experiment (Appendix D shows the three runs differ in depth, width, context, and batch size), those are confounds in a heuristic justification, not a case where a predicted quantity is defined as its own input. The central empirical claim (Section 5.3, Figure 9a) is an external comparison of heldout loss per FLOP against RWKV linear attention and softmax attention on LongCrawl64; no baseline loss is computed from WSFR values or from p. The paper itself states limitations: 'a context length of 65536 is far larger than is compute-optimal in this setting' and 'we expect that given sufficient training FLOPs, the attention model would overtake the power attention model'. Self-citations (Buckman and Gelada [a,b], Buckman 2024) supply the chunked form, the best-context-loss metric, and the dataset, but the chunked form is re-derived in Appendix A and the metric is a reporting convention; none is load-bearing in the main derivation. Thus no equation reduces by construction to its own inputs; score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two kinds of inputs: parameter-free math (symmetric power identity, chunked form) and domain assumptions about what makes an architecture compute-optimal (1:1 weight-state FLOP ratio) and about the dataset (LongCrawl64 has long-range structure). The p, chunk size, and d_tile are free hyperparameters chosen by the authors, not fitted constants in a derivation.

free parameters (3)
  • p (power attention degree) = 2 for main experiments
    Degree of the power kernel; controls state expansion via the symmetric power (Lemma 4.2). Chosen to balance weight-state FLOPs at 4k-64k contexts, not fitted to validation loss.
  • chunk size c = tuned per GPU (see Fig. 6c)
    Chunked-form chunk size trades off state FLOPs versus intra-chunk attention FLOPs; selected empirically for throughput.
  • d_tile (TSPOW tile size) = 8 for p=2, 4 for p=3
    Tile size in the TSPOW kernel controlling GPU efficiency and expansion dimension; found empirically.
assumptions (4)
  • domain assumption Compute-optimal models should have weight-state FLOP ratio near 1:1
    Argued in Section 3.2 with a doubling heuristic and one experiment (Figure 2); no formal proof. The design of power attention's p knob rests on this.
  • domain assumption The in-context learning curve of training loss at each token position measures genuine in-context learning ability
    Used throughout Sections 3 and 5 to compare attention variants; assumes loss decrease at long positions reflects the model using context, not other artifacts.
  • domain assumption LongCrawl64 contains documents with true long-term structure
    Stated in Section 5 intro as the reason for using this dataset; no quantitative evidence of long-range dependencies is provided.
  • domain assumption Softmax-style normalization requiring positive attention scores is appropriate
    The paper uses it and notes in Section 6 that it restricts to even powers, making p tuning coarse.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Context Requires Rethinking Attention." pith.science (2026). https://pith.science/paper/OTPPFXCN

@misc{pith2026250704239,
  author       = {Pith},
  title        = {Pith review of: Scaling Context Requires Rethinking Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTPPFXCN}},
  note         = {Machine review of arXiv:2507.04239}
}
read the original abstract

We argue that neither transformers nor sub-quadratic architectures are well suited to training at long sequence lengths: the cost of processing the context is too expensive in the former, too inexpensive in the latter. Approaches such as sliding window attention which reduce the cost-per-token of a transformer impair in-context learning, and so are also unsuitable. To address these limitations, we introduce power attention, an architectural layer for linear-cost sequence modeling whose state size can be adjusted independently of parameters, unlocking the advantages of linear attention on practical domains. We develop and open-source a set of GPU kernels for efficient power attention, identifying a novel pattern of operation fusion to avoid memory and bandwidth bottlenecks. Our experiments on the in-context learning of power attention shows that these models dominate both exponential attention and linear attention at long-context training.

Figures

Figures reproduced from arXiv: 2507.04239 by the authors.

Figure 1
Figure 1. Exponential attention (blue) vs linear attention (red). [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Compute-optimal transformers have a balanced WSFR. See Appendix D for details. Attention Context Length WSFR Exponential 1 024 8:1 Exponential 8 192 1:1 Exponential 65 536 1:8 Exponential 1 000 000 1:125 Linear 1 024 30:1 Linear 8 192 30:1 Linear 65 536 30:1 Linear 1 000 000 30:1 Window-8192 1 024 8:1 Window-8192 8 192 1:1 Window-8192 65 536 1:1 Window-8192 1 000 000 1:1 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparing learning curves if linear vs window-32 attention for several architectures and context lengths. (a) Window-32 attention. (b) Linear attention [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Hardware efficiency of Power Attention kernels. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 5
Figure 5. Figure 5: Illustration of TPOW, SPOW, and TSPOW. Our approach is to use the idea of tiling to in￾terpolate between TPOW and SPOW, harnessing benefits of both. Our proposed tiled symmet￾ric power expansion, TSPOW, operates on tiles of data (providing the GPU-friendly structure of…
Figure 7
Figure 7. Figure 7: Power attention demonstrates more ICL per FLOP than equivalent windowed attention. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: The impact of conventional scaling axes on in-context learning of power attention. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Comparison between different forms of attention on long context. The dashed line in 9a [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 10 canonical work pages

  1. [1]

    Simple linear attention language models balance the recall-throughput tradeoff, 2025

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff, 2025. URL https://arxiv.org/abs/2402.18668

    arXiv 2025

  2. [2]

    Longcrawl64: A Long-Context Natural-Language Dataset

    Jacob Buckman. Longcrawl64: A Long-Context Natural-Language Dataset . https://manifestai.com/articles/longcrawl64/, 2024. Accessed: 2025-05-15

    work page 2024

  3. [3]

    Linear Transformers Are Faster , a

    Jacob Buckman and Carles Gelada. Linear Transformers Are Faster , a

    work page

  4. [4]

    Compute-optimal Context Size , b

    Jacob Buckman and Carles Gelada. Compute-optimal Context Size , b

    work page

  5. [5]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    arXiv 1904

  6. [6]

    On the properties of neural machine translation: Encoder-decoder approaches, 2014

    Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches, 2014. URL https://arxiv.org/abs/1409.1259

    arXiv 2014

  7. [7]

    Flashattention-2: Faster attention with better parallelism and work partitioning, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691

    arXiv 2023

  8. [8]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/2205.14135

    arXiv 2022

Show all 46 references
  1. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  2. [10]

    Finding structure in time

    Jeffrey L Elman. Finding structure in time. Cognitive science, 14 0 (2): 0 179--211, 1990

    1990

  3. [11]

    Team Google, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lil...

    2025 arXiv

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  5. [13]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

    2024 arXiv

  6. [14]

    When attention sink emerges in language models: An empirical view

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.10781, 2024

    2024 arXiv

  7. [15]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  8. [16]

    Long short-term memory

    Sepp Hochreiter and J\" u rgen Schmidhuber. Long short-term memory. Neural Comput., 9 0 (8): 0 1735–1780, November 1997. ISSN 0899-7667. doi:10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735

    1997 doi

  9. [17]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  10. [18]

    Polysketchformer: Fast transformers via sketching polynomial kernels, 2024

    Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. Polysketchformer: Fast transformers via sketching polynomial kernels, 2024. URL https://arxiv.org/abs/2310.01655

    2024 arXiv

  11. [19]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  12. [20]

    Katharopoulos, A

    A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML), 2020. URL https://arxiv.org/abs/2006.16236

    2020 arXiv

  13. [21]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  14. [22]

    Forgetting transformer: Softmax attention with a forget gate

    Zhixuan Lin, Evgenii Nikishin, Xu Owen He, and Aaron Courville. Forgetting transformer: Softmax attention with a forget gate. arXiv preprint arXiv:2503.02130, 2025

    2025 arXiv

  15. [23]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  16. [24]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  17. [25]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/, 2025. Accessed: 2025-04-05

    2025

  18. [26]

    Online normalizer calculation for softmax, 2018

    Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax, 2018. URL https://arxiv.org/abs/1805.02867

    2018 arXiv

  19. [27]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilimbi, Benoit Prabhakaran, Mic...

    2019

  20. [28]

    Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Kris...

    2023 arXiv

  21. [29]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021

    2021 arXiv

  22. [30]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  23. [31]

    Theory, analysis, and best practices for sigmoid self-attention

    Jason Ramapuram, Federico Danieli, Eeshan Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self-attention. arXiv preprint arXiv:2409.04431, 2024

    2024 arXiv

  24. [32]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 0 68539--68...

    2023

  25. [33]

    Linear transformers are secretly fast weight programmers, 2021

    Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers, 2021. URL https://arxiv.org/abs/2102.11174

    2021 arXiv

  26. [34]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  27. [35]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  28. [36]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  29. [37]

    Retentive network: A successor to transformer for large language models

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  30. [38]

    Triton: an intermediate language and compiler for tiled neural network computations

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10--19, 2019

    2019

  31. [39]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  32. [40]

    Deep neural network based low-latency speech separation with asymmetric analysis-synthesis window pair

    Shanshan Wang, Gaurav Naithani, Archontis Politis, and Tuomas Virtanen. Deep neural network based low-latency speech separation with asymmetric analysis-synthesis window pair. In 2021 29th European Signal Processing Conference (EUSIPCO), pages 301--305. IEEE, 2021

    2021

  33. [41]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  34. [42]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37: 0 50528--50652, 2024

    2024

  35. [43]

    FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism , January 2024

    Songlin Yang and Yu Zhang. FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism , January 2024. URL https://github.com/fla-org/flash-linear-attention

    2024

  36. [44]

    Gated linear attention transformers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023

    2023 arXiv

  37. [45]

    Gated delta networks: Improving mamba2 with delta rule, 2025

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule, 2025. URL https://arxiv.org/abs/2412.06464

    2025 arXiv

  38. [46]

    Gated slot attention for efficient linear-time sequence modeling

    Yu Zhang, Songlin Yang, Rui-Jie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear-time sequence modeling. Advances in Neural Information Processing Systems, 37: 0 116870--116898, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.