ReGLA: Refining Gated Linear Attention

Boxing Chen; Ivan Kobyzev; Mehdi Rezagholizadeh; Peng Lu; Philippe Langlais

REVIEW 3 major objections 5 minor 12 references

ReGLA: Refining Gated Linear Attention

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A gated linear attention called ReGLA uses a bounded exponential feature map, a variance-reduction scale, and a refining forget gate to reach a WikiText-103 perplexity of 19.0, versus 20.8 for the prior gated baseline and 18.5 for softmax.

desk verdict A useful empirical paper on gated linear attention, but the variance-reduction theorem is derived for the wrong feature map; the scaling factor should be treated as tuned, not proven. read the letter →

arxiv 2502.01578 v3 pith:SHXSOUE3 submitted 2025-02-03 cs.CL

Peng Lu , Ivan Kobyzev , Mehdi Rezagholizadeh , Boxing Chen , Philippe Langlais This is my paper · ORCID

classification cs.CL

keywords linearattentiongatedfeaturemapsvariancereductionforgetgatesaturationlanguagemodelingefficienttransformers

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that three components of gated linear attention—feature map, normalization, and forget gate—can each be repaired to bring linear-time sequence models much closer to softmax attention quality. It proposes ReGLA, which uses a bounded exponential feature map $\phi(x)=\exp(x-\max(x))$, a variance-reduction scaling factor $\frac{1}{e\sqrt{d(e^2-1)}}$ derived for products of exponentials, and a refining gate that keeps gradient flow near gate saturation. In their experiments, ReGLA reaches perplexity 19.0 on WikiText-103 trained from scratch, versus 20.8 for the best prior gated linear baseline and 18.5 for softmax attention, and a hybrid that keeps half of the softmax layers reaches 17.8. If these results hold, linear attention no longer has to trade away a large share of modeling quality in exchange for constant-memory, linear-time inference.

What carries the argument

The machinery that carries the argument is the gated linear recurrence itself, updated as $S_t = F_t \odot S_{t-1} + v_t \phi(k_t)^\top$, with the refined forget gate $F_t = ((1-r_t)\odot g_t^2 + r_t\odot(1-(1-g_t)^2))\mathbf{1}^\top$. Three pieces make it work: the normalized exponential feature map, whose inner product is bounded by $d$ and therefore avoids the forward-computation instability of unbounded maps; the variance-reduction factor of Theorem 3.1, which rescales the dot product so its variance no longer grows like $e^2(e^2-1)d$; and the refining gate, which gives the forget gate nonzero gradient even when the gate activation is near 0 or 1. The paper also retains a stable normalization layer after the attention computation, arguing that its role goes beyond bounding gradients because the variance of the output $h_t$ depends on input length once sum normalization is dropped. The ablation in Table 6 attributes the perplexity gain from 20.7 to 19.0 to the new scaling factor.

What would settle it

Measure the empirical standard deviation of $\phi_q(x)^\top \phi_k(x)$ over many activation pairs from a trained ReGLA checkpoint at $d=64$ and compare it with the $\sqrt{e^2(e^2-1)d}$ predicted by Theorem 3.1 after the Appendix A.1 adjustment; a large mismatch would mean the variance-reduction factor is not doing what the theory says. A second check is to train the same model on a different corpus, such as code or books, and see whether the $\frac{1}{e\sqrt{d(e^2-1)}}$ scaling still beats $\frac{1}{\sqrt{d}}$ by the margin Table 6 reports.

Watch

Extended reading notes

Core claim

The central claim is that the previous gap between gated linear attention and softmax attention is not inherent but comes from three overlooked details. First, feature maps such as identity, ReLU, and ELU+1 produce unbounded inner products, so ReGLA uses $\phi(x)_i=\exp(x_i-\max_j x_j)$, whose inner product is always between 0 and the feature dimension $d$. Second, the standard $\frac{1}{\sqrt{d}}$ scaling factor does not match the variance of this feature map; Theorem 3.1 computes the variance as $e^2(e^2-1)d$ for standard normal inputs, and the paper therefore scales the inner product by $\frac{1}{e\sqrt{d(e^2-1)}}$. Third, the sigmoid forget gate saturates and loses gradient, so ReGLA interpolates it between $g_t^2$ and $1-(1-g_t)^2$ with an additional refining gate $r_t$. With these changes, the paper reports that plain ReGLA outperforms prior gated linear attention variants on language modeling and commonsense benchmarks, while the hybrid variant with half of the layers left as softmax outperforms the softmax transformer in their setup.

Load-bearing premise

The variance-reduction factor is derived for independent standard-normal inputs to a plain exponential map, but ReGLA uses $\exp(x-\max(x))$ on real activations. The appendix's repair replaces the sample maximum with its asymptotic expectation $\sqrt{2\ln d}$, an approximation that is exact only for large $d$ and near-normal hidden states; if those do not hold, the scaling factor that the paper credits for its main gain lacks theoretical support.

Editorial extensions

If this is right

Plain ReGLA reaches perplexity 19.0 on WikiText-103 from scratch, the closest value among the compared gated linear attention variants to the 18.5 of softmax attention in this setup.
A hybrid that leaves six of twelve softmax layers in place and replaces the rest with ReGLA beats the full softmax transformer in the paper's experiments, reaching 17.8 perplexity from scratch.
Post-linearization with continual pre-training recovers most of the pretrained model's quality, with ReGLA averaging 45.3 on six commonsense benchmarks zero-shot versus 44.9 for the fast-decay baseline.
ReGLA keeps the constant-memory, linear-time inference profile of prior gated linear attention, with memory and speed essentially identical to the fast-decay GLA while softmax memory grows with generation length.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One extension the paper leaves untested is measuring per-layer empirical variances of the feature-map inner product and either confirming the fixed factor or replacing it with a learned, layer-dependent scale.
The refined forget gate's anti-saturation property could transfer to other recurrent architectures whose gates saturate, such as state-space models or RWKV-style blocks, though the paper does not test this.
The hybrid result suggests the practical question is not linear versus softmax but how to allocate layers; the paper does not explore which layers should stay softmax, leaving room for a data-driven layer-allocation rule.
The normalized exponential feature map combined with gating behaves like a bounded, recency-biased kernel, which may open a theoretical connection to softmax kernel approximation; this is an interpretation, not a claim of the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a modest, mostly honest paper about three knobs in gated linear attention—feature map, normalization, variance scaling, and gating. The empirical work is decent for 160M-scale language modeling: ReGLA narrows the perplexity gap to softmax attention from roughly 2.3 points to 0.5 points, and the hybrid variant edges out softmax. The ablations in Table 6 suggest each ingredient contributes, and the baselines (ReLU, ELU, Hedgehog, Fast Decay) are fair. Credit where due: the exp(x−max(x)) feature map is not new, but testing it with the refined gate from Gu et al. (2020) inside GLA is reasonable, and the paper does it cleanly.

The soft spot is the variance theorem. Theorem 3.1 computes Var(Σ exp(x_i)exp(y_i)) under independent standard normal x_i, y_i. The actual feature map is exp(x_i − max_j x_j), and the subtrahend is a random variable that depends on the same entries. Appendix A.1 replaces the max with its asymptotic expectation and claims the result is an upper bound. That is not a valid reduction: conditional on the max, the remaining coordinates are truncated and correlated, so x_i − max has mean O(1), not −√(2 ln d). The variance of the true inner product is not e^{−4√(2 ln d)} e²(e²−1)d; for d=64 that would be absurdly small, while normalized exponential features actually give inner products of order 1. So the scaling factor 1/(e√(d(e²−1))) is not theoretically justified by this derivation. It might still be a good empirical choice—Table 6 shows it beats 1/√d—but the paper should say so and not hang a theorem on it.

Other concerns are minor: single-run experiments without error bars, 160M scale only, and the commonsense average hides task-level regressions (PIQA, HellaSwag). No code release limits reproducibility, but the setup is described well enough to reimplement.

Bottom line: this is a useful incremental paper for the efficient-attention crowd. The empirical comparison is informative even if the theory is shaky. It deserves a serious referee rather than a desk reject, but the final version should reframe the variance factor as an empirical hyperparameter or fix the derivation.

Referee Report

3 major / 5 minor

Summary. The paper proposes ReGLA, a variant of Gated Linear Attention (GLA) with three claimed contributions: a normalized exponential feature map exp(x − max x), a variance reduction scaling factor derived from a theorem, and a refined gating mechanism borrowed from prior recurrent-network work (Gu et al., 2020). Experiments on WikiText-103 language modeling from scratch and on post-linearization of Pythia-160m with continual pre-training report lower perplexity than GLA baselines (e.g., 19.0 vs. 20.8 from scratch) and modest average gains on commonsense reasoning benchmarks. The paper also includes ablations of feature dimension, feature map, and scaling factor.

Significance. If the results hold, ReGLA would be one of the closest linear-attention variants to softmax attention in the tested setting, and the paper's systematic decomposition of feature map, normalization, and gating would be a useful reference for the efficient-attention community. The authors provide a clean implementation based on the Flash Linear Attention library and a transparent ablation table. However, the central theoretical contribution — the variance reduction factor — is derived under assumptions that do not match the actually deployed feature map, which undermines the principled justification for one of the three main components. The empirical claims are also based on single runs without error bars, which is a concern for the small differences reported on common-sense benchmarks.

major comments (3)

[§3.1 and Appendix A.1] The variance reduction factor 1/(e√(d(e²−1))) is derived in Theorem 3.1 for inner products of exp(x)·exp(y) with independent standard-normal coordinates, but the deployed feature map is exp(x − max x). The Appendix's fix, replacing the random sample maximum by its asymptotic mean √(2 ln d) and treating x_i − √(2 ln d) as independent normals, is not a valid distributional reduction: conditional on the maximum, all other coordinates are truncated above it and are strongly dependent, so Theorem A.1 does not give the variance of the actual normalized-exponential inner product. Concretely, for d=64, Theorem A.1 implies that after applying the proposed scaling the variance is e^{−4√(2 ln 64)} ≈ 1e−5, which would make the scaled inner product nearly zero rather than variance-1. Thus the theoretical justification for the scaling factor is not supported, and the empirical gain in Table 6 (20.7 → 19.0) is left without a principled explanation.
[§4, Tables 4–6] All experimental results are reported for a single run with no error bars, confidence intervals, or significance tests. The differences between ReGLA and strong baselines on commonsense reasoning are small (e.g., Table 5 average 45.3 vs. 44.9 for Fast Decay in 0-shot), and the claim that ReGLA 'outperforms previous Gated Linear Attention mechanisms in extensive tasks' should be backed by repeated-seed statistics or at least a statement of variance across runs.
[§5.3, Table 6] The ablation does not isolate the contribution of the refined gating mechanism. Table 6 varies feature size, feature map, and scaling factor, but all rows use the refined gate. Since the refined gate is one of the paper's three main contributions, an ablation that removes it (e.g., comparing GLA with the same exp(x−max x) feature map and normalization, but with the vanilla outer-product gate of Eq. 10) is needed to quantify its effect.

minor comments (5)

[Figure 2 and surrounding text] The text and figure caption repeatedly say 'standard derivation' where 'standard deviation' is meant; also the label 'f(d) = e d(e2 1)' in the left panel is garbled and should be f(d) = e√(d(e²−1)).
[Table 5] The column headers 'Truth_QA1' and 'Truth_Qa2' are inconsistent; use 'TruthfulQA 1' and 'TruthfulQA 2' uniformly.
[Table 3] The 'Sum/Stable' column entries such as '✓/✗' are ambiguous; please clarify which symbol refers to sum normalization and which to stable normalization, and state the default configuration used in experiments.
[§6 Related Work] The sentence 'Wang et al. improve hybrid models...' cites a reference without a year or a full citation in the text; please add the proper citation.
[§3.1] The synthetic-data validation in Figure 2 uses the unnormalized exponential exp(x)·exp(y), not the deployed normalized exponential exp(x−max x)·exp(y−max y). The paper should state this explicitly, since the figure does not empirically validate the variance claim for the actual feature map.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the variance-reduction factor is computed from an explicit theorem, the refined gate is credited to external work, and no load-bearing step reduces to its own inputs.

full rationale

The paper's claimed derivation chain is self-contained. The variance-reduction factor 1/(e sqrt(d(e^2-1))) follows from Theorem 3.1 (Appendix A computes Var(sum exp(x_i)exp(y_i)) = e^2(e^2-1)d for independent standard-normal entries) and is then applied as a fixed scaling constant; it is not fitted to the target perplexity, and Table 6 provides an independent ablation (exp with 1/sqrt(d): 20.7 vs. exp with the proposed factor: 19.0). The exp(x-max(x)) feature map is defined directly in Eqs. 11-12 and its boundedness is shown by construction; no hidden equivalence to the result is involved. The refined gating mechanism is explicitly attributed to Gu et al. (2020), an external source, and is combined with the outer-product gate form of Mao (2022)/Yang et al. (2024a), so the central novelty is not justified by a self-citation chain. The self-citations in the paper (Lu et al. 2019, 2021, 2023; Wang et al. 2024; Xie et al. 2024) appear only in contextual related-work or background sentences and do not carry any load-bearing premise. The Appendix A.1 replacement of the sample maximum by its asymptotic expectation is an approximation that may raise a correctness concern about the theorem's applicability to the normalized feature map, but that is a technical-validity question, not a circularity: the scaling factor is not defined in terms of the reported perplexities, and the empirical comparison in Table 6 does not reduce to the theorem. No circular step can be quoted and exhibited, so the appropriate finding is 'no significant circularity' with score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central contribution is a combination of previously known components: normalized exponential feature maps, a variance scaling factor derived from log-normal statistics, and a refined gate borrowed from LSTM research. No new entities are postulated. The main assumptions are distributional and transfer claims.

assumptions (4)

domain assumption Feature map inputs are independent standard normals for the variance analysis
Theorem 3.1 assumes xi, yi ~ N(0,1) to derive the variance of exponentials; real activations are not standard normals.
ad hoc to paper Replacing sample max with its asymptotic expectation preserves the variance scaling
Appendix A.1 substitutes E[max] approx sqrt(2 ln d) to make the normalized exponential feature map tractable, which is an approximation not exact for finite d.
domain assumption Bounded inner products of feature maps stabilize training
Section 3.1 posits that unbounded inner products cause instability; the empirical ablation supports it but the mechanism is not proven.
domain assumption Refined gating analysis from LSTM literature transfers to GLA
Section 3.3 adapts the refined gate from Gu et al. 2020 to linear attention without new theoretical analysis specific to this architecture.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ReGLA: Refining Gated Linear Attention." pith.science (2026). https://pith.science/paper/SHXSOUE3

@misc{pith2026250201578,
  author       = {Pith},
  title        = {Pith review of: ReGLA: Refining Gated Linear Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHXSOUE3}},
  note         = {Machine review of arXiv:2502.01578}
}

read the original abstract

Recent advancements in Large Language Models (LLMs) have set themselves apart with their exceptional performance in complex language modelling tasks. However, these models are also known for their significant computational and storage requirements, primarily due to the quadratic computation complexity of softmax attention. To mitigate this issue, linear attention has been designed to reduce the quadratic space-time complexity that is inherent in standard transformers. In this work, we embarked on a comprehensive exploration of three key components that substantially impact the performance of the Gated Linear Attention module: feature maps, normalization, and the gating mechanism. We developed a feature mapping function to address some crucial issues that previous suggestions overlooked. Then we offered further rationale for the integration of normalization layers to stabilize the training process. Moreover, we explored the saturation phenomenon of the gating mechanism and augmented it with a refining module. We conducted extensive experiments and showed our architecture outperforms previous Gated Linear Attention mechanisms in extensive tasks including training from scratch and post-linearization with continual pre-training.

Figures

Figures reproduced from arXiv: 2502.01578 by the authors.

**Figure 1.** The overall model architecture of our REGLA. The right side depicts the regular linear attention with our safe exp feature maps and normalization layer and the left side depicts the refining gate mechanism. utilize this fast decay rule and evaluate their recurrent linear transformer in reinforcement learning problems. Compared to softmax attention’s implicit unbounded memory footprint requirement: KV cache (Kwon e… view at source ↗

**Figure 3.** Gradient analysis of refined forgetting Gates [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 2.** The left figure shows the standard derivation [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Plot of memory usage and the total prompt [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages

[8]

In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, Singapore, Decem- ber 6-10, 2023, pages 14048–14077

RWKV: reinventing rnns for the transformer era. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, Singapore, Decem- ber 6-10, 2023, pages 14048–14077. Association for Computational Linguistics. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021. Random feature attention. In 9th Internat...

arXiv 2023
[12]

Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim

OpenReview.net. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2024b. Parallelizing linear transform- ers with the delta rule over sequence length. In The Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tañón, Phili...

arXiv 2020
[626]

Stephanie Lin, Jacob Hilton, and Owain Evans

ACM. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics. Peng Lu, Ting Bai, an...

work page 2022
[2015]

CoRR, abs/1503.02531

Distilling the knowledge in a neural network. CoRR, abs/1503.02531. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulni...

arXiv 1997
[2019]

In Pro- ceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@PLDI 2019, Phoenix, AZ, USA, June 22, 2019, pages 10–19

Triton: an intermediate language and com- piler for tiled neural network computations. In Pro- ceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@PLDI 2019, Phoenix, AZ, USA, June 22, 2019, pages 10–19. ACM. 12 Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth...

arXiv 2019
[2020]

In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020

Reformer: The efficient transformer. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020. OpenReview.net. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonza- lez, Hao Zhang, and Ion Stoica. 2023. Efficient mem- ory management for large language model ...

work page 2020
[2021]

Linear transformers are secretly fast weight programmers. In Proceedings of the 38th Interna- tional Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Pro- ceedings of Machine Learning Research, pages 9355–

work page 2021
[2023]

Albert Gu and Tri Dao

A framework for few-shot language model evaluation. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces.CoRR, abs/2312.00752. Albert Gu, Karan Goel, and Christopher Ré. 2022. Effi- ciently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR ...

arXiv 2023

Show all 12 references

[2024]

arXiv preprint arXiv:2405.06640

Linearizing large language models. arXiv preprint arXiv:2405.06640. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, ...

2017 arXiv
[6565]

Huanru Henry Mao

Association for Computational Linguistics. Huanru Henry Mao. 2022. Fine-tuning pre-trained trans- formers into decaying fast weights. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, Decembe...

2022
[7439]

Hanting Chen, Zhicheng Liu, Xutao Wang, Yuchuan Tian, and Yunhe Wang

AAAI Press. Hanting Chen, Zhicheng Liu, Xutao Wang, Yuchuan Tian, and Yunhe Wang. 2024. Dijiang: Efficient large language models through compact kernelization. CoRR, abs/2403.19928. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ge...

2024 arXiv
[9366]

Jürgen Schmidhuber

PMLR. Jürgen Schmidhuber. 1992. Learning to control fast- weight memories: An alternative to dynamic recur- rent networks. Neural Comput., 4(1):131–139. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Ja- cob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B t...

1992 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [8]

In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, Singapore, Decem- ber 6-10, 2023, pages 14048–14077

RWKV: reinventing rnns for the transformer era. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, Singapore, Decem- ber 6-10, 2023, pages 14048–14077. Association for Computational Linguistics. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021. Random feature attention. In 9th Internat...

arXiv 2023

[2] [12]

Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim

OpenReview.net. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2024b. Parallelizing linear transform- ers with the delta rule over sequence length. In The Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tañón, Phili...

arXiv 2020

[3] [626]

Stephanie Lin, Jacob Hilton, and Owain Evans

ACM. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics. Peng Lu, Ting Bai, an...

work page 2022

[4] [2015]

CoRR, abs/1503.02531

Distilling the knowledge in a neural network. CoRR, abs/1503.02531. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulni...

arXiv 1997

[5] [2019]

In Pro- ceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@PLDI 2019, Phoenix, AZ, USA, June 22, 2019, pages 10–19

Triton: an intermediate language and com- piler for tiled neural network computations. In Pro- ceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@PLDI 2019, Phoenix, AZ, USA, June 22, 2019, pages 10–19. ACM. 12 Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth...

arXiv 2019

[6] [2020]

In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020

Reformer: The efficient transformer. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020. OpenReview.net. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonza- lez, Hao Zhang, and Ion Stoica. 2023. Efficient mem- ory management for large language model ...

work page 2020

[7] [2021]

Linear transformers are secretly fast weight programmers. In Proceedings of the 38th Interna- tional Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Pro- ceedings of Machine Learning Research, pages 9355–

work page 2021

[8] [2023]

Albert Gu and Tri Dao

A framework for few-shot language model evaluation. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces.CoRR, abs/2312.00752. Albert Gu, Karan Goel, and Christopher Ré. 2022. Effi- ciently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR ...

arXiv 2023