Pith. sign in

REVIEW 3 major objections 4 minor 40 references

The paper argues that the optimal attention operation is a per-token decision, not a per-layer one, and that a language model can learn to route each chunk of tokens to either Gated DeltaNet linear attention or softmax attention during trai

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:51 UTC pith:XUL7KC2J

load-bearing objection Promising token-level hybrid attention, but the router-training story needs to be made explicit before the 'search' claim can be trusted. the 3 major comments →

arxiv 2602.03681 v2 pith:XUL7KC2J submitted 2026-02-03 cs.CL cs.LG

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models

Difan Deng , Andreas Bentzen Winje , Lukas Fehring , Marius Lindauer This is my paper
classification cs.CL cs.LG
keywords token-level hybrid attentionlinear attentionGated DeltaNetsoftmax attentionattention routinglong-context language modelingneural architecture searchlength extrapolation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a language model can decide, separately for each small group of tokens, whether to process them with fast recurrent linear attention or with full softmax attention—and that this per-token choice can be learned from the language-modeling objective rather than fixed by architecture. The proposed model, NAtS-L, routes chunks through either Gated DeltaNet or softmax attention using a lightweight score layer, applying both operations within the same layer. If the central claim holds, hybrid attention no longer has to be layer-wise: softmax tokens can be reserved for the few places where long-term retrieval matters, while the rest of the sequence gets the cheaper linear path. The paper reports that this token-level split beats plain softmax, plain linear, and layer-wise hybrid baselines on retrieval benchmarks, preserves perplexity when the context is extended from 4k to 65k tokens, and cuts prefill cost by up to 5.4x versus a transformer at long context. The reason a reader should care is that it points toward a flexible efficiency–expressivity trade-off that adapts to content rather than being imposed by a fixed layer schedule.

Core claim

On its own terms, the paper's claim is that the optimal attention operation is a property of the token, not of the layer. NAtS-L splits every input into chunks, computes a score for each chunk from its mean-pooled features, and routes the chunk to either Gated DeltaNet linear attention or softmax attention via a hard top-1 choice. The two branches share the same query, key, and value projections; a column-wise mask keeps each branch from seeing tokens routed to the other, and the branch outputs are RMS-normalized and combined with q-dependent weights. Trained on 15B and 50B tokens at 4096 context, NAtS-L Hybrid reaches the best average accuracy on six retrieval benchmarks, keeps its perplexi

What carries the argument

The core mechanism is the Attention Score Layer with a hard per-chunk router. A mean-pooling operation followed by a linear layer maps each chunk to two scores; the higher score decides whether that chunk follows the linear branch (Gated DeltaNet, a delta-rule linear attention with a decay term) or the softmax branch. Branch selection is enforced by a column-wise causal mask: tokens assigned to one operation are skipped by the other, so softmax only loads its selected KV values and the linear hidden state only updates on linear chunks, with a decay applied to the hidden state even across softmax chunks. The two outputs are RMS-normalized and summed with weights produced from the query projec

Load-bearing premise

Load-bearing premise: the hard top-1 chunk-to-operation assignment of Section 4.1 can actually be trained by gradient descent; since the paper provides no smooth approximation of that argmax, the gradient formulas of Section 4.2 do not by themselves show that learning can change the router.

What would settle it

Compute the derivative of the hard chunk mask with respect to the router score: because the mask is piecewise-constant through an argmax, the gradient is zero almost everywhere unless a relaxation or straight-through estimate is added. A direct experiment would train only the router weights and check whether routing assignments actually change; if gradient norms on the score layer are always zero or assignments remain frozen, the paper's 'searching' claim is not supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the routing is genuinely learned, layer-wise hybrids become a special case: softmax attention can be placed exactly where long-term recall is needed instead of on fixed layers.
  • Long-context extrapolation improves: NAtS-L Hybrid keeps useful perplexity out to 65k tokens despite training at 4k, while the layer-wise hybrid baseline collapses beyond the training length.
  • Efficiency follows the content: because softmax only attends to its selected tokens, prefill cost scales with the number of softmax tokens, giving up to 5.4x prefill and 2.3x decode speedups over a transformer at 128k.
  • Retrieval robustness becomes a property of the hybrid: NAtS-L variants do not fail on the tasks where pure linear and pure softmax models each drop sharply, and NAtS-L Hybrid wins five of six retrieval benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The absence of any pure-softmax head in the learned solutions suggests that a simpler hand-designed schedule—softmax only in deeper layers, linear everywhere else—might capture much of NAtS-L's gain; the paper itself notes shallow layers are mostly linear and softmax heads concentrate in intermediate and deeper layers.
  • The router could be reused as a cheap inference-time sparsity predictor: if routing decisions are accurate, softmax KV cache entries for linear-routed chunks could be evicted or never written, reducing memory footprint even further than the reported speedups.
  • Expanding from two operations to a broader search space of linear attention variants is the natural next step; the same routing machinery would apply unchanged.
  • A direct way to test whether the router truly learns content-dependent routing, rather than relying on other design choices like decay or normalization, is to freeze all weights except the score layer and check whether routing assignments still adapt to different input distributions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Neural Attention Search Linear (NAtS-L), a token-level hybrid attention architecture in which chunks of a sequence are assigned either to a Gated DeltaNet linear-attention path or to a softmax-attention path by an input-dependent score layer. The authors give a unified chunk-wise formulation for both attention families, describe how the score layer is supposedly optimized, and evaluate NAtS-L and NAtS-L Hybrid at 380M/800M scale on language modeling, retrieval benchmarks, RULER, LongBench, length extrapolation at 65k context, and inference latency. They report substantial gains, in particular on retrieval tasks and on RULER beyond the 4k training context, and they make the code publicly available.

Significance. If the central routing claim is supported, this is a meaningful contribution: token-level rather than layer-level hybrid attention is a plausible route to combining linear-attention efficiency with softmax-attention retrievability, and the reported empirical gains—best average retrieval, strong length extrapolation, 5.4x prefill and 2.3x decode speedups over a transformer baseline—are notable. The paper also provides a useful unified description of the two attention families and ships code and experiments on public benchmarks, which strengthens reproducibility. However, the core novelty is the claim that the score layer learns the optimal operation per chunk, and that claim currently rests on an unspecified gradient path through a hard argmax assignment. Until that is resolved, the empirical results cannot be attributed to learned routing instead of the other architectural changes.

major comments (3)
  1. [§4.1–§4.2, Eqs. (8), (11)–(16)] The gradient path to the Attention Score Layer is not established. Equation (8) produces continuous scores, but Equations (11)–(12) turn them into a hard binary mask via argmax. The derivative of that mask with respect to score_t is zero almost everywhere, so Equations (14)–(16) do not actually provide dscore unless a straight-through estimator, Gumbel-softmax relaxation, or another estimator is explicitly introduced. The sentence 'we set dscore=0 if M_t=0' only addresses inactive chunks; it does not supply a nonzero path for active chunks. This is load-bearing because the paper's abstract and introduction claim that NAtS-L 'automatically determines' and 'searches for' optimal attention types. As written, the observed token-type distributions in Figure 5 could arise from an input-dependent but untrained W_score, with the gains coming from weighted mixing, decay, or the two complementary
  2. [§5.3, Tables 4–5] The ablation study is not attached to a stated model scale. The 'NAtS-L Hybrid' row in Table 4 is identical to the 380M-scale retrieval numbers in Table B.1/Table 5, not to the 800M-scale results in Table 1. The text introduces Section 5.3 without saying whether the ablations use the 380M or 800M setup. If the ablations are at 380M, that needs to be stated and the design conclusions scoped accordingly; if they are intended to support the 800M claims, the numbers are the wrong scale. In addition, the ablations do not include a fixed- or random-router baseline, so they cannot isolate whether the routing is actually learned.
  3. [§4.3, Eq. (19)] The paper applies the linear-attention decay alpha to hidden states even across softmax-attention chunks ('S[t+1] = prod ... if t /∈ t_la'). This is an ad hoc design choice that changes the semantics of the softmax-chunk boundary, yet it is only tested in one ablation (w/o LAttn Decay). If the routing gradient issue is fixed, this choice should be justified more thoroughly or framed as a hyperparameter, since it directly interacts with the meaning of a 'softmax chunk' in the hybrid state update.
minor comments (4)
  1. [General] There are several typos: 'sotmax' in §4.2, 'Lenght' in Figure 3 caption, 'NarratieQA' in §5.1, 'Weighs FromX' in Table 4, and 'Reserach Center' in the author affiliations. These should be corrected.
  2. [§A.1] The description of GDN Hybrid is inconsistent: the main text says '5 transformer layers and 17 GDN layers, 802M' and also describes a 3:1 ratio, while Appendix A.1 says '22 layers for the GDN Hybrid blocks with 5 transformer layers and 17 GDN layers.' Please clarify the exact layer count and the intended ratio.
  3. [Tables 1 and 5] The column header 'LMB.LMB.' appears to be a duplicated label. Also, the zero-shot average column includes both Wiki. and LMB. perplexities along with accuracies; averaging ppl and acc in one 'Avg.' is not standard and should be clarified or split.
  4. [References] Several references are placeholders (e.g., 'Proceedings of the 38th International Conference...'), and some citations have malformed trailing text such as 'NeurIPS’25)neu' and 'ICL (2020)'. The reference list needs to be cleaned and completed.

Circularity Check

0 steps flagged

No significant circularity: the NAtS-L router is learned from the language-modeling loss, and the headline claims rest on external benchmark measurements rather than on refitting the router's own scores.

full rationale

The claimed derivation chain (Eqs. 8, 11-12, 14-16, 19-20) defines a score layer W_score and hard attention masks chosen by argmax, but the 'predictions' that support NAtS-L — retrieval accuracies, RULER/LongBench scores, 65k-context perplexity, and latency — are measured after training on held-out external benchmarks. W_score is not fitted to, or defined in terms of, those benchmark outputs. Hence there is no self-definitional step and no fitted parameter renamed as a prediction. The only author self-citation (Deng & Lindauer 2025, NAtS) supplies the score-layer and column-wise-mask-gradient recipe; it is used as a technique citation, not as a load-bearing uniqueness theorem or as an ansatz that by itself forces the NAtS-L conclusions. The paper provides its own equations, ablations, and comparisons against independently implemented baselines. The hard-argmax gradient issue in Section 4.2 (Eqs. 14-16) is a real derivation gap — the derivative of the argmax mask is zero almost everywhere, and no straight-through estimator or relaxation is stated — but this is a correctness/omitted-proof concern, not circularity: it does not equate a predicted quantity to an input by construction. Conservatively, the circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. The main load-bearing assumptions are the hard-routing trainability (under-specified), the token-type dichotomy, and the correctness of borrowed chunkwise implementations. Free parameters are architectural choices that affect the central efficiency and performance claims.

free parameters (3)
  • NAtS chunk size C = not stated in main text; only constrained to be no smaller than the GDN chunk size
    Controls the granularity of routing and the efficiency-expressivity tradeoff; central to the claim but not specified in §4.1 or the appendix.
  • Hybrid layer ratio (6 NAtS-L + 15 GDN layers) = 6 NAtS-L layers out of 21
    Chosen by hand for the NAtS-L Hybrid variant; affects comparisons and is not searched or ablated.
  • Attention head grouping = 12/18 softmax heads grouped into 6 GDN heads
    Manual design choice in §4.3 / appendix; affects routing behavior and parameter count.
axioms (4)
  • domain assumption Chunks/tokens have a well-defined binary type: short-term-impact tokens are safe for fixed hidden-state linear attention, while long-term-retrieval tokens need softmax attention.
    Core premise of §4; if this dichotomy is false, learned routing cannot deliver the claimed expressivity-efficiency tradeoff.
  • ad hoc to paper The hard argmax routing in §4.1 can be optimized by the gradient formulas in §4.2 without a specified relaxation.
    Eqs. 14-16 compute gradients with respect to mask values, but the mask is defined as an argmax of scores; no Gumbel-softmax, straight-through estimator, or REINFORCE term is given.
  • standard math The chunkwise parallel forms of Gated DeltaNet and flash attention are correct and interchangeable with the recurrent/softmax forms.
    Relies on Yang et al. 2024/2025 and Dao et al. 2022; not re-derived in the paper.
  • ad hoc to paper Applying the linear-attention decay alpha to hidden states even across softmax-attention chunks (Eq. 19) is a beneficial design.
    This design choice is ablated, but the mechanism is not derived from first principles; it is presented as a practical heuristic.

pith-pipeline@v1.3.0-alltime-deepseek · 20244 in / 10710 out tokens · 117552 ms · 2026-08-03T04:51:12.939277+00:00 · methodology

0 comments
read the original abstract

The quadratic computational complexity of softmax transformers has become a bottleneck in long-context scenarios. In contrast, linear attention model families provide a promising direction towards a more efficient sequential model. These linear attention models compress past KV values into a single hidden state, thereby efficiently reducing complexity during both training and inference. However, their expressivity remains limited by the size of their hidden state. Previous work proposed interleaving softmax and linear attention layers to reduce computational complexity while preserving expressivity. Nevertheless, the efficiency of these models remains bottlenecked by their softmax attention layers. In this paper, we propose Neural Attention Search Linear (NAtS-L), a framework that applies both linear attention and softmax attention operations within the same layer on different tokens. NAtS-L automatically determines whether a token can be handled by a linear attention model, i.e., tokens that have only short-term impact and can be encoded into fixed-size hidden states, or require softmax attention, i.e., tokens that contain information related to long-term retrieval and need to be preserved for future queries. By searching for optimal Gated DeltaNet and softmax attention combinations across tokens, we show that NAtS-L provides a strong yet efficient token-level hybrid architecture.

Figures

Figures reproduced from arXiv: 2602.03681 by Andreas Bentzen Winje, Difan Deng, Lukas Fehring, Marius Lindauer.

Figure 1
Figure 1. Figure 1: A comparison between different attention modules. In 1(a), softmax attention needs to compute each QK-pair. In 1(b), linear attention models perform chunk-wise computation and there￾fore only maintain a fixed hidden state (the color darkness indicates hidden states from different time steps). 1(c), NAtS-L automati￾cally determines if a model belongs to softmax or linear attention and merge their output to … view at source ↗
Figure 2
Figure 2. Figure 2: The NAtS-L Architecture. The projection layers for q, k, v are a linear layer followed by a short conv and SiLU acti￾vation function. The scores for each operation are computed by a mean pooling layer followed by a linear layer (Equation 8). The model then selects the operations with the highest score for each chunk and feeds them to different attention operations. The outputs from each attention model are… view at source ↗
Figure 3
Figure 3. Figure 3: Per-token perplexity on different datasets, all the models are trained with 4096 tokens (the black vertical line) RULER (Hsieh et al., 2024) and LongBench (Bai et al., 2024). For the RULER benchmark, we test with input context lengths of 4k, 8k, and 16k on the retrieval tasks. Since all the models are trained only with a context length of 4096, this also tests whether they can extrapolate beyond that lengt… view at source ↗
Figure 4
Figure 4. Figure 4: illustrates the pre-filling and decoding time for [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Fraction of softmax attention tokens within each layer for NAtS-L Hybrid in different tasks. spite that, we can see an overall trend: several heads contain only linear attention, while the others are a mix of linear and softmax attention. Additionally, although some heads only contain linear attention operations, no head contains pure softmax attention operations (the head 0 in layer 4 is close, but it cou… view at source ↗
Figure 6
Figure 6. Figure 6: Per-token perplexity on different datasets on a smaller model size, all the models are trained with 4096 tokens (the black vertical line) [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Per-Task Ruler performance. (Left) RULER performance with 380M parameters. (Right) RULER results with 800M parameters We show the overall mean scores of the RULER benchmark in [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Token number distributions of NAtS-L on PG19, NarrativeQA, and CodeParrot dataset 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 24 linked inside Pith

  1. [2]

    Sim- ple linear attention language models balance the recall- throughput tradeoff.arXiv:2402.18668 [cs.CL],

    Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Zinsley, D., Zou, J., Rudra, A., and R ´e, C. Sim- ple linear attention language models balance the recall- throughput tradeoff.arXiv:2402.18668 [cs.CL],

    Pith/arXiv arXiv

  2. [5]

    As shown in Figure 8, the softmax attention token distributions generally follow a similar trend, with some of the heads might change their roles given the input context

    Here, we illustrate the task-wise experimental results. As shown in Figure 8, the softmax attention token distributions generally follow a similar trend, with some of the heads might change their roles given the input context. However, the shallower layers might contain more linear attention heads while the softmax attention heads might be located more in...

    arXiv

  3. [9]

    org/abs/2512.02556

    URL https://arxiv. org/abs/2512.02556. Deng, D. and Lindauer, M. Neural attention search. In Proceedings of the 38th International Conference on Advances in Neural Information Processing Systems (NeurIPS’25)neu (2025). Dong, X. and Yang, Y . Searching for a robust neural ar- chitecture in four gpu hours. InProceedings of the In- ternational Conference on ...

    Pith/arXiv arXiv 2025

  4. [10]

    Du, J., Sun, W., Lan, D., Hu, J., and Cheng, Y

    URL https://arxiv.org/abs/ 2411.13676. Du, J., Sun, W., Lan, D., Hu, J., and Cheng, Y . Mom: Linear sequence modeling with mixture-of-memories. arXiv:2502.13685 [cs.CL],

    Pith/arXiv arXiv

  5. [11]

    V ., Ye, H., Byeon, W., Karnati, Y ., Liebenwein, L., Khadkevich, M., Keller, A., Kautz, J., Lin, Y ., and Molchanov, P

    Fu, Y ., Dong, X., Diao, S., keirsbilck, M. V ., Ye, H., Byeon, W., Karnati, Y ., Liebenwein, L., Khadkevich, M., Keller, A., Kautz, J., Lin, Y ., and Molchanov, P. Nemotron- flash: Towards latency-optimal hybrid small language models. InProceedings of the 38th International Con- ference on Advances in Neural Information Processing Systems (NeurIPS’25)neu...

    2025

  6. [12]

    org/abs/2506.04761

    URL https://arxiv. org/abs/2506.04761. Hsieh, C., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y ., and Ginsburg, B. Ruler: What’s the real context size of your long-context language models? arXiv:2404.06654 [cs.CL],

    arXiv

  7. [13]

    Published on- line:iclr.cc

    ICLR. Published on- line:iclr.cc. Jiang, H., Li, Y ., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A., Li, D., Lin, C., Yang, Y ., and Qiu, L. Min- ference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. InProceedings of the 38th International Conference on Advances in Neural Infor- mation Processing Systems (NeurI...

    2024

  8. [14]

    Kimi linear: An expressive, efficient attention architecture.arXiv:2510.26692 [cs.LG],

    Kimi-Team. Kimi linear: An expressive, efficient attention architecture.arXiv:2510.26692 [cs.LG],

    Pith/arXiv arXiv

  9. [15]

    Reformer: The efficient transformer

    Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. InThe Eigth International Confer- ence on Learning Representations (ICLR’20)ICL (2020). Published online:iclr.cc. Kocisk´y, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K., Melis, G., and Grefenstette, E. The narrativeqa read- ing comprehension challenge.Transactions of the ...

    2020

  10. [16]

    Snapkv: LLM knows what you are looking for before generation

    Li, Y ., Huang, Y ., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: LLM knows what you are looking for before generation. arXiv:2404.14469 [cs.CL],

    Pith/arXiv arXiv

  11. [17]

    URL https://arxiv.org/abs/ 2503.24067. 10 Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y ., Shalev- Shwartz, S., Abend, O., Alon, R., Asida, T., Bergman, A., Glozman, R., Gokhman, M., Manevich, A., Ratner, N., Ro...

    arXiv

  12. [18]

    DARTS: Differentiable architecture search

    Liu, H., Simonyan, K., and Yang, Y . DARTS: Differentiable architecture search. InThe Seventh International Confer- ence on Learning Representations (ICLR’19)ICL (2019). Published online:iclr.cc. Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. arXiv:2310.01889 [cs:CL],

    Pith/arXiv arXiv 2019

  13. [19]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. InThe Seventh International Conference on Learning Representations (ICLR’19)ICL (2019). Pub- lished online:iclr.cc. Lozhkov, A., Allal, L. B., von Werra, L., and Wolf, T. Fineweb-edu: the finest collection of educational content,

    2019

  14. [20]

    URL https://huggingface.co/ datasets/HuggingFaceFW/fineweb-edu. Lu, E., Jiang, Z., Liu, J., Du, Y ., Jiang, T., Hong, C., Liu, S., He, W., Yuan, E., Wang, Y ., Huang, Z., Yuan, H., Xu, S., Xu, X., Lai, G., Chen, Y ., Zheng, H., Yan, J., Su, J., Wu, Y ., Zhang, N., Yang, Z., Zhou, X., Zhang, M., and Qiu, J. Moba: Mixture of block attention for long-context...

    Pith/arXiv arXiv

  15. [21]

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A

    URL https: //arxiv.org/abs/2505.23666. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv:1809.02789 [cs.CL],

    arXiv

  16. [22]

    org/abs/2501.08313

    URL https://arxiv. org/abs/2501.08313. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern´andez, R. The LAMBADA dataset: Word pre- diction requiring a broad discourse context. InPro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2016, August 7-12,...

    Pith/arXiv arXiv 2016

  17. [24]

    Mixture of sparse attention: Content-based learnable sparse attention via expert-choice routing.arXiv:2505.00315 [cs.LG],

    Piekos, P., Csord´as, R., and Schmidhuber, J. Mixture of sparse attention: Content-based learnable sparse attention via expert-choice routing.arXiv:2505.00315 [cs.LG],

    Pith/arXiv arXiv

  18. [25]

    Compressive transformers for long- range sequence modelling

    Rae, J., Potapenko, A., Jayakumar, S., Hillier, C., and Lillicrap, T. Compressive transformers for long- range sequence modelling. InThe Eigth International Conference on Learning Representations (ICLR’20) ICL (2020). URL https://openreview.net/ forum?id=SylKikSYDH. Published online: iclr. cc. Ren, L., Liu, Y ., Lu, Y ., Shen, Y ., Liang, C., and Chen, W....

    2020

  19. [26]

    Sakaguchi, K., Bras, R

    Association for the Advancement of Artificial Intelligence, AAAI Press. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y . Winogrande: An adversarial winograd schema challenge at scale. In Rossi et al. (2020), pp. 8732–8740. Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.).Proceedings of th...

    2020

  20. [27]

    Retentive network: A successor to transformer for large language models.arXiv:2307.08621 [cs.CL],

    Sun, Y ., Dong, L., Huang, S., Ma, S., Xia, Y ., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models.arXiv:2307.08621 [cs.CL],

    Pith/arXiv arXiv

  21. [28]

    LLaMA: Open and efficient foundation language models.arXiv:2302.13971 [cs.CL],

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozi `ere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. LLaMA: Open and efficient foundation language models.arXiv:2302.13971 [cs.CL],

    Pith/arXiv arXiv

  22. [30]

    Test-time regression: a unifying framework for designing sequence models with associative memory.arXiv:2501.12352 [cs.LG],

    Wang, K., Shi, J., and Fox, E. Test-time regression: a unifying framework for designing sequence models with associative memory.arXiv:2501.12352 [cs.LG],

    Pith/arXiv arXiv

  23. [31]

    and Hoff, M

    Widrow, B. and Hoff, M. Adaptive switching circuits,. In 1960 IRE WESCON Convention Record, New York: IRE

    1960

  24. [33]

    Duoattention: Efficient long- context llm inference with retrieval and streaming heads

    Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y ., and Han, S. Duoattention: Efficient long- context llm inference with retrieval and streaming heads. arXiv:2410.10819 [cs.CL],

    Pith/arXiv arXiv

  25. [34]

    Con- text parallelism for scalable million-token inference

    Yang, A., Yang, J., Ibrahim, A., Xie, X., Tang, B., Sizov, G., Reizenstein, J., Park, J., and Huang, J. Con- text parallelism for scalable million-token inference. arXiv:2411.01783 [cs.DC], 2024a. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, C., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H.,...

    Pith/arXiv arXiv

  26. [35]

    Yang, S., Wang, B., Shen, Y ., Panda, R., and Kim, Y

    URL https://github.com/ fla-org/flash-linear-attention. Yang, S., Wang, B., Shen, Y ., Panda, R., and Kim, Y . Gated linear attention transformers with hardware-efficient train- ing. In Salakhutdinov et al. (2024). Yang, S., Wang, B., Zhang, Y ., Shen, Y ., and Kim, Y . Par- allelizing linear transformers with the delta rule over se- quence length. InProc...

    Pith/arXiv arXiv 2024

  27. [36]

    Hellaswag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D. R., and M`arquez, L. (eds.),Proceedings of the 57th Conference of the Associa- tion for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4791–4800,

    2019

  28. [37]

    Zhang, Y

    URL https://arxiv.org/abs/2410.10254. Zhang, Y . and Yang, S. Flame: Flash language modeling made easy, January

    Pith/arXiv arXiv

  29. [38]

    Zhong, S., Xu, M., Ao, T., and Shi, G

    URL https: //openreview.net/forum?id=RkRrPp7GKO. Zhong, S., Xu, M., Ao, T., and Shi, G. Understanding transformer from the perspective of associative memory. arXiv:2505.19488 [cs.LG],

    Pith/arXiv arXiv

  30. [1024]

    All the GDN layers have 6 heads across different parameter scales

    This value increases to 1536 for models with 800M parameters. All the GDN layers have 6 heads across different parameter scales. However, for the other operations, the number of heads scales with the number of parameters: mamba2 has 32 and 48 heads, while the transformer has 16 and 24 heads, respectively. Finally, the NAtS-L layers have 12 and 18 softmax ...

    1956

  31. [1960]

    Ef- ficient streaming language models with attention sinks

    Xiao, G., Tian, Y ., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv:2309.17453 [cs.CL],

    Pith/arXiv arXiv

  32. [2016]

    Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., and et al

    URL https://doi.org/10.18653/v1/p16-1144. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., and et al. RWKV: reinventing rnns for the transformer era. In Bouamor, H., Pino, J., and Bali, K. (eds.),Findings of the Association for Computational Lin- guistics: EMNLP 2023, pp. 14048–14077. Association for Computational Linguistics,

    doi:10.18653/v1/p16-1144 2023

  33. [2017]

    Mesanet: Sequence modeling by locally optimal test-time training

    von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Schlegel, M., Maile, K., Schimpf, Y ., Sieberling, O., Meulemans, A., Saurous, R., Lajoie, G., Frenkel, C., Pascanu, R., Arcas, B., and Sacramento, J. Mesanet: Sequence modeling by locally optimal test-time training. arXiv:2506.05233 [cs.LG],

    Pith/arXiv arXiv

  34. [2018]

    and Gu, A

    Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Salakhutdinov et al. (2024). Dao, T., Fu, D., Ermon, S., Rudra, A., and R´e, C. FlashAt- tention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Process- ing Systems, 35:16344–16359,

    2024

  35. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457 [cs.AI],

    Pith/arXiv arXiv

  36. [2020]

    Generating long sequences with sparse transformers

    Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv:1904.10509 [cs.LG],

    Pith/arXiv arXiv 1904

  37. [2022]

    Deepseek-v3 technical report

    DeepSeek-AI. Deepseek-v3 technical report. arXiv:2412.19437 [cs-CL], 2024a. 9 Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models DeepSeek-AI. Deepseek-v2: A strong, economi- cal, and efficient mixture-of-experts language model. arXiv:2405.04434 [cs-CL], 2024b. DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large...

    Pith/arXiv arXiv

  38. [2023]

    Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and R ´e, C

    URL https://arxiv.org/abs/ 2305.13245. Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and R ´e, C. Zoology: Measur- ing and improving recall in efficient language models. arXiv:2312.04927 [cs.CL],

    Pith/arXiv arXiv

  39. [2024]

    L., Gao, J., and Choi, Y

    Bisk, Y ., Zellers, R., Bras, R. L., Gao, J., and Choi, Y . PIQA: reasoning about physical commonsense in natural language. In Rossi et al. (2020), pp. 7432–7439. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G., Henighan, T., Child, R., R...

    2020

  40. [2025]

    Bai, Y ., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y ., Tang, J., and Li, J

    URLhttps://arxiv.org/abs/2510.04800. Bai, Y ., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y ., Tang, J., and Li, J. LongBench: A bilingual, multitask benchmark for long context understanding. In Ku, L., Martins, A., and Srikumar, V . (eds.),Proceedings of the 62nd Annual Meet- ing of the Association for Comp...

    Pith/arXiv arXiv