Pith. sign in

REVIEW 1 major objections 6 minor 56 references

By replacing the low-rank query-key product with high-rank structured matrices, attention can overcome its low-rank bottleneck and add a distance-dependent compute bias, beating standard attention at fixed compute on high-dimensional regres

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 21:25 UTC pith:4XXAZL77

load-bearing objection A clean framework for structured attention scoring, but the compute-efficiency headline rests on FLOPs counts rather than wall-clock, and the LM scaling-law evidence is thinner than the abstract suggests. the 1 major comments →

arxiv 2509.07963 v2 pith:4XXAZL77 submitted 2025-09-09 cs.LG

Customizing the Inductive Biases of Softmax Attention using Structured Matrices

Yilun Kuang , Noah Amsel , Sanae Lotfi , Shikai Qiu , Andres Potapczynski , Andrew Gordon Wilson This is my paper
classification cs.LG MSC 68T07
keywords softmax attentionstructured matriceslow-rank bottleneckmulti-level low rankblock tensor-trainin-context learninglanguage model scalingtime-series forecasting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that two weaknesses of standard softmax attention live in its scoring function: the query-key dot product has rank at most the head dimension, which is far smaller than the embedding size and loses information needed for high-dimensional tasks; and the same function is used for every token pair, so no extra compute is spent on nearby tokens. It proposes replacing the low-rank product with Block Tensor-Train (BTT) and Multi-Level Low Rank (MLR) matrices, which are efficient, high-rank, and can encode either full-rank scoring or a hierarchical distance-dependent compute bias. On in-context linear regression with high-dimensional inputs, the structured scoring functions outperform standard attention at any fixed compute budget. On character-level language modeling, 8-level MLR attention achieves lower validation loss than standard attention, sliding-window attention, and hybrid global-plus-window attention at fixed compute, yielding better scaling laws. On time-series forecasting with Chronos and ETT data, MLR attention matches or improves accuracy with less compute, with gains growing at longer horizons.

Core claim

The central claim is that the attention scoring function, not just the attention architecture, carries two tunable inductive biases: rank and position-dependence. Standard multi-head attention computes scores as x^T W_Q W_K^T x', a bilinear form of rank equal to the head dimension r; when r is much smaller than the embedding dimension D, tasks with intrinsically high-dimensional inputs are provably hard to express. The paper shows that replacing this low-rank product with structured matrices removes that bottleneck: BTT matrices use O(D^{3/2}) parameters yet are full rank, and MLR matrices sum block-diagonal low-rank terms across scales, so they are high-rank while costing about as many FLOP

What carries the argument

The central object is the structured matrix inside the attention scoring function. Standard attention uses the low-rank matrix W_Q W_K^T (rank r, the head dimension). The paper substitutes Block Tensor-Train (BTT) matrices—factorizations with O(D^{3/2}) parameters that can be full rank—and Multi-Level Low Rank (MLR) matrices—sums of block-diagonal low-rank terms at progressively finer block sizes. These matrices do two jobs: they raise the rank of the score matrix, and, in MLR attention, they allocate compute by distance, because a token pair's score uses only the levels whose blocks contain both tokens. The paper also introduces MLBTC (Multi-Level Block Tensor Contraction) as a unifying fam

Load-bearing premise

The load-bearing premise is that FLOPs are the right efficiency currency: the authors' own footnote reports their structured implementations are slower in wall-clock time than standard attention, and the only wall-clock check is one in-context regression run, so the scaling-law and time-series advantages could shrink or vanish on real hardware.

What would settle it

Train the same 6-layer transformer on OpenWebText with standard attention and 8-level MLR attention (rank split 32|8|6|4|4|4|4|2) at width 768, and compare validation loss against wall-clock time on the same GPU instead of FLOPs. If standard attention reaches equal or lower loss at equal or less wall-clock time, the central efficiency claim fails. A complementary check: run the in-context regression comparison at input dimension 128, width 256, and report throughput and energy alongside FLOPs.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On in-context linear regression with high-dimensional inputs, Bilinear BTT and Bilinear MLR reach lower error than multi-head attention at the same FLOP count, with full-rank structures working at smaller model widths.
  • On character-level OpenWebText, 8-level MLR attention achieves lower validation loss than standard attention across widths 256-768 and also beats sliding-window and global-plus-sliding-window hybrids at fixed compute, implying a better scaling law.
  • MLR attention reduces key-cache size during autoregressive generation; with 8 levels and equal rank split the cache drops by about 4x.
  • Because BTT and MLR are special cases of the MLBTC family, the design space for scoring functions includes interpolations between full-rank global scoring and hierarchical local scoring.
  • On the Chronos time-series model and ETT data, MLR attention reaches comparable or better forecasting accuracy with fewer FLOPs, with gains growing at longer horizons.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's FLOP wins are measured in arithmetic, not runtime; a natural extension is to benchmark these scores on tensor-core hardware, where batch-parallel contractions could close the observed wall-clock gap.
  • The hierarchical block structure of MLR attention could be aligned to semantic units rather than fixed power-of-two blocks—paragraphs, files, or code scopes—turning the compute-bias knob into a data-structure knob.
  • The same structured replacement could be applied to the value-output projection inside each head, or with different structures per head, to distribute labor across heads; the paper only sketches this direction.
  • MLBTC's parameterization suggests a search problem: choose the number of levels, block sizes, and rank splits to match a dataset's locality spectrum, which the paper does not explore.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper proposes replacing the low-rank attention scoring matrix W_Q W_K^T with structured matrix families: Block Tensor Train (BTT) and Multi-Level Low Rank (MLR) matrices, and introduces MLBTC as a unifying framework. It applies high-rank bilinear forms to in-context linear regression to overcome the low-rank bottleneck, and applies MLR with hierarchical block structure to encode a distance-dependent compute bias in language modeling and time-series forecasting. The authors report that BTT/MLR attention outperform standard attention at fixed FLOPs on in-context regression, and that MLR attention achieves improved scaling laws on OpenWebText compared with standard and sliding-window attention.

Significance. The conceptual contribution is timely: it identifies the attention scoring matrix as a locus for customizing inductive biases and provides concrete structured parameterizations with clear FLOP analyses. The tensor-contraction tables in Appendix D are useful, and the derivations connecting MLR to a distance-dependent score matrix are transparent. The code release is also a strength. However, the current evidence is not yet fully load-bearing: the main compute-controlled comparisons are measured in idealized FLOPs rather than wall-clock time, the headline plots lack error bars, and the central LM result depends on a single hand-picked rank allocation. If these gaps are closed, the work could make a solid contribution; as it stands, the empirical claims outrun the validation.

major comments (1)
  1. [Section 5.1, Figure 4; Section 5.2, Figure 5; Appendix G] The abstract and Section 4 claim that the proposed methods "outperform standard attention for any fixed compute budget." The experiments cover a limited range of model widths, input dimensions, and compute budgets; the statement as written is broader than the evidence. Please qualify the claim to the tested settings (or provide additional sweeps) so that the conclusion matches the scope of the experiments.
minor comments (6)
  1. [Section 3.2] Typo: "creates a information bottleneck" should be "creates an information bottleneck."
  2. [Section 3.3] The statement that BTT "can approximate arbitrary dense matrices" and thus MLBTC "can also express any D x D matrices" mixes approximation and exact expressiveness. If the intended meaning is exact representation, please state it precisely and cite the corresponding parameter regime from Qiu et al.
  3. [Section 3.4] The distance function d(j,j') is introduced for the two-level case and then used in Equation (9) for the general L-level case. The general definition should be given explicitly for L levels, since the example only defines d=1 and d=2.
  4. [Section 5.2 / Appendix I] Figure 15(a) reports bars for horizons 96, 192, and 336, but the text says improvements are observed for "both horizons 96 and 336." Please describe all three horizons and include uncertainty information.
  5. [References] There are duplicate entries for Behrouz et al. (2024a and 2024b) with the same arXiv identifier arXiv:2501.00663. Please consolidate or correct.
  6. [Appendix F] Figure 6 is cited as evidence of stable learning-rate transfer, but it uses a reduced context length of 256. This detail should be mentioned when the figure is referenced in the main text.

Circularity Check

0 steps flagged

No significant circularity; central claims are definitional arithmetic plus external empirical comparisons.

full rationale

The paper's derivation chain does not contain a step in which a prediction reduces to a fitted input or to a definitional equivalent. The structured scoring functions in Eqs. (6) and (7) are direct instantiations of the MLR and BTT definitions in Eqs. (4) and (5); the FLOPs counts in Table 1 and Appendix D are arithmetic consequences of those definitions, not assumed conclusions. The empirical claims are tested against external baselines (standard attention, sliding-window attention, and Chronos-style time-series models) with matched head dimension and model width, and no parameter is fit to the target metric and then reported as a prediction. The paper does rely on prior work by overlapping authors for two supporting facts: the BTT universality result from Qiu et al. (2024) and the rank-bottleneck theorem from Amsel et al. (2024). These are self-citations, but they are not load-bearing in a circular sense: they are stated as external theorems with assumptions that do not include the present paper's results, and the rank-bottleneck phenomenon is independently reproduced in the paper's own Figure 2. The MLBTC construction in Section 3.3 is a definitional generalization of MLR and BTT, not a derived prediction, and the equivalence between the scoring-function form and the matrix form is proved in Lemma C.1 rather than assumed. Footnote 2 in Section 4 concedes that the structured implementations are 'somewhat slower in wall-clock time than standard attention'; this is a practical validity limitation of the FLOPs-based compute control, not a circularity, because the FLOPs axis is not equated to validation loss by construction. Overall, the paper is self-contained against external benchmarks and its reported advantages follow from the proposed definitions plus empirical evaluation.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claims rest on standard linear algebra, an imported BTT expressivity theorem from the authors' prior work (Qiu et al. 2024), a domain assumption about block locality, muP transfer, and the FLOPs-as-compute metric. The MLR rank allocations and number of levels are hand-chosen hyperparameters that the reported gains depend on. No new physical or empirical entities are postulated; MLBTC is a definitional umbrella, not a conjectured object.

free parameters (4)
  • MLR per-level rank allocation r_1|...|r_L = 32|8|6|4|4|4|4|2 (LM); 32|10|8|8|4|2, 48|16, 40|16|4|4 (time series)
    Hand-chosen (Section 3.4, Section 5.1). The LM result uses one allocation with total rank 64; no sensitivity analysis is provided.
  • Number of MLR levels L = 8 (LM), 6 (Chronos), 2 and 4 (ETT)
    Hyperparameter chosen per task; paper states L<=8 and p_l=2^{l-1}.
  • BTT internal rank s = 1 or 2
    Section 3.1: 'For efficiency, we set s=1 or s=2'; this trades expressivity against cost.
  • QK LayerNorm constants C and C* = tunable (not specified)
    Section E.2: 'for some tunable constant C' and 'C*'; these normalization constants affect training stability and final loss.
axioms (5)
  • standard math A BTT matrix with a=b=c=d=s=sqrt(D) can express any D x D matrix (Qiu et al. 2024).
    Invoked in Section 3.3 to argue MLBTC/BTT can be full-rank; imported from authors' prior work.
  • standard math A rank-r matrix W_Q W_K^T with r << D cannot represent the inner-product scoring function x^T x'.
    Linear algebra used in Section 2.2 to define the low-rank bottleneck.
  • domain assumption Text and time-series data exhibit block-local structure aligned with fixed powers-of-two partitions.
    Motivates MLR attention in Sections 2.3 and 3.4; if locality is not block-structured, the fixed partition is suboptimal.
  • domain assumption muP learning-rate and initialization rules transfer to the structured attention variants.
    Used in Section 3.5 and Appendix E; LM comparisons assume both models are optimally tuned. Figure 6 tests transfer only at context length 256.
  • domain assumption FLOPs excluding embedding projections is a fair compute metric for comparing attention variants.
    Used for all compute-controlled plots; Footnote 2 concedes the implementations are slower in wall-clock time.

pith-pipeline@v1.3.0-alltime-deepseek · 23939 in / 19363 out tokens · 176526 ms · 2026-08-04T21:25:27.240724+00:00 · methodology

0 comments
read the original abstract

The core component of attention is the scoring function, which transforms the inputs into low-dimensional queries and keys and takes the dot product of each pair. While the low-dimensional projection improves efficiency, it causes information loss for certain tasks that have intrinsically high-dimensional inputs. Additionally, attention uses the same scoring function for all input pairs, without imposing a distance-dependent compute bias for neighboring tokens in the sequence. In this work, we address these shortcomings by proposing new scoring functions based on computationally efficient structured matrices with high ranks, including Block Tensor-Train (BTT) and contiguous Multi-Level Low Rank (MLR) matrices. On in-context regression tasks with high-dimensional inputs, our proposed scoring functions outperform standard attention for any fixed compute budget. On language modeling, a task that exhibits locality patterns, our MLR-based attention method achieves improved scaling laws compared to both standard attention and variants of sliding window attention. Additionally, we show that both BTT and MLR fall under a broader family of efficient structured matrices capable of encoding either full-rank or distance-dependent compute biases, thereby addressing significant shortcomings of standard attention. Finally, we show that MLR attention has promising results for long-range time-series forecasting.

Figures

Figures reproduced from arXiv: 2509.07963 by Andres Potapczynski, Andrew Gordon Wilson, Noah Amsel, Sanae Lotfi, Shikai Qiu, Yilun Kuang.

Figure 1
Figure 1. Figure 1: Overview of two ways to customize the inductive biases of softmax attention with structured matrices. (a) Standard attention computes the dot product between a query and a key via a low-rank bilinear transformation. In this work, we replace the low rank product with other structured matrices such as Multi-Level Low Rank (MLR) and Block Tensor Train (BTT) as introduced in Section 3.2. (b) Standard attention… view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Both Bilinear BTT and Bilinear MLR outperform standard attention for in-context regression tasks. (a) Bilinear BTT and 1-head attention are full rank, so they can learn the task using a smaller model width (x-axis) than 8-head attention. Here model width refers to the embedding dimension D. All points on this figure correspond to models that was trained for the same number of steps. (b) For a fixed model w… view at source ↗
Figure 4
Figure 4. Figure 4: MLR Attention achieves lower validation loss on OpenWebText compared to standard multi-head attention and variants of sliding window attention when controlling for compute. (a) We plot the validation loss of both MLR attention with 8 levels and standard attention against compute. We vary over 4 values of model width D ∈ {256, 384, 512, 768}. MLR attention outperforms standard attention across model widths.… view at source ↗
Figure 5
Figure 5. Figure 5: MLR attention reduces computational cost when com￾pared to standard attention. We plot the validation loss of three distinct model sizes when substituting the T5 attention mechanism in Chronos (Ansari et al., 2024) from standard attention to MLR attention. Here GA stands for global attention, which is just the standard attention mechanism. As shown in the figure, MLR atten￾tion achieves slightly lower vali… view at source ↗
Figure 6
Figure 6. Figure 6: We show the validation loss of an 8-Level MLR attention and standard attention on OpenWebText across a variety of learning rates. Here we sweep over model widths D ∈ {512, 768, 1024} with a reduced context length of 256 due to compute constraints. As the figure shows, our MLR attention shares the same optimal learning rate across model width, and it’s also consistently better than standard attention when b… view at source ↗
Figure 7
Figure 7. Figure 7: The performance of standard multi-head attention as a function of the rank ratio. We take the runs from [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: We show in-context regression performances as functions of training steps and training compute (excluding the input and output linear projection layers) across model width D ∈ {16, 32, 64, 128, 256, 512} for dinput = 16. In general, Bilinear BTT outperforms attention with varying number of heads. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: We show in-context regression performances as functions of training steps and training compute (excluding the input and output linear projection layers) across model width D ∈ {16, 32, 64, 128, 256, 512} for dinput = 32. In general, Bilinear BTT outperforms attention with varying number of heads. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: We show in-context regression performances as functions of training steps and training compute (excluding the input and output linear projection layers) across model width D ∈ {16, 32, 64, 128, 256, 512} for dinput = 64. In general, Bilinear BTT outperforms attention with varying number of heads. Bilinear MLR also outperforms Bilinear BTT when D = 256 or 512. The different Bilinear MLR lines correspond to… view at source ↗
Figure 11
Figure 11. Figure 11: We show in-context regression performances as functions of training steps and training compute (excluding the input and output linear projection layers) across model width D ∈ {16, 32, 64, 128, 256, 512} for dinput = 128. In general, Bilinear BTT outperforms attention with varying number of heads. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Bilinear BTT and 1-head attention are full rank structures and thus achieve lower regression errors (y-axis) compared to 8-head attention at smaller model width (x-axis). Each one of the four figures correspond to one particular input dimension dinput. 0 25 50 75 100 125 150 175 200 Wall Time (minutes) 10−2 10−1 Squared Error (log scale) Structure Attention (8 Head) BilinearBTT (a) Loss as a function wall… view at source ↗
Figure 13
Figure 13. Figure 13: Wall time comparisons between bilinear BTT and standard attention for input dimension at dinput = 16 and model width at D = 64 and number of attention heads H = 8. We omit the 1 head attention since it has wall time very close to 8 head attention. (a) we show the squared regression error in log scale (y-axis) against the wall time between Bilinear BTT and standard attention. We observe that Bilinear BTT s… view at source ↗
Figure 14
Figure 14. Figure 14: MLR attention outperforms both purely sliding window attention (SWA) and combinations of sliding window attention and standard attention with global context across model widths. SWA refers to all layers being sliding window attention. Global + SWA means sequentially stacking standard attention and SWA. 96 192 336 Horizon −0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Relative Improvement in MAE (%) (a) 2 Levels MLR Att… view at source ↗
Figure 15
Figure 15. Figure 15: As the time horizon (i.e. sequence length) becomes larger, MLR attention outperforms standard attention in oil temperature prediction accuracy by around 1%. (a) We show the relative improvement in Mean Absolute Error (MAE) of a 2-levels MLR attention with rank distributions 48|16. For both horizons 96 and 336, MLR attention achieves better oil temperature forecasting accuracy. (b) We show that a 4-levels … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 19 canonical work pages · 5 internal anchors

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. URL https://arxiv.org/abs/2305.13245

    Pith/arXiv arXiv 2023

  3. [3]

    On the Benefits of Rank in Attention Layers

    Amsel, N., Yehudai, G., and Bruna, J. On the benefits of rank in attention layers, 2024. URL https://arxiv.org/abs/2407.16153

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S

    Ansari, A. F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P., Kapoor, S., Zschiegner, J., Maddix, D. C., Wang, H., Mahoney, M. W., Torkkola, K., Wilson, A. G., Bohlke-Schneider, M., and Wang, Y. Chronos: Learning the Language of Time Series . arXiv:2403.07815, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Simple linear attention language models balance the recall-throughput tradeoff, 2024

    Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Zinsley, D., Zou, J., Rudra, A., and Ré, C. Simple linear attention language models balance the recall-throughput tradeoff, 2024. URL https://arxiv.org/abs/2402.18668

    Pith/arXiv arXiv 2024

  6. [6]

    Neural machine translation by jointly learning to align and translate, 2016

    Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate, 2016. URL https://arxiv.org/abs/1409.0473

    Pith/arXiv arXiv 2016

  7. [8]

    Titans: Learning to memorize at test time, 2024 b

    Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning to memorize at test time, 2024 b . URL https://arxiv.org/abs/2501.00663

    Pith/arXiv arXiv 2024

  8. [9]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer, 2020. URL https://arxiv.org/abs/2004.05150

    Pith/arXiv arXiv 2020

  9. [10]

    S., Reddi, S., and Kumar, S

    Bhojanapalli, S., Yun, C., Rawat, A. S., Reddi, S., and Kumar, S. Low-rank bottleneck in multi-head attention models. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020

    work page 2020

  10. [11]

    A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M

    Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., G...

    Pith/arXiv arXiv 2022

  11. [12]

    Scatterbrain: Unifying sparse and low-rank attention

    Chen, B., Dao, T., Winsor, E., Song, Z., Rudra, A., and R\' e , C. Scatterbrain: Unifying sparse and low-rank attention. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 17413--17426. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper...

    work page 2021

  12. [13]

    Generating long sequences with sparse transformers, 2019

    Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers, 2019. URL https://arxiv.org/abs/1904.10509

    Pith/arXiv arXiv 2019

  13. [14]

    Rethinking attention with performers, 2022

    Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., Belanger, D., Colwell, L., and Weller, A. Rethinking attention with performers, 2022. URL https://arxiv.org/abs/2009.14794

    Pith/arXiv arXiv 2022

  14. [15]

    and Gu, A

    Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL https://arxiv.org/abs/2405.21060

    Pith/arXiv arXiv 2024

  15. [16]

    Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations

    Dao, T., Gu, A., Eichhorn, M., Rudra, A., and Ré, C. Learning fast algorithms for linear transforms using butterfly factorizations, 2020. URL https://arxiv.org/abs/1903.05895

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  16. [17]

    Monarch: Expressive Structured Matrices for Efficient and Accurate Training

    Dao, T., Chen, B., Sohoni, N., Desai, A., Poli, M., Grogan, J., Liu, A., Rao, A., Rudra, A., and Ré, C. Monarch: Expressive structured matrices for efficient and accurate training, 2022. URL https://arxiv.org/abs/2204.00595

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  17. [18]

    Y., Dao, T., Saab, K

    Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Ré, C. Hungry hungry hippos: Towards language modeling with state space models, 2023. URL https://arxiv.org/abs/2212.14052

    Pith/arXiv arXiv 2023

  18. [19]

    S., and Valiant, G

    Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? a case study of simple function classes. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 30583--30598. Curran Associates, Inc., 2022. URL https://arxiv.org/abs/2208.01066

    Pith/arXiv arXiv 2022

  19. [20]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

    Pith/arXiv arXiv 2024

  20. [21]

    Efficiently modeling long sequences with structured state spaces, 2022

    Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces, 2022. URL https://arxiv.org/abs/2111.00396

    Pith/arXiv arXiv 2022

  21. [22]

    SLT rain: a sparse plus low rank approach for parameter and memory efficient pretraining

    Han, A., Li, J., Huang, W., Hong, M., Takeda, A., Jawanpuria, P., and Mishra, B. SLT rain: a sparse plus low rank approach for parameter and memory efficient pretraining. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=MXze4H7opg

    work page 2024

  22. [23]

    Global context vision transformers

    Hatamizadeh, A., Yin, H., Heinrich, G., Kautz, J., and Molchanov, P. Global context vision transformers. In International Conference on Machine Learning, pp.\ 12633--12646. PMLR, 2023

    work page 2023

  23. [24]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  24. [25]

    Advancing transformer architecture in long-context large language models: A comprehensive survey

    Huang, Y., Xu, J., Lai, J., Jiang, Z., Chen, T., Li, Z., Yao, Y., Ma, X., Yang, L., Chen, H., et al. Advancing transformer architecture in long-context large language models: A comprehensive survey. arXiv preprint arXiv:2311.12351, 2023

    Pith/arXiv arXiv 2023

  25. [26]

    Hydra: Bidirectional state space models through generalized matrix mixers, 2024

    Hwang, S., Lahoti, A., Dao, T., and Gu, A. Hydra: Bidirectional state space models through generalized matrix mixers, 2024. URL https://arxiv.org/abs/2407.09941

    Pith/arXiv arXiv 2024

  26. [27]

    M., and Malach, E

    Jelassi, S., Brandfonbrener, D., Kakade, S. M., and Malach, E. Repeat after me: transformers are better than state space models at copying. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    work page 2024

  27. [28]

    Transformers are rnns: Fast autoregressive transformers with linear attention, 2020

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020. URL https://arxiv.org/abs/2006.16236

    Pith/arXiv arXiv 2020

  28. [29]

    Towards Understanding Inductive Bias in Transformers: A View From Infinity

    Lavie, I., Gur-Ari, G., and Ringel, Z. Towards understanding inductive bias in transformers: A view from infinity, 2024. URL https://arxiv.org/abs/2402.05173

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  29. [30]

    E., and Stoica, I

    Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., and Stoica, I. Tune: A research platform for distributed model selection and training, 2018. URL https://arxiv.org/abs/1807.05118

    Pith/arXiv arXiv 2018

  30. [31]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    Pith/arXiv arXiv 2019

  31. [32]

    Lotfi, S., Finzi, M., Kuang, Y., Rudner, T. G. J., Goldblum, M., and Wilson, A. G. Non-vacuous generalization bounds for large language models, 2024 a . URL https://arxiv.org/abs/2312.17173

    Pith/arXiv arXiv 2024

  32. [33]

    Lotfi, S., Kuang, Y., Amos, B., Goldblum, M., Finzi, M., and Wilson, A. G. Unlocking tokens as data points for generalization bounds on larger language models, 2024 b . URL https://arxiv.org/abs/2407.18158

    Pith/arXiv arXiv 2024

  33. [34]

    Luong, T., Pham, H., and Manning, C. D. Effective approaches to attention-based neural machine translation. In M \`a rquez, L., Callison-Burch, C., and Su, J. (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.\ 1412--1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi:10.1...

    doi:10.18653/v1/d15-1166 2015

  34. [35]

    G., Challú, C., Garza, A., Canseco, M

    Olivares, K. G., Challú, C., Garza, A., Canseco, M. M., and Dubrawski, A. NeuralForecast : User friendly state-of-the-art neural forecasting models. PyCon Salt Lake City, Utah, US 2022, 2022. URL https://github.com/Nixtla/neuralforecast

    work page 2022

  35. [36]

    Factor fitting, rank allocation, and partitioning in multilevel low rank matrices, 2023

    Parshakova, T., Hastie, T., Darve, E., and Boyd, S. Factor fitting, rank allocation, and partitioning in multilevel low rank matrices, 2023. URL https://arxiv.org/abs/2310.19214

    work page arXiv 2023

  36. [37]

    Fitting Multilevel Factor Models

    Parshakova, T., Hastie, T., and Boyd, S. Fitting multilevel factor models, 2024. URL https://arxiv.org/abs/2409.12067

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  37. [38]

    Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C

    Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C. Hyena hierarchy: Towards larger convolutional language models, 2023. URL https://arxiv.org/abs/2302.10866

    Pith/arXiv arXiv 2023

  38. [39]

    D., and Wilson, A

    Potapczynski, A., Qiu, S., Finzi, M., Ferri, C., Chen, Z., Goldblum, M., Bruss, B., Sa, C. D., and Wilson, A. G. Searching for efficient linear layers over a continuous space of structured matrices, 2024. URL https://arxiv.org/abs/2410.02117

    Pith/arXiv arXiv 2024

  39. [40]

    Qiu, S., Potapczynski, A., Finzi, M., Goldblum, M., and Wilson, A. G. Compute better spent: Replacing dense layers with structured matrices, 2024. URL https://arxiv.org/abs/2406.06248

    Pith/arXiv arXiv 2024

  40. [41]

    Combiner: full attention transformer with sparse computation cost

    Ren, H., Dai, H., Dai, Z., Yang, M., Leskovec, J., Schuurmans, D., and Dai, B. Combiner: full attention transformer with sparse computation cost. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393

    work page 2021

  41. [42]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al

    Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv. org/abs/2408.00118, 1 0 (3), 2024

    Pith/arXiv arXiv 2024

  42. [43]

    Representational strengths and limitations of transformers

    Sanford, C., Hsu, D., and Telgarsky, M. Representational strengths and limitations of transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=36DxONZ9bA

    work page 2023

  43. [44]

    A., Choromanski, K

    Sehanobish, A., Dubey, K. A., Choromanski, K. M., Chowdhury, S. B. R., Jain, D., Sindhwani, V., and Chaturvedi, S. Structured unrestricted-rank matrices for parameter efficient finetuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=MXOzgjlWDF

    work page 2024

  44. [45]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    Pith/arXiv arXiv 2023

  45. [46]

    T., Gu, A., Dao, T., Rudra, A., and R\' e , C

    Thomas, A. T., Gu, A., Dao, T., Rudra, A., and R\' e , C. Learning compressed transforms with low displacement rank. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp.\ 9066–9078, Red Hook, NY, USA, 2018. Curran Associates Inc

    work page 2018

  46. [47]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proc...

    2017

  47. [48]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference

    Warner, B., Chaffin, A., Clavi \'e , B., Weller, O., Hallstr \"o m, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13663, 2024

    Pith/arXiv arXiv 2024

  48. [49]

    Building on efficient foundations: Effective training of LLM s with structured feedforward layers

    Wei, X., Moalla, S., Pascanu, R., and Gulcehre, C. Building on efficient foundations: Effective training of LLM s with structured feedforward layers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=WxLVYZbIew

    work page 2024

  49. [50]

    Mswa: Refining local attention with multi-scalewindow attention

    Xu, Y., Nag, S., Li, D., Tian, L., and Barsoum, E. Mswa: Refining local attention with multi-scalewindow attention. arXiv preprint arXiv:2501.01039, 2025

    Pith/arXiv arXiv 2025

  50. [51]

    J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J

    Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022. URL https://arxiv.org/abs/2203.03466

    Pith/arXiv arXiv 2022

  51. [52]

    B., and Bernstein, J

    Yang, G., Simon, J. B., and Bernstein, J. A spectral condition for feature learning, 2024. URL https://arxiv.org/abs/2310.17813

    Pith/arXiv arXiv 2024

  52. [53]

    Bp-transformer: Modelling long-range context via binary partitioning

    Ye, Z., Guo, Q., Gan, Q., Qiu, X., and Zhang, Z. Bp-transformer: Modelling long-range context via binary partitioning. arXiv preprint arXiv:1911.04070, 2019

    Pith/arXiv arXiv 1911

  53. [54]

    X., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W

    Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y. X., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W. Native sparse attention: Hardware-aligned and natively trainable sparse attention, 2025. URL https://arxiv.org/abs/2502.11089

    Pith/arXiv arXiv 2025

  54. [55]

    The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry

    Zhang, M., Bhatia, K., Kumbong, H., and Re, C. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=4g02l2N2Nx

    work page 2024

  55. [56]

    Informer: Beyond efficient transformer for long sequence time-series forecasting, 2021

    Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting, 2021. URL https://arxiv.org/abs/2012.07436

    Pith/arXiv arXiv 2021

  56. [57]

    and Soricut, R

    Zhu, Z. and Soricut, R. H -transformer-1 D : Fast one-dimensional hierarchical attention for sequences. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.\ 3801--...

    work page doi:10.18653/v1/2021.acl-long.294 2021