Quantifying Memory Utilization with Effective State-Size

Alessandro Moro; Armin W. Thomas; Atsushi Yamashita; Michael Poli; Neehal Tumma; Qi An; Rom N. Parnichkun; Stefano Massaroli; Taiji Suzuki

REVIEW 3 major objections 5 minor 79 references

Quantifying Memory Utilization with Effective State-Size

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A rank-based metric measures how much memory a sequence model actually uses, not just how much it could.

desk verdict A useful, well-packaged memory-utilization metric whose LTI theory is solid and whose LIV interpretation rests on strong but fragile correlations; needs a revision that hardens the empirical bridge. read the letter →

arxiv 2504.19561 v1 pith:CX7N4YGB submitted 2025-04-28 cs.LG

Rom N. Parnichkun , Neehal Tumma , Armin W. Thomas , Alessandro Moro , Qi An , Taiji Suzuki , Atsushi Yamashita , Michael Poli

show 1 more author

Stefano Massaroli

This is my paper · ORCID

classification cs.LG

keywords effectivestate-sizememoryutilizationsequencemodelsminimalrealizationlinearinput-varyingsystemsassociativerecallmodeldistillationstatecollapse

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces effective state-size (ESS), a single number computed from a model's realized linear operator, and argues that ESS measures how much of a sequence model's state is actually engaged for a given input. Motivated by classical minimal realization theory, the authors show that for linear systems the rank of a certain operator submatrix equals the minimal state size of an equivalent recurrence. For the larger class of input-varying linear systems (which include attention, gated recurrences, and convolutions), they argue the same quantity serves as a lower bound on the required state and, empirically, as a much stronger proxy for task performance than cache size. They demonstrate that across associative recall, selective copying, and compression, task-normalized ESS correlates with accuracy far better than the theoretically realizable state-size (TSS), and they use ESS to expose two training failure modes, guide distillation and regularization, and quantify how language models reset their memory at delimiter tokens. The paper's central claim is that what matters for memory is not how large the state is, but how much of it the model actually uses, and that ESS is the right way to measure that.

What carries the argument

The central object is the strictly lower-triangular operator submatrix $\mathbf{H}_i = \mathbf{T}_{i:,:i-1}$ of the causal operator $\mathbf{T}$ realized by the model. The paper proves that $\mathrm{rank}(\mathbf{H}_i)$ is the minimal state size of any recurrent realization of an input-invariant linear system, and then computes two rank-smoothed versions of this quantity: tolerance-ESS, which counts singular values above a threshold $\tau$, and entropy-ESS, which exponentiates the spectral entropy of $\mathbf{H}_i$. The paper also defines theoretically realizable state-size (TSS) as the model-recurrent state dimension, which upper-bounds ESS and plays the role of a 'capacity' baseline throughout the comparisons.

What would settle it

Find a realistic input-varying sequence model for which ESS decreases or stays flat over training while recall accuracy rises substantially, and where a competing capacity-based measure explains the accuracy differences better across a held-out task-model sweep. Concretely: train a family of gated linear attention models on MQAR, sweep kv and TSS as in the paper, and compute the split-half correlation of length-normalized ESS/kv against accuracy; if TSS/kv or a simple attention-focused spectral statistic wins or ties across the randomized halves, the claim that ESS outperforms capacity as a memory-utilization proxy would fail.

Watch

Extended reading notes

Core claim

For a causal sequence model materialized as a linear operator $\mathbf{T}$, the effective state-size at sequence index $i$ is the rank of the submatrix $\mathbf{H}_i \equiv \mathbf{T}_{i:,:i-1}$, which collects the strictly causal part of the operator up to time $i$. By minimal realization theory, this rank exactly equals the minimal state size of any recurrence realizing the same input-invariant operator (Theorem 3.2); for input-varying linear systems it is a lower bound, requiring an empirical bridge. The paper's principal empirical discovery is that task-adjusted ESS ($\mathrm{ESS}/kv$ for associative recall) correlates with post-training accuracy across a sweep of four featurizer families (gated, weighted, and plain linear attention, plus softmax attention) substantially better than the corresponding cache-size measure $\mathrm{TSS}/kv$. It also identifies two failure modes: state saturation, where ESS is pinned near TSS because the state is too small; and state collapse, where a sufficiently large state is left largely unused. On language models, ESS exhibits dips when the model encounters end-of-speech tokens, a phenomenon the authors call state modulation, which they connect to recall performance on bigram tasks and to cross-architectural differences in handling context boundaries.

Load-bearing premise

That the numerical rank of a single forward pass's operator submatrix $\mathbf{H}_i$ tracks how much trainable memory a model actually engages when the model is input-varying, since the formal minimal-state-size theorem only covers input-invariant systems and LIVs are supported by an empirical correlation bridge.

Editorial extensions

If this is right

ESS can serve as a diagnostic before and during training: measuring ESS at initialization can reveal whether an architecture is preconditioned to use all its states, as demonstrated by the poor ESS-TSS scaling of S6 at initialization.
ESS can guide model-order reduction: a teacher layer with high ESS leads to larger distillation loss, so ESS is a predictor of compressibility that can steer the choice of student state sizes.
ESS can be targeted by regularizers: pushing gated and weighted linear attention recurrences toward the identity, which increases ESS, cures state collapse and lets them outperform plain linear attention on recall tasks.
ESS is a quantitative tool for studying language models: the dip in ESS at separator tokens (state modulation) can be used to compare architectures, and the hierarchy SA>GLA>WLA>LA in state modulation mirrors their bigram recall performance.
ESS reconciles an apparent paradox in efficient sequence models: a model can have a large cache yet realize low memory utilization, which explains why capacity metrics alone mispredict recall behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

ESS could be adapted to probe the per-layer or per-head distribution of memory use in very large models, potentially identifying which layers are redundant before any training run.
The state modulation measurement suggests a testable design principle for hybrid architectures: place layers with unbounded state capacity after layers that need to reset, which is consistent with why attention-first hybrids fail in the paper's experiments.
The task-normalization step (dividing by the number of key-value pairs) is a place where a deeper theory could be built: if ESS were grounded in a task-dependent information requirement, the same metric could be extended to natural-language recall without hand-picking the normalizer.
Entropy-ESS and tolerance-ESS disagree on state collapse, which suggests that a spectrally graded version of ESS could sharpen failure-mode diagnosis beyond the paper's binary categorization.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper gives the community a genuinely useful measurement: ESS, the rank of the operator submatrix Hi, packaged with tolerance and entropy variants. For LTI systems the theory is classical minimal realization and they cite it properly. What's new is the packaging, the empirical validation on synthetic recall tasks, and the downstream applications—distillation, initialization diagnostics, regularization, and the state-modulation finding in LLMs. The code in the appendix is a real asset.

The empirical case is solid but needs a careful read. Every model in the validation (GLA, WLA, LA, SA) is an LIV, and for LIVs the paper honestly concedes that the minimal realization may not preserve causality, so ESS is only a lower bound on the state size of any causal recurrence. That means the whole interpretation—that ESS tracks usable memory—rests on correlation evidence. The correlations are genuine but fragile: they require the kv normalization the authors call critical, they drop epoch 0, they filter SA models with accuracy above 0.95, and they report no confidence intervals. The choice between entropy- and tolerance-ESS is made per analysis, which is a bit ad hoc. These are the soft spots, and they are fixable. The stress-test note is right: there is no direct test that ESS equals usable state size for an LIV.

Credit where due: the paper is unusually transparent about its analysis choices and doesn't hide the limitations of the theory. The state modulation result is a real observation—attention models and gated SSMs respond differently to separator tokens, and that connects nicely to bigram recall. The initialization analysis of S6 is a nice diagnostic.

Who should read it: anyone building or analyzing linear-attention, Mamba-style, or hybrid architectures. It will be cited.

Recommendation: send to peer review. The referee process should push for uncertainty quantification, robustness of the correlations to the disclosed filters, and ideally a direct test of the causal realization claim. The paper deserves that effort.

Referee Report

3 major / 5 minor

Summary. This paper introduces effective state-size (ESS), a per-sequence-index measure of memory utilization computed as the rank (or its tolerance- or entropy-smoothed variant) of the submatrix H_i = T_{i:,:i-1} of the materialized causal operator T = f_T(u). The authors prove (Theorem 3.2, Appendix D) that for input-invariant linear systems, rank(H_i) is the minimal state size of any recurrent realization, and they argue that for input-varying linear operators (LIVs) it lower-bounds the state size. They validate ESS by training GLA/LA/WLA/SA models on MQAR, selective copying, and compression, showing that ESS/kv correlates with accuracy better than TSS/kv (Section 4.1), that ESS tracks within-task performance over training (Section 4.2), and that failure modes (state saturation, state collapse) are visible in ESS/TSS. Applications include ESS-informed featurizer analysis of S6 at initialization, an A-to-I regularizer, model-order reduction via distillation, and the observation of 'state modulation' in LLM states at separator tokens.

Significance. The central theoretical contribution—equating rank(H_i) with the minimal recurrent state size for input-invariant operators—is a clean application of classical minimal realization theory, and the proof sketches in D.2.4 and D.3 are plausible. The entropy-ESS default is parameter-free, the PyTorch implementation is shipped in E.1.3, and the analysis choices that affect the headline correlations (kv normalization, the accuracy>0.95 filter for SA, the epoch-0 drop) are disclosed in E.3 rather than hidden; this transparency is a genuine strength. If the empirical bridge for LIVs holds, ESS would be a useful, cheap diagnostic: it exposes that S6's state initialization under-utilizes added capacity (Section A) and that GLA/WLA suffer state collapse that an A-to-I regularizer can fix (Section B)—both falsifiable claims. The main risk is that every model in the validation set is an LIV, for which ESS is only a lower bound on causal state size, so the memory-utilization interpretation rests on the correlation evidence alone.

major comments (3)

[§4.1 and §E.3] Section 4.1 states that 'measured over the entire task-model space, ESS/kv exhibits a significantly higher correlation with accuracy than TSS/kv,' but Section E.3 discloses three choices that materially shape this result: SA configurations with accuracy above 0.95 are filtered from the cross-task correlation, epoch 0 is dropped from the within-task correlations, and the kv normalization is called 'critical for observing the cross task-model correlations' (unnormalized ESS correlations 'break down,' per Figure 10g). Since this is the paper's principal validation of ESS for LIVs, the Section 4.1 claim should be re-scoped to the non-saturated task-model configurations, and the authors should report the number of points entering each correlation plus robustness checks such as unfiltered SA, alternative normalizations, or a partial correlation controlling for kv. The epoch-0 exclusion likewise deserves reporting alongside Figure 3, since Figure 25 shows a phase in which ESS decreases while accuracy is flat, which contradicts the 'ESS tracks performance' reading at initialization.
[§3 'Interpreting effective state-size'; §D.2.2] The paper concedes that for LIVs the minimal realization of the materialized operator may be acausal, so ESS is only a lower bound on the state size required by a causal recurrence; the 'proxy for memory utilization' interpretation for LIVs is then justified only by the correlation evidence of Section 4. A direct test is available but not performed: Section D.2.2 defines the 'causally determinable minimal state-size (causal ESS)' as the rank of the causal projection [A_{i-1;1}B_0 ... A_{i-1}B_{i-2} B_{i-1}], which is computable from the trained features of GLA/LA/WLA and is exactly the state dimension a causal recurrence actually engages. Computing causal ESS on the sweep and comparing it with ESS (e.g., a scatter plot or the ratio ESS/causal-ESS) would quantify the acausality gap and either support or bound the memory-utilization interpretation. Without this comparison, the central claim cannot be distinguished from the weaker claim that ESS predicts performance.
[§4.1, Figure 4] The headline comparison 'ESS/kv exhibits a significantly higher correlation with accuracy than TSS/kv' is reported without confidence intervals, standard errors, or a significance test, despite the large scatter in Figure 2 and the non-independence of the points (they are bucketed by featurizer, TSS, kv, and sequence length, with ESS averaged over only 8 samples). A bootstrap over task-model configurations, or at least a report of the sample size and a correlation-difference test, is needed to support the word 'significantly'; as presented, part of the gap in Figure 4 may reflect the discreteness of TSS (a hyperparameter taking a handful of values) rather than the greater informativeness of ESS.

minor comments (5)

[§3.1 'Computational complexity of ESS'] The complexity is stated as O((dℓ)^3) per layer, but the per-sequence-index definition requires an SVD of H_i at every i, giving O(d · Σ_i (ℓ−i)·i·min(ℓ−i,i)) = O(dℓ^4) per layer if all indices are computed; please specify whether the stated bound refers to a single index and describe the protocol used for the (total ESS)_i plots in Section 5.2.
[References] The reference list contains stray page annotations such as '(pages 1, 19)' after URLs (e.g., Abnar & Zuidema, Akaike, and most other entries); these appear to be leftover bibliography artifacts and should be removed.
[Figure 6 / §4.2] Figure 6 and the state-saturation discussion use tolerance-ESS at tolerance 1e−3 (per Section E.3), but the main-text caption and prose do not state this; please add the tolerance to the figure caption.
[§A, Eq. (A.0.1)] Equation (A.0.1) writes the GLA gate as diag(sigmoid(Wu)^{1/β}), which is inconsistent with Eq. (E.2.6) in Section E.2, diag(sigmoid(W_{A2}W_{A1}u_i)^{1/β}); the two formulations should be aligned.
[§5.2 and §E.8] The 'preemptive state modulation' explanation attributes to SSMs a learned reliance on explicit EOS signals, but the evidence is descriptive ESS trajectories on a small fixed set of sentences; a control condition (e.g., non-semantic separator tokens) or a more cautious framing would strengthen the causal reading.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ESS is defined from the materialized operator independently of task accuracy, and the paper's correlations are empirical findings rather than reductions.

full rationale

The paper computes ESS as a rank functional of the operator submatrix Hi = Ti:,:i−1, with the entropy-ESS variant having no fitted constants and with no reference to the accuracies it later correlates with. The main validation claim, 'Measured over the entire task-model space, ESS/kv exhibits a significantly higher correlation with accuracy than TSS/kv', is an empirical correlation, not a derived equivalence. The theoretical result (Theorem 3.2) is a minimal-realization argument from classical linear system theory, and the proof rests on the rank-nullity theorem applied to factorizations of Hi, not on the paper's own conclusions. For LIVs, the paper explicitly concedes that the minimal realization process may not preserve causality and that ESS only lower-bounds the state size; the transition from lower bound to memory-utilization proxy is therefore an empirically tested interpretation, not a definitional tautology. The paper also discloses the sensitivity of the cross-task correlations to kv normalization, the SA accuracy filter, and the exclusion of epoch 0; these are analysis choices that bear on robustness and validity, not fitted parameters renamed as predictions. Citations to the authors' earlier MAD framework provide reproducible task definitions, and the core rank-to-state-size relation is not imported from those citations. Residual concerns about the LIV gap and the strength of the validation are correctness risks, not circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 2 invented entities

The core theoretical input is classical realization theory, purchased upstream; the paper's own contributions carry one domain assumption (the LIV materialization) and one empirical bridge (rank of a single input's operator lower-bounds and proxies state usage for LIVs). The kv normalization, filtering thresholds, and metric-variant choices are analysis parameters selected post hoc and are the main hand-set degrees of freedom in the empirical claims.

free parameters (6)

kv normalization (ESS/kv) = number of key-value pairs in the task
The cross-task correlation that supports the central claim is destroyed without dividing ESS by kv; the paper states this normalization is critical (Section E.3, Figure 10), so it is an analysis parameter chosen to make the correlation visible.
SA accuracy filter threshold = 0.95
Models with accuracy above 0.95 are removed from the SA cross-model correlations to observe meaningful variation (Section E.3); this post-hoc exclusion affects the headline ESS-accuracy comparison.
epoch-0 exclusion = drop epoch 0
Within-model ESS-accuracy correlations drop epoch 0 because ESS decreases while accuracy is flat at training start (Section E.3); the exclusion changes the reported within-task correlations.
tolerance values (tolerance-ESS) = 1e-1, 1e-3, 1e-5 and 1e+0 across analyses
Tolerance-ESS requires a manually selected cutoff; the paper uses different values for state utilization (1e-3) and LLM state modulation (1e-1 to 1e+0) depending on which reveals the trend.
entropy-ESS clip = 1e-12
Numerical stability clip on normalized singular values (Section E.1.3); minor but a hand-set constant in the default metric.
GLA-S6 arange normalization alpha = 1000 (default), swept upward
In the initialization analysis, alpha is scaled up to raise initialization ESS and improve MQAR accuracy (Section A), i.e., a hyperparameter tuned to the target result.

assumptions (6)

domain assumption Most sequence models of interest can be faithfully expressed as y = Tu with input-invariant or input-varying linear operators (LIVs).
Section 3 and E.2: the entire framework, and thus ESS, applies only to models that materialize a linear operator; non-linear RNNs are outside scope.
standard math Classical minimal realization theory: the rank of the operator submatrix equals the minimal state dimension of a causal realization for input-invariant operators.
Theorems 3.1 and 3.2 rely on DeWilde and van der Veen ch. 3; the proof is sketched in Section D.3 and is standard.
domain assumption For LIVs, ESS only lower-bounds the state size because minimal realizations may be non-causal; the proxy interpretation is empirical.
Section 3 'Interpreting effective state-size': the paper states the minimal realization may depend on future inputs, so the memory-utilization reading is an empirical claim validated only by correlation.
standard math Eckart-Young-Mirsky theorem justifies tolerance-ESS as the minimum state size for an approximation with spectral error below tolerance.
Section 3.1 uses this theorem to interpret tolerance-ESS.
domain assumption Performance on MAD synthetic tasks (MQAR, selective copying, compression) approximates performance on large-scale language tasks.
Section 4 and C: the synthetic tasks come from Poli et al. 2024 (same research lineage); the generalization to language is assumed and only partially tested via bigram recall.
domain assumption The four 1B models (LA, WLA, GLA, SA) are comparable because they are trained under identical conditions.
Section 5.2 and Table 9: comparability underpins the state-modulation hierarchy; the training dataset is not specified in the provided text.

invented entities (2)

State modulation
purpose: Describes the observed input-dependent variation of ESS, especially dips at separator/EOS tokens, used to explain cross-architectural recall differences (Section 5.2).
Newly named phenomenon; the predicted hierarchy SA > GLA > WLA > LA is tested within the paper against bigram recall perplexity, but no external replication is cited.
Preemptive state modulation
purpose: Mechanistic explanation for why limited-cache SSMs like Falcon Mamba show strong EOS-triggered ESS dips (Section F.5).
A postulated strategy attributed to SSMs; supported only by qualitative patterns on a small set of open-weight models.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Quantifying Memory Utilization with Effective State-Size." pith.science (2026). https://pith.science/paper/CX7N4YGB

@misc{pith2026250419561,
  author       = {Pith},
  title        = {Pith review of: Quantifying Memory Utilization with Effective State-Size},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CX7N4YGB}},
  note         = {Machine review of arXiv:2504.19561}
}

read the original abstract

The need to develop a general framework for architecture analysis is becoming increasingly important, given the expanding design space of sequence models. To this end, we draw insights from classical signal processing and control theory, to develop a quantitative measure of \textit{memory utilization}: the internal mechanisms through which a model stores past information to produce future outputs. This metric, which we call \textbf{\textit{effective state-size}} (ESS), is tailored to the fundamental class of systems with \textit{input-invariant} and \textit{input-varying linear operators}, encompassing a variety of computational units such as variants of attention, convolutions, and recurrences. Unlike prior work on memory utilization, which either relies on raw operator visualizations (e.g. attention maps), or simply the total \textit{memory capacity} (i.e. cache size) of a model, our metrics provide highly interpretable and actionable measurements. In particular, we show how ESS can be leveraged to improve initialization strategies, inform novel regularizers and advance the performance-efficiency frontier through model distillation. Furthermore, we demonstrate that the effect of context delimiters (such as end-of-speech tokens) on ESS highlights cross-architectural differences in how large language models utilize their available memory to recall information. Overall, we find that ESS provides valuable insights into the dynamics that dictate memory utilization, enabling the design of more efficient and effective sequence models.

Figures

Figures reproduced from arXiv: 2504.19561 by the authors.

**Figure 1.** An overview of the effective state-size metric and its various downstream applications. alone often fail to highlight critical properties that explain how different models construct internal representations of the input data. Moreover, prior attempts in obtaining quantitative metrics, such as through spectral analysis of the operator T (Min & Li, 2024; Bhojanapalli et al., 2020), are either limited to a specific mod… view at source ↗

**Figure 2.** Scatter plots of accuracy vs ESS/kv across featurizers. Within each featurizer plot, all task-model configurations from the sweep corresponding to each featurizer are shown. where p m i = σm i ∥σi∥1 . Entropy-ESS is particularly useful for summarizing metrics across the entire tolerance space, whereas tolerance-ESS offers a more precise and readily interpretable depiction of rank concerning approximation error. Unle… view at source ↗

**Figure 3.** Correlation between ESS and accuracy over the course of model training bucketed by TSS and kv. Task space. To explore ESS in an extensive, yet controlled, manner, we iterate on a set of synthetic tasks proposed by Poli et al. (2024) which have been shown to effectively approximate model performance on large-scale language tasks. Specifically, we train models on the multi-query associative recall (MQAR), selective co… view at source ↗

Figures from the paper (42 more)

**Figure 4.** Figure 4: ESS/kv vs TSS/kv as a proxy for model performance as measured by correlation. Our first goal is to understand how ESS empirically captures memory utilization by studying its correlation with posttraining MQAR performance across the entire task-model space. To appropri…

**Figure 5.** Figure 5: ESS-distillation loss (activation) correlation. it by the memory demands of MQAR, constructing an adjusted form of ESS given by ESS/kv. Measured over the entire task-model space, ESS/kv exhibits a significantly higher correlation with accuracy than TSS/kv (Figures 2, 4…

**Figure 6.** Figure 6: Accuracy and state utilization as a function of kv for low and high TSS models. State saturation refers to the scenario in which a model has insufficient TSS to fully learn a task, resulting in its ESS converging near its TSS. This is reflected in its ESS/TSS (which w…

**Figure 7.** Figure 7: (a) The effect of separator tokens over Falcon Mamba 7B. See Section F.5 for plots of other open-weight models. (b) Comparison of standard perplexity and bigram recall perplexity (Arora et al., 2023). space, increasing ESS is a generally viable approach to improving m…

**Figure 8.** Figure 8: (a) ESS-TSS scaling in the S6, GLA and GLA-S6 featurizers. (b) ESS and accuracy on MQAR as a function of TSS in GLA. (c) ESS and accuracy on MQAR as a function of normalization factor for initialization in GLA-S6. Initialization in weight space plays a crucial role in …

**Figure 9.** Figure 9: (a) ESS/kv and ∥ Q i Ai∥F as a function of sequence length. (b) Accuracy of models as a function of ESS-based regularizer strength. To motivate the idea of increasing ESS mid-training, we revisit the concept of state collapse – a phenomenon that arises due to trainabil…

**Figure 10.** Figure 10: (a) TSS/kv vs accuracy across featurizers. This demonstrates that TSS/kv (i.e. memory capacity) is a worse proxy for model performance than ESS/kv as discussed in Section 4. (b) (total TSS)/kv vs accuracy across featurizers. This demonstrates that (total TSS)/kv is a …

**Figure 11.** Figure 11: MQAR accuracies marginalized across different dimensions. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_11.png]

**Figure 12.** Figure 12: MQAR ESS/kv marginalized across different dimensions. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_12.png]

**Figure 13.** Figure 13: MQAR (total ESS)/kv marginalized across different dimensions. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_13.png]

**Figure 14.** Figure 14: MQAR ESS/TSS marginalized across different dimensions. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_14.png]

**Figure 15.** Figure 15: MQAR (total ESS)/(total TSS) marginalized across different dimensions. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_15.png]

**Figure 16.** Figure 16: MQAR ESS-accuracy correlations computed over training marginalized across different dimensions. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_16.png]

**Figure 17.** Figure 17: Accuracy vs various forms of tolerance-ESS across task-model space. Plots are entirely analogous to those shown in [PITH_FULL_IMAGE:figures/full_fig_p041_17.png]

**Figure 18.** Figure 18: MQAR ESS/kv marginalized across different dimensions. 42 [PITH_FULL_IMAGE:figures/full_fig_p042_18.png]

**Figure 19.** Figure 19: MQAR (total ESS)/kv marginalized across different dimensions. 43 [PITH_FULL_IMAGE:figures/full_fig_p043_19.png]

**Figure 20.** Figure 20: MQAR ESS/TSS marginalized across different dimensions. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_20.png]

**Figure 21.** Figure 21: MQAR (total ESS)/(total TSS) marginalized across different dimensions. 45 [PITH_FULL_IMAGE:figures/full_fig_p045_21.png]

**Figure 22.** Figure 22: MQAR ESS-accuracy correlations computed over training marginalized across different dimensions. 46 [PITH_FULL_IMAGE:figures/full_fig_p046_22.png]

**Figure 23.** Figure 23: Selective copying results. Note that ESS here refers to entropy-ESS and we abbreviate num. tokens to copy as ntc in plots above. (a) ESS/ntc vs accuracy across featurizers. (b) (total ESS)/ntc vs accuracy across featurizers. (c) TSS/ntc vs accuracy across featurizers.…

**Figure 24.** Figure 24: Compression results. Note that ESS here refers to entropy-ESS and we abbreviate vocab size as vs in plots above. (a) ESS/vs vs accuracy across featurizers. (b) (total ESS)/vs vs accuracy across featurizers. (c) TSS/vs vs accuracy across featurizers. (d) (total TSS)/vs…

**Figure 25.** Figure 25: Training dynamics of ESS in select models (dmodel=256, heads=8) trained on MQAR (seqlen=2048, kv=64). We min-max normalize the ESS curves over the course of training to emphasize the shape of the curve as opposed to its magnitude. Note that the tolerance-ESS shown her…

**Figure 26.** Figure 26: Loss curves of S6 and GLA-S6 showing that the models are unable to improve beyond random guessing on MQAR, across various state-sizes. 2 5 2 7 TSS 2 4 2 5 ESS GLA ESS 0.00 0.25 0.50 0.75 Accuracy Acc. (Mean) Acc. (Min-Max) [PITH_FULL_IMAGE:figures/full_fig_p049_26.png]

**Figure 27.** Figure 27: ESS and MQAR accuracy as a function of TSS on a custom task regime (sequence length = 1024, num. kv pairs = 256). This figure illustrates a strong correlation between MQAR accuracy, ESS and TSS. 49 [PITH_FULL_IMAGE:figures/full_fig_p049_27.png]

**Figure 28.** Figure 28: An example of the training dynamics of ESS in select models (dmodel=512, heads=4) trained on MQAR (seqlen=2048, kv=128) that undergo state collapse (i.e. GLA and WLA). We min-max normalize the ESS curves over the course of training to emphasize the shape of the curve …

**Figure 29.** Figure 29: Per-layer ESS/kv as a function of MQAR sequence length for the GLA and WLA featurizers. ESS shown here is computed using a tolerance of 1e-3. Layers are 0-indexed. Using 0-indexing for the layers, [PITH_FULL_IMAGE:figures/full_fig_p050_29.png]

**Figure 30.** Figure 30: This figure compares MQAR accuracy and ESS across reduction scales for layers 0 and 1. The lower ESS in layer 0 of the teacher model leads to better downstream performance after distillation compared to distilling layer 1. 2 1 2 3 2 5 ESS (teacher) 2 2 2 4 2 6 ESS (st…

**Figure 31.** Figure 31: Teacher ESS vs distilled student ESS. As expected, we observe a clear trend: an increase in the student TSS results in the student’s ESS more closely matching the teacher’s ESS. Plots like these can help provide additional context during the distillation process. 51 …

**Figure 32.** Figure 32: All results presented here are computed using tolerance-based ESS with a tolerance set at 1e-1. Network layers are 0-indexed. (a) Per-layer ESS of all possible 4-layer GLA-SA hybrid networks. Experimental settings can be found in Section E.7. (b) Per-layer ESS of all …

**Figure 33.** Figure 33: (c) Model accuracy and max/average ESS of SA layers in the 4-layer GLA-SA hybrid networks. (d) Model accuracy and max/average ESS of GLA layers in the 4-layer GLA-SA hybrid networks. Recall in Section 5, we demonstrated an application of post-training ESS analysis thr…

**Figure 34.** Figure 34: M ang oes are rich in vitamin C and can be blended into a refreshing smooth ie<sep> Gir aff es are the tall est mammals on Earth due to their long ne cks and legs <sep> She collects vintage type writers from the 1940 s<sep> Jupiter ’s Great Red Spot is a giant storm t…

**Figure 35.** Figure 35: 54 [PITH_FULL_IMAGE:figures/full_fig_p054_35.png]

**Figure 36.** Figure 36: M ango es are rich in vit amin C and can be bl ended into a ref res hing smooth ie<sep> G ira ff es are the tall est m amm als onEarth due to their long ne cks and legs <sep> She collect s v int age type writ ers from the 1 9 4 0 s<sep> Jup iter ’ sGreat Red Sp ot is …

**Figure 37.** Figure 37: <bos> M ang oes are rich in vit amin C and can be bl ended into a refres hing smooth ie<sep> G ir aff es are the tall est mamm als onEarth due to their long neck s and legs <sep> She collect svintage type writ ers from the 1 9 4 0 s<sep> Jupiter ’ sGreat Red Spot is a…

**Figure 38.** Figure 38: 55 [PITH_FULL_IMAGE:figures/full_fig_p055_38.png]

**Figure 39.** Figure 39: <bos> M ango es are rich in vitamin C and can be blended into a refreshing smooth ie<sep> Gir aff es are the tallest mammals on Earth due to their long neck s and legs <sep> She collects vintage typ ew riters from the 194 0 s<sep> Jupiter ’sGreat Red Spot is a giant s…

**Figure 40.** Figure 40: The figures above reveal significant cross-architectural differences in context processing. The attention-based model at a similar 7B scale ( [PITH_FULL_IMAGE:figures/full_fig_p056_40.png]

**Figure 41.** Figure 41: An illustration of the effect of different separator tokens over different layers across different tolerances. Softmax attention exhibits the most pronounced state modulation, beginning at a tolerance level of 1e−2, followed by gated linear attention with significant …

**Figure 42.** Figure 42: Effective state-size over a quake fast inverse square root algorithm’s code. Here ESS is summed across channels and layers. 1 #include <iostream> 2 #include <cmath> 3 4 // Quake Fast Inverse Square Root function 5 float quakeFastInvSqrt(float number) { 6 long i; 7 flo…

**Figure 43.** Figure 43: Effective state-size over a Fibonacci sequence generator algorithm’s code. Here ESS is summed across channels and layers. 1 #include <iostream> 2 int fibonacci(int n) { 3 if (n <= 1) { 4 return n; 5 } 6 return fibonacci(n - 1) + fibonacci(n - 2); // Recursive case 7 }…

**Figure 44.** Figure 44: The variation in effective state-size with a varying number of shots (2.7B Attention). Here ESS is summed across channels and layers. Eg. 1 Eg. 2 Eg. 3 Eg. 4 Eg. 5 Q&A 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Total Effective State Size (tol=0.001) ×106 Mamba-2.8B on MMLU (Elem…

**Figure 45.** Figure 45: The variation in effective state-size with a varying number of shots (2.8B State-Space Model). Here ESS is summed across channels and layers. 60 [PITH_FULL_IMAGE:figures/full_fig_p060_45.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 17 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

and Zuidema, W

Abnar, S. and Zuidema, W. Quantifying attention flow in transformers, 2020. URL https://arxiv.org/abs/2005.00928

arXiv 2020
[3]

Stochastic theory of minimal realization

Akaike, H. Stochastic theory of minimal realization. IEEE Transactions on Automatic Control, 19 0 (6): 0 667--674, 1974. doi:10.1109/TAC.1974.1100707

arXiv 1974
[4]

In-context language learning: Architectures and algorithms, 2024

Akyürek, E., Wang, B., Kim, Y., and Andreas, J. In-context language learning: Architectures and algorithms, 2024. URL https://arxiv.org/abs/2401.12973

arXiv 2024
[5]

The hidden attention of mamba models, 2024

Ali, A., Zimerman, I., and Wolf, L. The hidden attention of mamba models, 2024. URL https://arxiv.org/abs/2403.01590

arXiv 2024
[6]

and Li, Y

Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.3, knowledge capacity scaling laws, 2024. URL https://arxiv.org/abs/2404.05405

arXiv 2024
[7]

A., Alnumay, Y., Alrashed, S., Alsubaie, S., Almushaykeh, Y., Mirza, F., Alotaibi, N., Altwairesh, N., Alowisheq, A., Bari, M

Alzahrani, N., Alyahya, H. A., Alnumay, Y., Alrashed, S., Alsubaie, S., Almushaykeh, Y., Mirza, F., Alotaibi, N., Altwairesh, N., Alowisheq, A., Bari, M. S., and Khan, H. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards, 2024. URL https://arxiv.org/abs/2402.01781

arXiv 2024
[8]

Zoology: Measuring and improving recall in efficient language models, 2023

Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and Ré, C. Zoology: Measuring and improving recall in efficient language models, 2023. URL https://arxiv.org/abs/2312.04927

arXiv 2023

Show all 79 references

[9]

Simple linear attention language models balance the recall-throughput tradeoff, 2024

Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Zinsley, D., Zou, J., Rudra, A., and Ré, C. Simple linear attention language models balance the recall-throughput tradeoff, 2024. URL https://arxiv.org/abs/2402.18668

2024 arXiv
[10]

Z., and Ionescu, C

Ba, J., Hinton, G., Mnih, V., Leibo, J. Z., and Ionescu, C. Using fast weights to attend to the recent past, 2016. URL https://arxiv.org/abs/1610.06258

2016 arXiv
[11]

Beliczynski, B., Kale, I., and Cain, G. D. Approximation of fir by iir digital filters: An algorithm based on balanced model reduction. IEEE Transactions on Signal Processing, 40 0 (3): 0 532--542, 1992

1992
[12]

Understanding in-context learning in transformers and llms by learning to learn discrete functions, 2023

Bhattamishra, S., Patel, A., Blunsom, P., and Kanade, V. Understanding in-context learning in transformers and llms by learning to learn discrete functions, 2023. URL https://arxiv.org/abs/2310.03016

2023 arXiv
[13]

S., Reddi, S

Bhojanapalli, S., Yun, C., Rawat, A. S., Reddi, S. J., and Kumar, S. Low-rank bottleneck in multi-head attention models, 2020. URL https://arxiv.org/abs/2002.07028

2020 arXiv
[14]

Y., Xing, E

Bick, A., Li, K. Y., Xing, E. P., Kolter, J. Z., and Gu, A. Transformers to ssms: Distilling quadratic knowledge to subquadratic models, 2024. URL https://arxiv.org/abs/2408.10189

2024 arXiv
[15]

Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., Schubert, L., Voss, C., Egan, B., and Lim, S. K. Thread: Circuits. Distill, 2020. doi:10.23915/distill.00024. https://distill.pub/2020/circuits

2020 doi
[16]

Linear System Theory and Design

Chen, C.-T. Linear System Theory and Design. Oxford University Press, Inc., USA, 3rd edition, 1998. ISBN 0195117778

1998
[17]

and Gu, A

Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL https://arxiv.org/abs/2405.21060

2024 arXiv
[18]

L., Fernando, A., Botev, A., Cristian-Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., Desjardins, G., Doucet, A., Budden, D., Teh, Y

De, S., Smith, S. L., Fernando, A., Botev, A., Cristian-Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., Desjardins, G., Doucet, A., Budden, D., Teh, Y. W., Pascanu, R., Freitas, N. D., and Gulcehre, C. Griffin: Mixing gated linear recurrences with local ...

2024 arXiv
[19]

and van der Veen, A

DeWilde, P. and van der Veen, A. Time-Varying Systems and Computations. Springer US, 1998. ISBN 9780792381891. URL https://books.google.co.jp/books?id=n3bEniJ2Wx8C

1998
[20]

Attention is not all you need: Pure attention loses rank doubly exponentially with depth, 2023

Dong, Y., Cordonnier, J.-B., and Loukas, A. Attention is not all you need: Pure attention loses rank doubly exponentially with depth, 2023. URL https://arxiv.org/abs/2103.03404

2023 arXiv
[21]

Dupont, E., Doucet, A., and Teh, Y. W. Augmented neural odes, 2019. URL https://arxiv.org/abs/1904.01681

2019 arXiv
[22]

A mathematical framework for transformer circuits

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan...

2021
[23]

Y., Dao, T., Saab, K

Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Ré, C. Hungry hungry hippos: Towards language modeling with state space models, 2023. URL https://arxiv.org/abs/2212.14052

2023 arXiv
[24]

and Juang, J.-N

Gawronski, W. and Juang, J.-N. Model reduction in limited time and frequency intervals. International Journal of Systems Science, 21 0 (2): 0 349--376, 1990. doi:10.1080/00207729008910366. URL https://doi.org/10.1080/00207729008910366

1990 doi
[25]

Zamba: A compact 7b ssm hybrid model, 2024

Glorioso, P., Anthony, Q., Tokpanov, Y., Whittington, J., Pilault, J., Ibrahim, A., and Millidge, B. Zamba: A compact 7b ssm hybrid model, 2024. URL https://arxiv.org/abs/2405.16712

2024 arXiv
[26]

and Bengio, Y

Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Teh, Y. W. and Titterington, M. (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine ...

2010
[27]

and Dao, T

Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

2024 arXiv
[28]

Efficiently modeling long sequences with structured state spaces, 2022 a

Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces, 2022 a . URL https://arxiv.org/abs/2111.00396

2022 arXiv
[29]

On the parameterization and initialization of diagonal state space models, 2022 b

Gu, A., Gupta, A., Goel, K., and Ré, C. On the parameterization and initialization of diagonal state space models, 2022 b . URL https://arxiv.org/abs/2206.11893

2022 arXiv
[30]

Liquid structural state-space models, 2022

Hasani, R., Lechner, M., Wang, T.-H., Chahine, M., Amini, A., and Rus, D. Liquid structural state-space models, 2022. URL https://arxiv.org/abs/2209.12951

2022 arXiv
[31]

Measuring massive multitask language understanding, 2021

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

2021 arXiv
[32]

Ho, B. L. and Kalman, R. E. Effective construction of linear state-variable models from input/output functions. at - Automatisierungstechnik, 14 0 (1-12): 0 545--548, 1966. doi:doi:10.1524/auto.1966.14.112.545

1966 doi
[33]

and Schmidhuber, J

Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Comput., 9 0 (8): 0 1735–1780, nov 1997. ISSN 0899-7667. doi:10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735

1997 doi
[34]

A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vin...

2022 arXiv
[35]

Transformers are rnns: Fast autoregressive transformers with linear attention, 2020

Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020. URL https://arxiv.org/abs/2006.16236

2020 arXiv
[36]

Kiranyaz, S., Avci, O., Abdeljaber, O., Ince, T., Gabbouj, M., and Inman, D. J. 1d convolutional neural networks and applications: A survey, 2019. URL https://arxiv.org/abs/1905.03554

2019 arXiv
[37]

Jamba: A hybrid transformer-mamba language model, 2024

Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev-Shwartz, S., Abend, O., Alon, R., Asida, T., Bergman, A., Glozman, R., Gokhman, M., Manevich, A., Ratner, N., Rozen, N., Shwartz, E., Zusman, M., and Shoham, Y. Ja...

2024 arXiv
[38]

and Picci, G

Lindquist, A. and Picci, G. On the stochastic realization problem. SIAM J. Control Optim., 17 0 (3): 0 365–389, May 1979. ISSN 0363-0129. doi:10.1137/0317028. URL https://doi.org/10.1137/0317028

1979 doi
[39]

System Identification: Theory for the User

Ljung, L. System Identification: Theory for the User. Prentice Hall information and system sciences series. Prentice Hall PTR, 1999. ISBN 9780136566953. URL https://books.google.com/books?id=nHFoQgAACAAJ

1999
[40]

and Hutter, F

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

2019 arXiv
[41]

Lynn, P. A. and Fuerst, W. Introductory digital signal processing with computer applications (revised ed.). John Wiley & Sons, Inc., USA, 1994. ISBN 0471943746

1994
[42]

and Cundy, C

Martin, E. and Cundy, C. Parallelizing linear recurrent neural nets over sequence length, 2018. URL https://arxiv.org/abs/1709.04057

2018 arXiv
[43]

Dissecting neural odes, 2021

Massaroli, S., Poli, M., Park, J., Yamashita, A., and Asama, H. Dissecting neural odes, 2021. URL https://arxiv.org/abs/2002.08071

2021 arXiv
[44]

Y., Kumbong, H., Parnichkun, R

Massaroli, S., Poli, M., Fu, D. Y., Kumbong, H., Parnichkun, R. N., Timalsina, A., Romero, D. W., McIntyre, Q., Chen, B., Rudra, A., Zhang, C., Re, C., Ermon, S., and Bengio, Y. Laughing hyena distillery: Extracting compact recurrences from convolutions, 2023. URL https://arxi...

2023 arXiv
[45]

and Li, Z

Min, Z. and Li, Z. On the efficiency of transformers: The effect of attention rank, 2024. URL https://openreview.net/forum?id=U9sHVjidYH

2024
[46]

In-context learning and induction heads

Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Cla...

2022
[47]

In-context learning and induction heads

Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Cla...

2022
[48]

V., Willsky, A

Oppenheim, A. V., Willsky, A. S., and Nawab, S. H. Signals & systems (2nd ed.). Prentice-Hall, Inc., USA, 1996. ISBN 0138147574

1996
[49]

L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, S

Orvieto, A., Smith, S. L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, S. Resurrecting recurrent neural networks for long sequences, 2023. URL https://arxiv.org/abs/2303.06349

2023 arXiv
[50]

N., Massaroli, S., Moro, A., Smith, J

Parnichkun, R. N., Massaroli, S., Moro, A., Smith, J. T. H., Hasani, R., Lechner, M., An, Q., Ré, C., Asama, H., Ermon, S., Suzuki, T., Yamashita, A., and Poli, M. State-free inference of state-space models: The transfer function approach, 2024. URL https://arxiv.org/abs/2405.06147

2024 arXiv
[51]

On the difficulty of training recurrent neural networks, 2013

Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks, 2013. URL https://arxiv.org/abs/1211.5063

2013 arXiv
[52]

B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L

Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024

2024
[53]

Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C

Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C. Hyena hierarchy: Towards larger convolutional language models, 2023. URL https://arxiv.org/abs/2302.10866

2023 arXiv
[54]

W., Nguyen, E., Ponnusamy, P., Deiseroth, B., Kersting, K., Suzuki, T., Hie, B., Ermon, S., Ré, C., Zhang, C., and Massaroli, S

Poli, M., Thomas, A. W., Nguyen, E., Ponnusamy, P., Deiseroth, B., Kersting, K., Suzuki, T., Hie, B., Ermon, S., Ré, C., Zhang, C., and Massaroli, S. Mechanistic design and scaling of hybrid architectures, 2024. URL https://arxiv.org/abs/2403.17844

2024 arXiv
[55]

Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022

Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022. URL https://arxiv.org/abs/2201.02177

2022 arXiv
[56]

K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S

Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks is all you need, 2021. URL https://arxiv.or...

2021 arXiv
[57]

W., Kuzina, A., Bekkers, E

Romero, D. W., Kuzina, A., Bekkers, E. J., Tomczak, J. M., and Hoogendoorn, M. Ckconv: Continuous kernel convolution for sequential data, 2022. URL https://arxiv.org/abs/2102.02611

2022 arXiv
[58]

and Vetterli, M

Roy, O. and Vetterli, M. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference, pp.\ 606--610, 2007

2007
[59]

Rumelhart, D. E. and McClelland, J. L. Learning Internal Representations by Error Propagation, pp.\ 318--362. 1987

1987
[60]

Glu variants improve transformer, 2020

Shazeer, N. Glu variants improve transformer, 2020. URL https://arxiv.org/abs/2002.05202

2020 arXiv
[61]

Mutual information scaling and expressive power of sequence models, 2019

Shen, H. Mutual information scaling and expressive power of sequence models, 2019. URL https://arxiv.org/abs/1905.04271

2019 arXiv
[62]

Smith, J. T. H., Warrington, A., and Linderman, S. W. Simplified state space layers for sequence modeling, 2023. URL https://arxiv.org/abs/2208.04933

2023 arXiv
[63]

Roformer: Enhanced transformer with rotary position embedding, 2023

Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

2023 arXiv
[64]

Z., and Liu, Z

Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models, 2024. URL https://arxiv.org/abs/2402.17762

2024 arXiv
[65]

Retentive network: A successor to transformer for large language models, 2023

Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models, 2023. URL https://arxiv.org/abs/2307.08621

2023 arXiv
[66]

H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R

Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: A unified understanding of transformer's attention via the lens of kernel, 2019. URL https://arxiv.org/abs/1908.11775

2019 arXiv
[67]

Leveraging low-rank and sparse recurrent connectivity for robust closed-loop control, 2023

Tumma, N., Lechner, M., Loo, N., Hasani, R., and Rus, D. Leveraging low-rank and sparse recurrent connectivity for robust closed-loop control, 2023. URL https://arxiv.org/abs/2310.03915

2023 arXiv
[68]

A note on the representation and definition of semiseparable matrices

Vandebril, R., Van Barel, M., and Mastronardi, N. A note on the representation and definition of semiseparable matrices. Numerical Linear Algebra with Applications, 12 0 (8): 0 839--858, 2005. doi:https://doi.org/10.1002/nla.455. URL https://onlinelibrary.wiley.com/doi/abs/10....

2005 doi
[69]

N., Kaiser, L., and Polosukhin, I

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

2023 arXiv
[70]

A multiscale visualization of attention in the transformer model, 2019

Vig, J. A multiscale visualization of attention in the transformer model, 2019. URL https://arxiv.org/abs/1906.05714

2019 arXiv
[71]

Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024

Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL https://arxiv...

2024 arXiv
[72]

Willems, J. C. Models for Dynamics, pp.\ 171--269. Vieweg+Teubner Verlag, Wiesbaden, 1989. ISBN 978-3-322-96657-5. doi:10.1007/978-3-322-96657-5_5. URL https://doi.org/10.1007/978-3-322-96657-5_5

1989 doi
[73]

On the role of attention masks and layernorm in transformers, 2024

Wu, X., Ajorlou, A., Wang, Y., Jegelka, S., and Jadbabaie, A. On the role of attention masks and layernorm in transformers, 2024. URL https://arxiv.org/abs/2405.18781

2024 arXiv
[74]

Xia, J., Chandrasekaran, S., Gu, M., and Li, X. S. Fast algorithms for hierarchically semiseparable matrices. Numerical Linear Algebra with Applications, 17 0 (6): 0 953--976, 2010. doi:https://doi.org/10.1002/nla.691. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/nla.691

2010 doi
[75]

Efficient streaming language models with attention sinks, 2024

Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks, 2024. URL https://arxiv.org/abs/2309.17453

2024 arXiv
[76]

Gated linear attention transformers with hardware-efficient training, 2024 a

Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efficient training, 2024 a . URL https://arxiv.org/abs/2312.06635

2024 arXiv
[77]

Parallelizing linear transformers with the delta rule over sequence length, 2024 b

Yang, S., Wang, B., Zhang, Y., Shen, Y., and Kim, Y. Parallelizing linear transformers with the delta rule over sequence length, 2024 b . URL https://arxiv.org/abs/2406.06484

2024 arXiv
[78]

B'mojo: Hybrid state space realizations of foundation models with eidetic and fading memory, 2024

Zancato, L., Seshadri, A., Dukler, Y., Golatkar, A., Shen, Y., Bowman, B., Trager, M., Achille, A., and Soatto, S. B'mojo: Hybrid state space realizations of foundation models with eidetic and fading memory, 2024. URL https://arxiv.org/abs/2407.06324

2024 arXiv
[79]

Hellaswag: Can a machine really finish your sentence?, 2019

Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830

2019 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

and Zuidema, W

Abnar, S. and Zuidema, W. Quantifying attention flow in transformers, 2020. URL https://arxiv.org/abs/2005.00928

arXiv 2020

[3] [3]

Stochastic theory of minimal realization

Akaike, H. Stochastic theory of minimal realization. IEEE Transactions on Automatic Control, 19 0 (6): 0 667--674, 1974. doi:10.1109/TAC.1974.1100707

arXiv 1974

[4] [4]

In-context language learning: Architectures and algorithms, 2024

Akyürek, E., Wang, B., Kim, Y., and Andreas, J. In-context language learning: Architectures and algorithms, 2024. URL https://arxiv.org/abs/2401.12973

arXiv 2024

[5] [5]

The hidden attention of mamba models, 2024

Ali, A., Zimerman, I., and Wolf, L. The hidden attention of mamba models, 2024. URL https://arxiv.org/abs/2403.01590

arXiv 2024

[6] [6]

and Li, Y

Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.3, knowledge capacity scaling laws, 2024. URL https://arxiv.org/abs/2404.05405

arXiv 2024

[7] [7]

A., Alnumay, Y., Alrashed, S., Alsubaie, S., Almushaykeh, Y., Mirza, F., Alotaibi, N., Altwairesh, N., Alowisheq, A., Bari, M

Alzahrani, N., Alyahya, H. A., Alnumay, Y., Alrashed, S., Alsubaie, S., Almushaykeh, Y., Mirza, F., Alotaibi, N., Altwairesh, N., Alowisheq, A., Bari, M. S., and Khan, H. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards, 2024. URL https://arxiv.org/abs/2402.01781

arXiv 2024

[8] [8]

Zoology: Measuring and improving recall in efficient language models, 2023

Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and Ré, C. Zoology: Measuring and improving recall in efficient language models, 2023. URL https://arxiv.org/abs/2312.04927

arXiv 2023