Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models

Fusheng Liu; Qianxiao Li

REVIEW 3 major objections 4 minor 13 references

Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The correct timescale for initializing a state space model depends on the autocorrelation spectrum of its input, and zeroing the state eigenvalues' real part can extend memory without sacrificing stability.

desk verdict Useful timescale bound and a genuine conditioning insight, but Theorem 2 is false as stated because it silently assumes positive imaginary parts. read the letter →

arxiv 2411.19455 v1 pith:MW2O7OLU submitted 2024-11-29 cs.LG

Fusheng Liu , Qianxiao Li This is my paper · ORCID

classification cs.LG MSC 68T0768T0568Q32

keywords statespacemodelsinitializationschemestimescalescalinginputautocorrelationtrainingstabilitycurseofmemoryGrammatrixconditioningapproximation-estimationtradeoff

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper gives a theoretical account of how to initialize a diagonal state space model (SSM) for fixed-length sequence tasks. Its first result bounds the expected square of the initialized output by $\Delta^2 m^2 L \lambda_{\max}(\mathbb{E}[xx^\top])$, which ties the timescale $\Delta$ to both the sequence length $L$ and the autocorrelation spectrum of the input. The paper concludes that $\Delta$ should scale as $O(1/\sqrt{L\,\lambda_{\max}(\mathbb{E}[xx^\top])})$, ranging from $L^{-1/2}$ for uncorrelated inputs to $L^{-1}$ for fully correlated ones. It then argues that with this $\Delta$, setting the real part of the state matrix eigenvalues to zero avoids exponential decay of the SSM kernel and eases the curse of memory while keeping initialization stable. Finally, it shows that the imaginary parts of the eigenvalues control the conditioning of a convex training objective, with well-separated imaginary parts giving fast convergence but risking a frequency mismatch with the target function, producing an approximation-estimation tradeoff.

What carries the argument

The load-bearing objects are the input autocorrelation matrix $\mathbb{E}[xx^\top]$ and the SSM kernel (memory) function $\Re(c^\top e^{w s})$. Theorem 1 is carried by writing the final output as $y_L=\Delta\, c^\top V J x$ with $V$ a Vandermonde-type matrix built from powers of the discretized state eigenvalues $e^{\Delta w_j}$; the bound follows from the factorization $V=\tfrac12\Phi^H D V_L$, the estimate $|(e^z-1)/z|\le 1$ for $\Re(z)\le 0$, and the observation that the exchange matrix $J$ is orthogonal, leaving $\lambda_{\max}(\mathbb{E}[xx^\top])$ as the only input-dependent scale. For the optimization analysis, the central identity is the Gram matrix $G_{jk}=\int_0^\infty \Re(e^{w_j s})\Re(e^{w_k s})\,ds$ of the basis kernels; its extreme eigenvalues are bounded in terms of the separation distance $\delta=\min_{j\ne k}|\Im(w_j)-\Im(w_k)|$, and the Hilbert-matrix case emerges when all imaginary parts vanish, giving exponentially growing condition numbers.

What would settle it

Run a zero-order-hold diagonal SSM with $\Re(w)=0$ and inputs drawn from an Ornstein-Uhlenbeck process with a fixed correlation length, for which $\lambda_{\max}(\mathbb{E}[xx^\top])$ grows sublinearly in $L$, and measure the largest $\Delta(L)$ at which $\mathbb{E}[y_L^2]$ stays bounded as $L$ increases. If $\Delta^*(L)\sqrt{L\,\lambda_{\max}(\mathbb{E}[xx^\top])}$ is not roughly constant—for instance, if the empirical threshold $\Delta^*$ decays much faster or slower than $1/\sqrt{L\,\lambda_{\max}}$—then the upper bound is not tight and the paper's scaling rule does not describe the true stability frontier.

Watch

Extended reading notes

Core claim

The paper's central claim is that the three classic components of SSM initialization—timescale, real part of the state eigenvalues, and imaginary part of the state eigenvalues—should be chosen from the data, not from a universal rule. For a zero-order-hold discretized diagonal SSM with $\Re(w_j)\le 0$ and Gaussian readout, Theorem 1 gives $\mathbb{E}_{c,x}[y_L^2]\le \Delta^2 m^2 L\,\lambda_{\max}(\mathbb{E}[xx^\top])$, and the paper treats this as the right dependence of $\Delta$ on $L$: initialize $\Delta=O(1/\sqrt{L\,\lambda_{\max}(\mathbb{E}[xx^\top])})$. Because $\lambda_{\max}(\mathbb{E}[xx^\top])$ ranges from $1$ for i.i.d. inputs to $L$ for constant inputs, the recommended scaling interpolates between $1/\sqrt{L}$ and $1/L$. With that timescale, the real part of $w$ may be initialized at zero without an output blow-up, removing the exponential decay that causes poor long-term memory. For optimization, the paper analyzes the Gram matrix $G_{jk}=\int_0^\infty \Re(e^{w_j s})\Re(e^{w_k s})\,ds$ of the kernel basis and proves, via the Gershgorin circle theorem, that its conditioning is governed by the separation of the imaginary parts $\Im(w_j)$; this explains the advantage of complex-valued over real-valued SSMs and reveals the tradeoff between well-conditioned optimization and accurate approximation of targets with close frequencies.

Load-bearing premise

The load-bearing premise is that the upper bound on the expected squared output actually marks the edge of stable training—if that bound is loose for correlated inputs, the recommended timescale is merely a safe overestimate and the claimed link between autocorrelation and initialization loses its force.

Editorial extensions

If this is right

For uncorrelated or whitened inputs, the paper's rule says to initialize $\Delta\sim L^{-1/2}$ rather than the common $L^{-1}$, and its copying-task experiments show this choice can turn a task that previously seemed to need large state size into a fixed-length task a small SSM can learn.
For strongly correlated inputs, $\lambda_{\max}(\mathbb{E}[xx^\top])\sim L$, so the rule reverts to $\Delta\sim L^{-1}$, matching common practice but now with a data-dependent justification.
Initializing the real part of the eigenvalues at zero with the correct timescale keeps the output bounded while making the kernel non-decaying, so targets with long-range spikes (like a shift by $L$ positions) become learnable without stable reparameterization.
Complex-valued SSMs with well-separated imaginary parts have Gram matrices whose condition number stays bounded as the hidden size grows, unlike real-valued SSMs, whose Gram matrix approaches a Hilbert matrix.
Spreading the imaginary frequencies too far improves conditioning but can miss the dominant frequencies of the target memory function, so initialization should balance conditioning against approximation error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not prove that the upper bound in Theorem 1 is tight for general autocorrelated inputs; if it is loose, the recommended $\Delta$ is an overestimate of safety, and larger timescales might be usable to retain more memory—a matching lower bound would settle this.
A practical pipeline the paper gestures at but does not automate is to estimate $\lambda_{\max}(\mathbb{E}[xx^\top])$ (or the full spectrum) from a batch of sequences, set $\Delta$ from that estimate, then choose the imaginary frequency grid by thresholding the recovered memory function's Fourier content; this could be tested end-to-end on the LRA tasks.
The conditioning analysis fixes $w$ and optimizes only the readout $c$; extending the argument to the full non-convex dynamics where $w$ and $\Delta$ also move would test whether the zero-real-part benefits survive gradient descent, since the paper only observes that some non-negative real parts remain after training.
Whitening or decorrelating inputs before an SSM changes $\lambda_{\max}(\mathbb{E}[xx^\top])$ to $1$, which by the paper's rule shifts the correct initialization from $\Delta\sim 1/L$ to $\Delta\sim 1/\sqrt{L}$; this connects the initialization theory directly to data preprocessing choices.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The genuinely new and useful piece is Theorem 1: for ZOH-discretized diagonal SSMs with Re(w)≤0 and Gaussian c, E[y_L^2] ≤ Δ² m² L λ_max(E[xx^T]). That gives practitioners a data-dependent rule Δ = O(1/√(L λ_max)), and the paper shows numerically that the right power of L shifts from 1/√L for i.i.d. to 1/L for strongly correlated sequences. The zero-real-part experiments and the curse-of-memory discussion are also worth reading; the LRA results suggest real benefit even without stable reparameterization.

The soft spot is Section 4.3, and it's not minor. Proposition 1 and Theorem 2 claim positive definiteness and a lower eigenvalue bound for any distinct real v_j, but that's false. Take v_1 = -a and v_2 = a. Since cos is even, ℜ(e^{w_1 s}) = ℜ(e^{w_2 s}); the two basis functions coincide, the Gram matrix is rank-one, and λ_min(G) = 0. That directly contradicts Theorem 2 for large a (δ = 2a, bound positive). The proof of Theorem 2 appears to assume v_j > 0 in the text ('the last inequality is because v_j > 0'), but the theorem statement never says that. Same for Proposition 1. This is a load-bearing flaw in the conditioning analysis, but it's repairable: require 0 < v_1 < ... < v_m, or replace v by |v|. The practical S4D-Lin initialization uses positive v_j, so the intended conclusion—well-separated positive frequencies give a well-conditioned Gram matrix—survives, and the comparison with the Hilbert matrix for real-valued SSMs is still convincing for the intended setting.

Other concerns are smaller. Theorem 1 is an upper bound and the paper sometimes talks as if it's a characterization; the tightness discussion is honest about the i.i.d. Re(w)=0 case. LRA experiments lack error bars, and no code is shipped. Both are addressable.

Bottom line: the timescale result is solid, the conditioning story is right in the intended regime but the theorem as printed is false and must be fixed. I'd send it to review with a clear request for that repair. It's not desk-reject material, but the authors need to add the positivity assumption and re-check Proposition 1 before this is publishable.

Referee Report

3 major / 4 minor

Summary. The paper studies initialization schemes for diagonal state space models (SSMs), focusing on three components: the timescale Delta, the real part of the eigenvalues of the state matrix, and the imaginary part. In Section 4.1, Theorem 1 gives an upper bound on the expected squared output at initialization, E_{c,x}[y_L^2] <= Delta^2 m^2 L lambda_max(E[xx^T]), under Re(w_j) <= 0 and Gaussian read-out, and the authors propose initializing Delta as O(1/sqrt(L lambda_max(E[xx^T]))). In Section 4.2, they argue that initializing Re(w)=0 can mitigate the curse of memory for fixed-length tasks. In Section 4.3, they analyze a convex least-squares problem with fixed w and show that the conditioning of the induced Gram matrix is controlled by the separation of the imaginary parts Im(w), leading to an approximation-estimation tradeoff. The paper includes numerical experiments on synthetic processes, sequential MNIST/CIFAR, copying tasks, and the LRA benchmark.

Significance. If the main results hold after repair, the paper provides a useful, data-dependent guideline for SSM initialization: the timescale should depend on the spectrum of the input autocorrelation matrix, not just on sequence length, and complex eigenvalues with well-separated imaginary parts can yield well-conditioned optimization problems. The derivations are self-contained, with no fitted constants, and the numerical experiments align with the sufficient bound in several cases. The paper also connects to the curse-of-memory literature and offers a concrete, falsifiable prediction about timescale scaling. However, the false statement of Proposition 1 and Theorem 2 as written, and the unproven necessity of the timescale recommendation, prevent acceptance in the current form.

major comments (3)

[Section 4.3, Proposition 1] Proposition 1 is false as stated. The statement allows v_j in R with all v_j distinct and nonzero, but if v_1 = -a and v_2 = a with the same real part, then Re(e^{w_1 s}) = e^{-s/2} cos(a s) = Re(e^{w_2 s}), so the two basis functions coincide and the Gram matrix G is rank-one, contradicting positive definiteness. The proof's Vandermonde argument also fails in this case because the nodes e^{gamma w_j} and e^{gamma \bar w_k} coincide when v_j = -v_k. The proposition needs an explicit positivity condition such as 0 < v_1 < ... < v_m, or a replacement of v by |v| with distinct absolute values.
[Section 4.3, Theorem 2] Theorem 2 inherits the same flaw as Proposition 1. Its statement assumes only v_j in R and delta := min_{j != k} |v_j - v_k| > 0, but for v_1 = -a and v_2 = a the Gram matrix is singular while delta = 2a > 0, so the claimed lower bound lambda_min(G) > 1.19 - (3 pi / (4 delta)) coth(pi/delta) is false for such choices. The proof in Appendix E explicitly uses the condition 'v_j > 0 and reordering {v_k}_{k>=1}', which is not stated in the theorem. The theorem must be restated with 0 < v_1 < ... < v_m. Additionally, the lower bound is vacuous for delta <= 2.3, a gap the authors acknowledge; the theorem should state clearly for which delta the bound is nontrivial.
[Section 4.1, Theorem 1 and following discussion] The paper claims in the abstract and introduction to 'rigorously characterize the dependency' of the timescale on sequence length based on autocorrelation, but Theorem 1 provides only an upper bound E_{c,x}[y_L^2] <= Delta^2 m^2 L lambda_max(E[xx^T]). The recommendation Delta = O(1/sqrt(L lambda_max(E[xx^T]))) is sufficient for the bound to be O(1), but no matching lower bound is proved showing that larger Delta leads to instability. The tightness discussion after Theorem 1 covers only Re(w)=0 with i.i.d. inputs and the constant-sequence case; the general autocorrelation case is asserted rather than proved. To support the 'characterize' language, the authors should either prove a converse under appropriate conditions or explicitly state that only a sufficient condition is established.

minor comments (4)

[Section 4.1, Figure references] In the paragraph discussing sequential image datasets, the text refers to 'Figure 1 (Left)' when describing the maximal eigenvalue scaling with resize rate; this should be 'Figure 1 (Right)', since the left panel shows the copying-task accuracy and the right panel shows lambda_max(E[xx^T]) for sMNIST and sCIFAR10.
[Table 1 and Table 3] The LRA results in Table 1 and the sCIFAR ablations in Table 3 report single runs without standard deviations or multiple seeds. Given that the improvements over baseline are a few points and that the claimed benefit is the central empirical support for Section 4.2, error bars or at least a note on run-to-run variability would strengthen the evidence.
[Remark 2] The pooling-mode formula writes y = (1/L) sum_{\ell=1}^L y_L^2, which appears to be a typo: it should be y = (1/L) sum_{\ell=1}^L y_\ell^2, with the average taken over output positions.
[Appendix E, proof line] The proof of Theorem 2 uses the identity 1/(1+v_j^2+v_k^2) <= 1/(1+v_j^2+delta^2 k^2) after reordering, which relies on both v_j > 0 and an ordering of the v_k. This dependency should be made explicit in the theorem statement, as noted in the major comments; the unattributed reordering makes the proof difficult to follow.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the core derivations are self-contained, and the flagged Theorem 2 issue is a correctness matter, not a self-referential argument.

full rationale

The paper's main derivation chain is self-contained. Theorem 1 is a genuine mathematical upper bound proved from the ZOH discretization, a Vandermonde representation, and standard inequalities; no fitted constant or assumed conclusion enters it. The recommended initialization Δ = O(1/sqrt(L λmax(E[xx^T]))) is a sufficient scaling derived from that bound, not a parameter fitted to the same data and then renamed a prediction. Section 4.2 argues from the definition of the SSM kernel that a zero real part avoids exponential decay, and it supports the claim with independent experiments; the citations to Li et al. (2021, 2022) for the phrase 'curse of memory' are background context, not the load-bearing proof. Section 4.3's Gram-matrix conditioning analysis is also self-contained: it computes the Gram entries via Lemma 5 and applies Gershgorin's theorem. The known defect in Theorem 2/Proposition 1—that allowing v_j = -a and v_k = +a makes the Gram matrix rank-one, so positive definiteness requires an additional positivity/sorting assumption—is a mathematical correctness gap, not a circular step, because the theorem's conclusion is not assumed in its proof and does not reduce to its inputs by construction. No equation in the paper is equivalent to its own inputs, and no self-citation chain forces the central claims.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

No fitted numerical constants enter the theorems; the bounds use standard lemmas. The main domain assumptions are Gaussian read-out and finite-second-moment inputs, ZOH discretization, fixed-length tasks, and a white-noise idealization for the optimization analysis. One proof uses an unstated positivity condition on v_j.

assumptions (8)

standard math Gershgorin circle theorem (Lemma 3) is used to bound the spectrum of the Gram matrix G in Theorem 2.
Invoked in Appendix E to convert diagonal and off-diagonal bounds into eigenvalue bounds; accepted background result.
domain assumption Ito isometry (Lemma 2) and a white-noise input model x(T-s)ds = dW_s are used to rewrite the expected risk as an L2 integral in Section 4.3.
The continuous-time input is idealized as white noise; this is not true for real discrete sequences, so the conditioning analysis applies to this idealization.
standard math Hanson-Wright inequality (Lemma 6) is used for the high-probability version of Theorem 1.
For concentration of quadratic forms in sub-Gaussian c; standard result.
standard math Vandermonde matrix invertibility is used in Proposition 1 to prove positive definiteness of G.
Requires distinct nodes e^{w_j}; the paper argues distinct imaginary parts give distinct nodes.
domain assumption Input sequences have finite second moment so E[xx^T] exists, and the read-out vector c is initialized as i.i.d. standard normal (Theorem 1).
These distributional assumptions are stated in Theorem 1 and are needed for the second-moment bound.
domain assumption ZOH discretization and SISO model are assumed; results are not directly extended to bilinear discretization (Remark 1).
The matrix forms for other discretizations differ, so the theory's scope is limited.
domain assumption The analysis is for fixed-length sequences; for varied lengths the paper suggests clustering or increasing feature dimension (Remark 2).
Stability is assessed at a terminal time L; length generalization is explicitly out of scope.
ad hoc to paper Theorem 2 proof assumes v_j > 0 when reordering frequencies, but the theorem statement only says v_j in R.
In Appendix E, the bound uses positivity of v_j to reorder the off-diagonal sum; this is not stated among the theorem hypotheses, creating a gap between statement and proof.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models." pith.science (2026). https://pith.science/paper/MW2O7OLU

@misc{pith2026241119455,
  author       = {Pith},
  title        = {Pith review of: Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MW2O7OLU}},
  note         = {Machine review of arXiv:2411.19455}
}

read the original abstract

Current methods for initializing state space model (SSM) parameters primarily rely on the HiPPO framework \citep{gu2023how}, which is based on online function approximation with the SSM kernel basis. However, the HiPPO framework does not explicitly account for the effects of the temporal structures of input sequences on the optimization of SSMs. In this paper, we take a further step to investigate the roles of SSM initialization schemes by considering the autocorrelation of input sequences. Specifically, we: (1) rigorously characterize the dependency of the SSM timescale on sequence length based on sequence autocorrelation; (2) find that with a proper timescale, allowing a zero real part for the eigenvalues of the SSM state matrix mitigates the curse of memory while still maintaining stability at initialization; (3) show that the imaginary part of the eigenvalues of the SSM state matrix determines the conditioning of SSM optimization problems, and uncover an approximation-estimation tradeoff when training SSMs with a specific class of target functions.

Figures

Figures reproduced from arXiv: 2411.19455 by the authors.

**Figure 1.** (Left) Training a diagonal SSM (3) on a copying task using i.i.d. data with a dimension of 128. We vary the minimal timescale ∆min = 1/L, 1/ √ L and the maximal timescale ∆max = 1/L, 1/ √ L, 0.1 w.r.t. sequence length L. (Middle) The maximal eigenvalue of the autocorrelation matrix E[xx⊤] on different random processes of x. (Right) The maximal eigenvalue of E[xx⊤] on sequential image datasets sMNIST and sCIFAR10 wit… view at source ↗

**Figure 2.** The expected magnitude of the SSM output value on synthetic sequences with different auto [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** The expected magnitude of the SSM output value on synthetic sequences with different autocor [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: The expected magnitude of the SSM output value on sequential image datasets with different [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: (Left) Training a diagonal SSM (3) on a task that requires long-term memory. The learned memory function ρ˜ effectively captures the spike in long-range dependencies. However, it struggles to do so when the real part is negative. (Middle) Test loss on the long-term mem…

**Figure 6.** Figure 6: Recovering the memory function ρ on the decorrelated sequential MNIST dataset by solving a linear equation X ∗ ρ = Y , where X ∈ R N×L is the collected sequence matrix, Y ∈ R N×10 is the one-hot label matrix, and ∗ is the convolution operator. Then ρ ∈ R L×10 has 10 ch…

**Figure 7.** Figure 7: (Left) Condition number κ(G) := λmax(G) λmin(G) for S4D-Real and S4D-Lin with different hidden size m. (Middle) κ(G) for S4D-Lin with different m by varying scaling factors of the imaginary part ℑ(a). (Right) κ(G) and approximation measure σmax(M) (in the approximation…

**Figure 8.** Figure 8: Behavior of the real part ℜ(w) after training on the synthetic task and the decorrelated sMNIST dataset. timescale ∆ = 1/ √ 128. We use Adam optimizer with learning rate 0.001 on ∆, ℜ(w), ℑ(w) and learning rate 0.01 on c [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

**Figure 9.** Figure 9: The expected magnitude of the SSM output value on synthetic sequences with S4D-Legs initial [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

**Figure 10.** Figure 10: The expected magnitude of the SSM output value on synthetic sequences with S4D-Legs initial [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

**Figure 11.** Figure 11: The expected magnitude of the SSM output value for S4D-Legs initialization on sequential image [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

**Figure 12.** Figure 12: (Left) Training a diagonal SSM (3) with S4D-Legs initialization on a task that requires longterm memory. The learned memory function ρ˜ effectively captures the spike in long-range dependencies. However, it struggles to do so when the real part is negative. (Middle) …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 1 canonical work pages

[1]

Transformers to ssms: Distilling quadratic knowledge to subquadratic models

Aviv Bick, Kevin Y Li, Eric P Xing, J Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. arXiv preprint arXiv:2408.10189,

arXiv
[3]

Mamba: Linear-time sequence modeling with selective state spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

arXiv
[5]

For any vj, vk ∈ R, we have Z ∞ 0 e−s cos(vjs) cos(vks)ds = 1 2 1 1 + (vj − vk)2 + 1 1 + (vj + vk)2 . Proof. Notice thatZ ∞ 0 e−s cos(vjs) cos(vks)ds = 1 2 Z ∞ 0 e−s cos((vj − vk)s)ds + 1 2 Z ∞ 0 e−s cos((vj + vk)s)ds = 1 2 Z ∞ 0 ℜ (exp (−s + i · (vj − vk)s)) ds + 1 2 Z ∞ 0 ℜ (exp (−s + i · (vj + vk)s)) ds = 1 2 ℜ 1 1 − i · (vj − vk) + 1 1 − i · (vj + vk)...

work page 2013
[6]

Adam: A method for stochastic optimization

Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

arXiv
[7]

Vmamba: Visual state space model

Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166,

arXiv
[10]

The mamba in the llama: Distilling and accelerating hybrid models

Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. arXiv preprint arXiv:2408.15237,

arXiv
[11]

There is hope to avoid hippos for long-memory state space models

15 Annan Yu, Michael W Mahoney, and N Benjamin Erichson. There is hope to avoid hippos for long-memory state space models. arXiv preprint arXiv:2405.13975,

arXiv
[12]

Recurrent neural networks: vanishing and exploding gradients are not the end of the story

Nicolas Zucchet and Antonio Orvieto. Recurrent neural networks: vanishing and exploding gradients are not the end of the story. arXiv preprint arXiv:2405.21064,

arXiv

Show all 13 references

[2013]

Un- derstanding the differences in foundation models: Attention, state space models, and recurrent neural networks

Jerome Sieber, Carmen Amo Alonso, Alexandre Didier, Melanie N Zeilinger, and Antonio Orvieto. Un- derstanding the differences in foundation models: Attention, state space models, and recurrent neural networks. arXiv preprint arXiv:2405.15731,

arXiv
[2016]

Hydra: Bidirectional state space models through generalized matrix mixers

Sukjun Hwang, Aakash Lahoti, Tri Dao, and Albert Gu. Hydra: Bidirectional state space models through generalized matrix mixers. arXiv preprint arXiv:2407.09941,

arXiv
[2017]

An empirical study of mamba-based language models

Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887,

arXiv
[2022]

Gaussian error linear units (gelus)

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415,

arXiv
[2024]

Theoretical foundations of deep selective state-space models

Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons. Theoretical foundations of deep selective state-space models. arXiv preprint arXiv:2402.19047,

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Transformers to ssms: Distilling quadratic knowledge to subquadratic models

Aviv Bick, Kevin Y Li, Eric P Xing, J Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. arXiv preprint arXiv:2408.10189,

arXiv

[2] [3]

Mamba: Linear-time sequence modeling with selective state spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

arXiv

[3] [5]

For any vj, vk ∈ R, we have Z ∞ 0 e−s cos(vjs) cos(vks)ds = 1 2 1 1 + (vj − vk)2 + 1 1 + (vj + vk)2 . Proof. Notice thatZ ∞ 0 e−s cos(vjs) cos(vks)ds = 1 2 Z ∞ 0 e−s cos((vj − vk)s)ds + 1 2 Z ∞ 0 e−s cos((vj + vk)s)ds = 1 2 Z ∞ 0 ℜ (exp (−s + i · (vj − vk)s)) ds + 1 2 Z ∞ 0 ℜ (exp (−s + i · (vj + vk)s)) ds = 1 2 ℜ 1 1 − i · (vj − vk) + 1 1 − i · (vj + vk)...

work page 2013

[4] [6]

Adam: A method for stochastic optimization

Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

arXiv

[5] [7]

Vmamba: Visual state space model

Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166,

arXiv

[6] [10]

The mamba in the llama: Distilling and accelerating hybrid models

Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. arXiv preprint arXiv:2408.15237,

arXiv

[7] [11]

There is hope to avoid hippos for long-memory state space models

15 Annan Yu, Michael W Mahoney, and N Benjamin Erichson. There is hope to avoid hippos for long-memory state space models. arXiv preprint arXiv:2405.13975,

arXiv

[8] [12]

Recurrent neural networks: vanishing and exploding gradients are not the end of the story

Nicolas Zucchet and Antonio Orvieto. Recurrent neural networks: vanishing and exploding gradients are not the end of the story. arXiv preprint arXiv:2405.21064,

arXiv