Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

SpectraLDS: Provable Distillation for Linear Dynamical Systems

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A trained Spectral Transform Unit can be provably converted into an explicit linear dynamical system, preserving accuracy while making generation constant-time per token.

desk verdict A genuinely new and useful STU-to-LDS distillation with strong empirical results, but the headline 'provable' claim rests on an unproven condition on lambda_max, so the paper needs a serious revision before the theory matches the abstract. read the letter →

arxiv 2505.17868 v2 pith:XNXKLTKU submitted 2025-05-23 cs.LG math.OC

Devan Shah , Shlomo Fortgang , Sofiia Druchyna , Elad Hazan This is my paper · ORCID
classification cs.LGmath.OC
keywords spectraltransformunitlineardynamicalsystemsdistillationstate-spacemodelsfilteringsequencepredictionconstant-timeinferenceconvexoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sequence models built on long convolutions, such as the Spectral Transform Unit (STU), capture long-range memory but pay per-token inference costs that grow with context. This paper claims the first provable method to distill a trained STU layer back into an explicit linear dynamical system (LDS), a plain recurrence of the form $x_t = A x_{t-1} + B u_t$, with an accuracy loss bounded by a term that shrinks as the number of spectral filters grows and does not depend on the original system's hidden dimension. If the construction works, a model trained as an STU can be switched at inference time to a recurrence that runs in constant time and constant memory per token while preserving predictive accuracy. The authors implement the distillation in a 340M-parameter language model and report benchmark scores statistically matched to the original STU, with much faster long-sequence generation.

What carries the argument

The load-bearing object is the coefficient matrix $M \in \mathbb{R}^{h \times k}$ built by Algorithm 2: row $i$ stores the coefficients that represent the geometric LDS impulse response $\mu_L(\alpha_i)$ in the spectral basis $\Phi_{1:k}$. The matrix $\tilde{M} := M^{-1}$, taken as the Penrose-Moore pseudo-inverse since $h \ge k$, maps geometric filters back to spectral filters. The proof uses a lemma from spectral filtering that each geometric response is approximated by the first $k$ spectral filters with error $e^{-k/\log L}$, together with the fact that the matrix of geometric responses $\mu_L(\alpha_1,\dots,\alpha_h)$ is a full-rank Vandermonde matrix. The unproved ingredient is control of $\lambda_{\max}$, the largest singular value of the pseudo-inverse, which the paper does not bound theoretically but reports empirically to decay as $h$ grows.

What would settle it

Compute the matrix $M$ from Algorithm 2 for $k=24$ and $h=100,200,\dots,1000$ using the paper's $\alpha$ distribution and evaluate $h \lambda_{\max}$; if this product grows without bound, or if the reconstruction error of $\Phi_{1:k}$ by $\tilde{M} \mu_L(\alpha_{1:h})$ exceeds the theorem's right-hand side, the guarantee is vacuous and the reported accuracy would rest on Algorithm 3's least-squares fine-tuning rather than Theorem 1.

Watch

Extended reading notes

Core claim

The central claim is that the $k$ spectral filters of an STU can be expressed, up to a controlled error, as a linear combination of impulse responses of $h$ one-dimensional geometric LDS filters. Concretely, Theorem 1 states that for $h \ge k$, Algorithm 2 returns a matrix $\tilde{M}$ such that $\|\Phi_{1:k} - \tilde{M} \mu_L(\alpha_1,\dots,\alpha_h)\| \le c \lambda_{\max} h e^{-k/\log L}$, where $\Phi_{1:k}$ are the spectral filters, $\mu_L(\alpha_i) = (1-\alpha_i)(1, \alpha_i, \alpha_i^2, \dots, \alpha_i^{L-1})$ are geometric impulse responses, and $\lambda_{\max}$ is the largest singular value of the Penrose-Moore pseudo-inverse of the $h \times k$ coefficient matrix $M$ that expresses each geometric filter in the spectral basis. Because convolving with a geometric filter is exactly the impulse response of a scalar LDS, this identity converts the spectral filters into a diagonal LDS with state dimension $h$, and the alternating (negative-eigenvalue) filters are handled by replacing $A$ with $-A$. The paper argues that the $e^{-k/\log L}$ spectral approximation error is inherited from spectral filtering theory, so the guarantee is independent of the hidden dimension of the original system.

Load-bearing premise

The bound in Theorem 1 only bites if $h \lambda_{\max}$ stays $O(1)$, meaning the matrix expressing geometric filters in the spectral basis has a pseudo-inverse with a small largest singular value; the paper proves this nowhere and supports it only with experiments in Appendix A.2.

Editorial extensions

If this is right

  • For a fixed distilled state dimension ($h \approx 160$ in the language experiments), generation drops to $O(1)$ time and $O(1)$ memory per token, independent of sequence length, replacing convolution or FFT-based inference.
  • The approximation error $e^{-k/\log L}$ means accuracy is provably retained up to an exponentially-in-$k$ spectral error term that does not depend on how large or how marginally stable the original LDS hidden state was.
  • The same machinery yields an LDS-to-LDS reduction: learn any high-dimensional symmetric LDS through its convex STU parameterization, then distill to a low-dimensional LDS with bounded error, giving a practical state-dimension compression (Section 5.3).
  • Because training stays in the convex STU representation, the method inherits the STU's training stability for systems with eigenvalues close to 1, and the synthetic experiments show much lower loss and much faster convergence than a gradient-trained LDS baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the theorem's practical force rests on $h \lambda_{\max} = O(1)$, which the paper verifies only empirically in Appendix A.2; turning that observation into a proof for the paper's $\alpha$ sampling distribution would make the reduction fully parameter-free.
  • Editorial: the construction is limited to symmetric $A$, i.e. real eigenvalues; replacing the geometric dictionary by complex exponentials would extend the same inversion to oscillatory modes and is the natural next test.
  • Editorial: because the spectral basis comes from a fixed universal Hankel matrix, the expensive part of the distillation can be computed offline, suggesting that any convolution filter lying near the span of geometric decays could be distilled this way, not only STU filters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SpectraLDS, a method for distilling the convolutional filters of a Spectral Transform Unit (STU) into an explicit symmetric linear dynamical system (LDS). The central theoretical result, Theorem 1, states that for h ≥ k randomly sampled geometric filters, Algorithm 2 returns a matrix fM such that the first k spectral filters Φ1:k are approximated by fM times the LDS impulse responses, with error bounded by c·λmax·h·e^{−k/log L}, where λmax is the largest singular value of the pseudo-inverse of the spectral coefficient matrix M. The paper also presents a practical variant (Algorithm 3) based on subset selection and gradient refinement, and evaluates the approach on synthetic LDS identification, filter reconstruction, large-scale language modeling, and inference speed. The main claim is that this distillation preserves predictive accuracy while enabling constant-time per-token inference with provable error bounds independent of the original system's state dimension.

Significance. If the technical obstacles in Theorem 1 were resolved, the contribution would be significant: it would provide a first provable conversion from a spectral-filter representation to a constant-time recurrent form, with approximation error that does not depend on the original LDS state dimension or effective memory. The algebraic transformation in Section 5.2 is elegant and correct, and the empirical work is substantial: the paper reports filter reconstruction errors around 1e-12, detailed synthetic system-identification comparisons, language-modeling benchmarks with near-parity to the original FlashSTU, and careful runtime measurements showing substantial speedups. The paper also makes a credible effort at reproducibility, with detailed hyperparameters and ablations. However, the central 'provable' claim is currently conditional on an unproven spectral-condition bound, and the algorithm actually deployed in the large-scale experiments is not the one analyzed in the theorem. These gaps materially affect the strength of the main contribution as stated.

major comments (3)
  1. [Section 5.1 / Theorem 1 / Appendix A.3] The error bound in Theorem 1 contains the factor λmax, the largest singular value of the Penrose-Moore pseudo-inverse of the spectral coefficient matrix M. The proof in Appendix A.3 bounds ||Φ1:k − M†µL|| by λmax·||E||_1 and then invokes the spectral filtering estimate ||E||_1 ≤ c·h·e^{−k/log L}. However, nothing in the argument controls λmax. The paper states in §5.1 that an upper bound is 'not immediate' and appeals to the experimental results in Appendix A.2, but those experiments use k = 48 with a tailored α distribution, not the k = 24, h = 160, L = 131072 configuration used in the language modeling experiments of Table 2. Consequently, the condition h·λmax = O(1), which is essential for the displayed bound to be non-vacuous in the relevant regime, is neither proved nor directly certified for the deployed setting. The abstract's claim of a 'provable method' is therefore overstated. The authors should either prove a high-probability bound on λmax for a well-specified sampling distribution, or explicitly restate the theorem as a conditional guarantee and provide a systematic empirical certificate in the exact regime used in the evaluations.
  2. [Appendix A.3, full-rank argument for M] The proof of Theorem 1 argues that M is full rank by observing that Φ1:k is an orthogonal basis and that µL(α1:h) is a Vandermonde matrix, concluding that 'both matrices are full rank.' This is a non sequitur: M is the h×k matrix whose rows are the spectral coefficient vectors m_i obtained by projecting each geometric filter µL(α_i) onto the spectral basis. The rank of M is a property of the composed linear map, not of either factor individually, and full rank of the Vandermonde matrix µL(α1:h) does not by itself imply that the coefficient matrix M has full column rank. Since the definition of M† and the equality Φ1:k − M†µL = M†E require M to have full column rank, the proof needs a probability estimate showing that i.i.d. samples α_i from the stated distribution yield a full-column-rank M with high probability. Without such an estimate, the 'w.h.p.' claim in Theorem 1 is unsupported.
  3. [Appendix A.4 / Algorithm 3 vs. Theorem 1] The large-scale evaluation in Section 6, including Table 2 and Figure 3, uses Algorithm 3, which generates a large set of 1D-LDS/STU pairs, selects a row subset by reconstruction error, expands it greedily, and then refines fM by gradient descent. Theorem 1 analyzes only Algorithm 2, which directly samples α_i and computes the pseudo-inverse of the coefficient matrix M. The paper does not provide any theoretical guarantee that the output of Algorithm 3 approximates the spectral filters; the reported reconstruction error 1.23e-12 in Appendix A.9 is purely empirical. Thus the 'provable distillation' advertised in the abstract does not cover the algorithm that is actually evaluated on language modeling. The authors should either extend the analysis to Algorithm 3 or explicitly scope the theoretical claim to Algorithm 2 and present Algorithm 3 as a heuristic with strong empirical support.
minor comments (6)
  1. [Section 5.1 / Theorem 1] The norm in the statement of Theorem 1 is unspecified. The proof in Appendix A.3 uses ||E||_1 and ||M^{-1}||_⋆, so the theorem should state the relevant norm (or explain the compatibility between the two sides) to make the bound well-defined.
  2. [Throughout] There are several typographical errors: 'spectal' in the discussion after Theorem 1, 'symmetrix' in Section 4.1, 'requries' in Section 3, and the ungrammatical phrase 'this distillation procedure this provides' in Section 1.1.
  3. [Algorithm 1] Line 3 constructs the impulse response as (1 − α)[1, α^i, α^{2i}, ..., α^{(L−1)i}], but the index i used in the vector entries is not defined by the algorithm input; it appears to be a typo for α.
  4. [Notation] The notation for the matrix of impulse responses is inconsistent: Theorem 1 writes µL(α1, . . . , αh), while Algorithm 2 and Appendix A.3 use µL(α1:h). Unify the notation for readability.
  5. [Table 1] The column header 'Runtime with K, T= O(L)' is ambiguous, and the formula 'Standard Conv (T K+ T log T + K^2)k' should use parentheses to make the grouping of terms unambiguous.
  6. [Abstract] The phrase 'first provable method' should be qualified in light of the conditional nature of Theorem 1; the comparison with Laughing Hyena in Section 2 is otherwise fair and appropriately drawn.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction found; Theorem 1 is a conditioning-dependent interpolation bound whose λmax premise is an explicit unproven limitation, not a fitted quantity reinserted as a prediction.

full rationale

The claimed derivation is not circular. In Appendix A.3, Theorem 1's proof uses Lemma 11.3 of [13] only to establish that each geometric impulse response µL(α_j) has a spectral-coefficient row m_j with ||m_j^T Φ1:k − µL(α_j)|| ≤ c e^{-k/log L}; this is the prior spectral-filtering approximation theorem, not the target statement. From there, ||E||_1 ≤ c h e^{-k/log L} and Holder's inequality yield ||Φ1:k − M†µ|| ≤ λmax c h e^{-k/log L}. The algebra M Φ ≈ µ ⇒ M† µ ≈ Φ is a linear-algebra consequence of how M is defined, not a reinsertion of a fitted quantity as a prediction: M is built from independent random α_i and their spectral representations, and the theorem's conclusion is a conditioning-dependent interpolation bound. The main caveat is explicit: 'We note that it is not immediate to upper bound λmax' (Section 5.1), and Section 5.3 makes the same premise explicit by saying 'Accepting that hλmax is O(1) for h ≫ k, as justified in Appendix A.2.' An empirically unproven premise is a correctness risk, not a circular step. The self-citations to [2, 13, 14, 25] are prior parameter-free theorems with stated assumptions that do not include the distilled result; under the review rules these constitute real evidence and do not raise circularity. The practical Algorithm 3 directly minimizes the reconstruction error ||Φ1:k − fM Ψ_sub||²_F, so Table 2's 'identical' accuracy is a fitting demonstration rather than an out-of-sample prediction derived from Theorem 1; again, no fitted parameter is renamed as a prediction. Conclusion: no significant circularity, with score 2 rather than 0 only because the central bound imports a substantial self-cited spectral-filtering lemma and depends on an unproved conditioning premise; neither is a circular reduction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two kinds of external support: the spectral filtering guarantee from prior work (which provides the m_j coefficients) and a well-conditioned coefficient matrix M. The latter is only validated empirically. The free parameters k, h, and the alpha sampling distribution are chosen by hand in the experiments; no new entities are introduced.

free parameters (4)
  • k (number of spectral filters) = 24 in experiments
    Number of spectral filters used in STU; appears in error bound e^{-k/log L}; chosen to balance approximation and compute (Appendix A.14).
  • h (LDS state dimension) = 80, 100, 160, 200, 400, 800 in experiments
    Dimension of the distilled LDS; controls approximation quality and per-token compute; chosen to achieve small lambda_max and low filter reconstruction error (Sections 5.3, 6).
  • alpha sampling distribution = near-uniform with oversampling near +-1 (Figure 6)
    Random scalars for the geometric impulse responses; the distribution is a design choice that affects the conditioning of M and hence lambda_max (Appendix A.2).
  • h_start (initial subset size in Algorithm 3) = varied (Figure 9); 10^6 trials per subset
    Initial number of 1D-LDS components selected greedily; affects reconstruction error and final state dimension (Appendix A.10).
assumptions (4)
  • domain assumption Spectral filtering lemma (Lemma 11.3 of [13]): for any alpha in [0,1] and any L, the impulse response mu_L(alpha) is approximated in the spectral basis with error c e^{-k/log L}.
    Used in Appendix A.3 to assert existence of the coefficient vectors m_j. This is a published result from the authors' prior work, not proved in this paper.
  • domain assumption A is symmetric real with ||A|| <= 1, so it can be diagonalized and its eigenvalues lie in [-1,1].
    The whole framework of spectral filtering and the STU is restricted to symmetric LDS with bounded norm (Sections 4.1, 4.2).
  • standard math The Vandermonde matrix of distinct alpha_i has full rank.
    Used in Appendix A.3 to argue that mu_L(alpha_1:h) has full rank; true for distinct alpha_i, but the step from that to full rank of the coefficient matrix M is where the argument is incomplete.
  • ad hoc to paper The Penrose-Moore pseudo-inverse of M has bounded largest singular value lambda_max, with h*lambda_max = O(1) for h sufficiently large.
    This is not proved; it is only supported by experiments in Appendix A.2. It is a load-bearing premise for the usefulness of Theorem 1 and for the LDS-to-LDS distillation claim in Section 5.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpectraLDS: Provable Distillation for Linear Dynamical Systems." pith.science (2026). https://pith.science/paper/XNXKLTKU

@misc{pith2026250517868,
  author       = {Pith},
  title        = {Pith review of: SpectraLDS: Provable Distillation for Linear Dynamical Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNXKLTKU}},
  note         = {Machine review of arXiv:2505.17868}
}
read the original abstract

We present the first provable method for identifying symmetric linear dynamical systems (LDS) with accuracy guarantees that are independent of the systems' state dimension or effective memory. Our approach builds upon recent work that represents symmetric LDSs as convolutions learnable via fixed spectral transformations. We show how to invert this representation, thereby recovering an LDS model from its spectral transform and yielding an end-to-end convex optimization procedure. This distillation preserves predictive accuracy while enabling constant-time and constant-space inference per token, independent of sequence length. We evaluate our method, SpectraLDS, as a component in sequence prediction architectures and demonstrate that accuracy is preserved while inference efficiency is improved on tasks such as language modeling.

Figures

Figures reproduced from arXiv: 2505.17868 by the authors.

Figure 1
Figure 1. Comparison of SpectraLDS and other methods learning an arbitrary symmetric LDS with [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Fit of Spectral Filters by an LDS of state dimension 80 where x-axis represents the time [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Runtime for generating sequences of increasing length across STU implementations. The [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Largest Singular Value as we increase h [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: λmax · h as we increase h. A.3 Analysis of Theorem 1 Proof of Theorem 1. Let µL(α) = (1 − α) [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Distribution of α Moreover, as the optimization problem is convex in mj and Lipschitz continuous, FindSpectralRep￾resentation will return such mj . Let M ∈ R h×k be the matrix whose rows are mj ∈ R k : M =   − m1− − m2− . . . − mh−   . Let E = M Φ1:k − µL(α1, .…
Figure 7
Figure 7. Figure 7: Visualization of the positive spectral filters and their approximation using an LDS with state [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Distributions of Eigenvalues for LDS trained to predict first 24 Spectral Filters. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Reconstruction error of spectral filters as a function of LDS state dimension. Each curve [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: A symmetric linear dynamical system with input dimension [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 10
Figure 10. Figure 10: Comparison of learning performance on synthetic LDS tasks between SpectraLDS and [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Autoregressive Inference Time (ms) across model architectures. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: FlashSTU architecture [22]. A.13.2 Implemention Efficiency Using the setup of [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spectral Distillation: From Nonlinear Dynamics to Linear State-Space Models

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A convex spectral learner followed by spectral-to-LDS distillation extracts compact linear state-space predictors from nonlinear dynamics with a provable one-step error decomposition.

Reference graph

Works this paper leans on

39 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Futurefill: Fast generation from convolutional sequence models, 2024

    Naman Agarwal, Xinyi Chen, Evan Dogariu, Vlad Feinberg, Daniel Suo, Peter Bartlett, and Elad Hazan. Futurefill: Fast generation from convolutional sequence models, 2024

    2024

  2. [2]

    Spectral state space models, 2024

    Naman Agarwal, Daniel Suo, Xinyi Chen, and Elad Hazan. Spectral state space models, 2024

    work page 2024

  3. [3]

    Anderson and John B

    Brian D.O. Anderson and John B. Moore. Linear Optimal Control. Prentice-Hall, 1971

    work page 1971

  4. [4]

    The condition number of real vandermonde, krylov and positive definite hankel matrices

    Bernhard Beckermann. The condition number of real vandermonde, krylov and positive definite hankel matrices. Numerische Mathematik, 85(4):553–577, 2000

    work page 2000

  5. [5]

    Learning long-term dependencies with gradient descent is difficult

    Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166, 1994

    work page 1994

  6. [6]

    Dynamic Programming and Optimal Control, volume 1

    Dimitri P Bertsekas. Dynamic Programming and Optimal Control, volume 1. Athena Scientific, 2005

    work page 2005

  7. [7]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR) , 2024

    2024

  8. [8]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

Show all 39 references
  1. [9]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024

    2024

  2. [10]

    Efficiently modeling long sequences with structured state spaces, 2022

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022

    2022

  3. [11]

    On the parameterization and initialization of diagonal state space models, 2022

    Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models, 2022

    2022

  4. [12]

    Hazan, H

    E. Hazan, H. Lee, K. Singh, C. Zhang, and Y . Zhang. Spectral filtering for general linear dynamical systems, 2018

    2018

  5. [13]

    Introduction to online nonstochastic control

    Elad Hazan and Karan Singh. Introduction to online nonstochastic control. arXiv preprint arXiv:2211.09619, 2022

    2022 arXiv

  6. [14]

    Learning linear dynamical systems via spectral filtering, 2017

    Elad Hazan, Karan Singh, and Cyril Zhang. Learning linear dynamical systems via spectral filtering, 2017

    2017

  7. [15]

    B. L. Ho and R. E. Kalman. Effective construction of linear state-variable models from input/output functions. Automatica, 3(1):49–59, 1966

    1966

  8. [16]

    Huang, S

    A. Huang, S. Subramanian, J. Sum, K. Almubarak, S. Biderman, and S. Rush. The annotated transformer, 2022

    2022

  9. [17]

    I. W. Jamaludin, N. A. Wahab, N. S. Khalid, S Sahlan, Z. Ibrahim, and M F. Rahmat. N4sid and moesp subspace identification methods. In 2013 IEEE 9th International Colloquium on Signal Processing and its Applications, pages 140–145, 2013

    2013

  10. [18]

    Jer-Nan Juang and Richard S. Pappa. An eigensystem realization algorithm for modal parameter identification and model reduction. Journal of Guidance, Control, and Dynamics, 8(5):620–627, 1985

    1985

  11. [19]

    Linear Systems

    Thomas Kailath. Linear Systems. Prentice-Hall, 1980

    1980

  12. [20]

    A new approach to linear filtering and prediction problems

    Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82.1:35–45, 1960

    1960

  13. [21]

    S. Y . Kung. A new identification and model reduction algorithm via singular value decom- positions. In Proc. 12th Asilomar Conf. Circuits, Systems and Computers , pages 705–714, 1978. 10

    1978

  14. [22]

    Y . I. Liu, W. Nguyen, Y . Devre, E. Dogariu, A. Majumdar, and E. Hazan. Flash stu: Fast spectral transform units, 2024

    2024

  15. [23]

    An introduction to observers

    David Luenberger. An introduction to observers. IEEE Transactions on Automatic Control , 16(6):596–602, 1971

    1971

  16. [24]

    Mega: Moving average equipped gated attention, 2023

    Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention, 2023

    2023

  17. [25]

    Provable length generalization in sequence prediction via spectral filtering, 2024

    Annie Marsden, Evan Dogariu, Naman Agarwal, Xinyi Chen, Daniel Suo, and Elad Hazan. Provable length generalization in sequence prediction via spectral filtering, 2024

    2024

  18. [26]

    Fu, Hermann Kumbong, Rom N

    Stefano Massaroli, Michael Poli, Daniel Y . Fu, Hermann Kumbong, Rom N. Parnichkun, Aman Timalsina, David W. Romero, Quinn McIntyre, Beidi Chen, Atri Rudra, Ce Zhang, Christopher Ré, Stefano Ermon, and Yoshua Bengio. Laughing hyena distillery: Extracting compact recurrences fr...

    2023

  19. [27]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024

    2024

  20. [28]

    Orvieto, S

    A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences, 2023

    2023

  21. [29]

    Non-asymptotic identification of lti systems from a single trajectory

    Samet Oymak and Necmiye Ozay. Non-asymptotic identification of lti systems from a single trajectory. In 2019 American control conference (ACC), pages 5655–5661. IEEE, 2019

    2019

  22. [30]

    Pascanu, T

    R. Pascanu, T. Mikolov, and Y . Bengio. On the difficulty of training recurrent neural networks, 2013

    2013

  23. [31]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017

    2017

  24. [32]

    Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y . Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models, 2023

    2023

  25. [33]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022

    2022

  26. [34]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. 11 A Appendix A.1 Experimental details and notation All experiments were performed on Nvidia H100-80GB GPUs in PyTorch...

    2017

  27. [35]

    Generating and Fitting Random LDSs. We first sample many random (a, b, c) system parame- ters to produce a large collection of one-dimensional LDSs: (1D-LDS) xt = a xt−1 + b ut, y t = c xt, where a ∈ R satisfies |a| ≤1. In particular, we skew toward generating LDSs with a matr...

  28. [36]

    For each of the retained 1D-LDSs, we explicitly record its impulse response, ψ(i) ∈ RL, as the vector ψ(i)[t] = c at−1 b for t = 1,

    Collecting LDS Impulses and STU Weights. For each of the retained 1D-LDSs, we explicitly record its impulse response, ψ(i) ∈ RL, as the vector ψ(i)[t] = c at−1 b for t = 1, . . . , L. Stacking these impulse responses row by row yields a matrix Ψ =   − ψ(1)− − ψ(2)− ... −...

  29. [37]

    We wish to project the STU filters on the basis Ψ of LDS impulses in RL

    Subset Selection and Pseudoinverse Next, we use matrices Ψ and Θ to approximate the target STU filters Φ1:k. We wish to project the STU filters on the basis Ψ of LDS impulses in RL. From the analysis in Section A.3, we look for a matrix fM such that Φ1:k ≈ fM Ψ. We have shown ...

  30. [38]

    Once we have a suitable subset, we refine the matrix fM = Θ† sub by gradient descent: fM = argminfM Φ1:k − fM Ψsub 2 F

    Gradient-Based Fine-Tuning. Once we have a suitable subset, we refine the matrix fM = Θ† sub by gradient descent: fM = argminfM Φ1:k − fM Ψsub 2 F . Since Ψsub and Φ1:k are both fixed, this is a simple least-squares objective. Empirically, we often observe an approximately 1.4...

  31. [39]

    Finally, we merge the chosen 1D-LDSs into a single higher- dimensional discrete-time system

    Combining Into a Single LDS. Finally, we merge the chosen 1D-LDSs into a single higher- dimensional discrete-time system. In particular, suppose our final subset Ψsub includes h different 1D systems, with parameters {(ai, bi, ci)}h i=1. The gradient refined matrix fM helps det...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.