Pith. sign in

REVIEW 3 major objections 4 minor 24 references

Spectral Distillation: From Nonlinear Dynamics to Linear State-Space Models

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that nonlinear dynamics can be compressed into a compact explicit linear state-space model by convex spectral learning followed by distillation, with the prediction error split between a learning term and an…

desk verdict A clean but over-sold composition of OSF and SpectraLDS: the end-to-end guarantee is new, yet the 'exponentially small' distillation term rests on an unproven empirical bound on lambda_max(F), so the fully provable claim is not yet earned. read the letter →

arxiv 2608.05416 v1 pith:2AUY6NRW submitted 2026-08-05 cs.LG

Liane Galanti , Devan Shah , Shlomo Fortgang , Elad Hazan This is my paper · ORCID
classification cs.LG MSC 68Q3293B30
keywords spectralfilteringlineardynamicalsystemsLuenbergerobservernonlinearsystemidentificationconvexlearningdistillationstate-spacemodelsbehaviorcloning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims there is a provably reliable route from raw observations of a nonlinear dynamical system to a compact explicit linear state-space model, avoiding direct non-convex system identification. The route has two stages: Observation Spectral Filtering (OSF) learns an implicit convolutional predictor in a convex feature space, and spectral-to-LDS distillation converts that predictor into a standard recurrent linear dynamical system. The main theorem bounds average one-step prediction error by the OSF learning error, which scales with the Luenberger complexity of the best observer, plus a distillation error that decays exponentially in the number of spectral filters. The bound is dimension-free: it does not depend on the latent dimension of the nonlinear system. This matters because non-convex identification is fragile, while convex learning followed by distillation could make compact recurrent models usable for nonlinear prediction and control.

What carries the argument

The argument is carried by approximating the top $k$ spectral filters, the leading eigenvectors of the length-$L$ Hankel spectral basis, with $h$ scalar geometric modes. SpectraLDS does this by choosing modes $\alpha_i$ and a mixing matrix $F$ satisfying $\Phi_{1:k} \approx F \mu_L(\alpha_{1:h})$, so each spectral feature becomes a linear combination of recurrent states $z_{t+1} = \alpha_i z_t + s_t$. This replacement turns the OSF convolutional predictor into an explicit LDS while keeping the filter approximation error exponentially small. The other load-bearing object is the Luenberger observer complexity $Q_\star$, which encodes how well the best observer's closed-loop matrix can be diagonalized and how large its spectral gap is; it controls the learning term rather than the hidden-state dimension.

What would settle it

Run the SpectraLDS algorithm with $k = \Theta(\log^2 T)$ filters and $h = 4k$ modes over increasing $T$ and numerically measure $\lambda_{\max}(F)$; if the value grows unboundedly with $k$ or $T$, the distillation term in Theorem 3.1 is not exponentially small and the claimed guarantee becomes vacuous. A second check would be to fit the pipeline on a nonlinear system whose discretized linear lifting is unobservable and see whether the bound still holds in practice.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that two previously separate tools compose into one end-to-end guarantee. With $k = \Theta(\log^2 T)$ spectral filters and $h$ recurrent modes with $k \le h \le 4k$, the distilled LDS satisfies an average prediction bound of the form $\widetilde{O}\left( \frac{Q_\star^2 \log(Q_\star) \log^7 T}{\sqrt{T}} + \lambda_{\max}(F)^2 \exp\left(-\frac{2k}{\log T}\right) \right)$. The first term is the OSF regret bound, governed by the Luenberger complexity $Q_\star$ of the best linear observer; the second is the cost of replacing spectral-filter convolutions with recurrent geometric modes. The resulting LDS has $O((d_{\mathrm{in}} + d_{\mathrm{out}})(h+m))$ memory and per-step cost, and its guarantee does not depend on the potentially enormous dimension of the discretized linear lifting used in the proof.

Load-bearing premise

The central guarantee requires that a certain norm of the matrix that maps spectral filters to recurrent modes stays bounded as the number of modes grows; the paper cites an empirical observation for this, but does not prove it, and if the norm grows the exponentially small distillation term is no longer small.

Editorial extensions

If this is right

  • A practitioner can obtain a deployable recurrent LDS from nonlinear trajectories using only convex optimization, with a finite-sample guarantee instead of a heuristic fit.
  • Systems whose best linear observer has low Luenberger complexity are learnable and simulatable at cost independent of their latent nonlinear dimension.
  • The exponential distillation term means adding $\Theta(\log^2 T)$ spectral filters can drive the conversion loss below any target, provided the SpectraLDS pseudoinverse norm stays bounded.
  • At equal parameter budgets, the distilled model should match or beat directly trained LDS baselines because it inherits the convex predictor's optimization advantage.
  • The same train-then-distill structure applies to both symmetric and non-symmetric LDS targets when negative spectral filters are included.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's bound stops at one-step teacher-forced error; a natural extension is to ask whether the one-step bound plus Lipschitz dynamics yields rollout error that grows only with horizon rather than exponentially.
  • Because the distillation term is controlled by $\lambda_{\max}(F)$, a proof that bounds that quantity, rather than the cited empirical observation, would make the theorem fully self-contained; choosing the $\alpha_i$ sampling distribution is the obvious avenue.
  • The train-then-distill recipe could transfer to structured state-space architectures, where the spectral predictor could serve as a provable initialization rather than the final model.
  • The MuJoCo results, in which students are trained teacher-forced but evaluated closed-loop, suggest distillation may act as an implicit regularizer under distribution shift; that is a testable hypothesis beyond the paper's open-loop guarantee.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a two-stage pipeline for learning a compact linear state-space model from nonlinear dynamical data: first run Observation Spectral Filtering (OSF), a convex method that learns a spectral predictor competing with the best linear observer, then apply SpectraLDS distillation to convert the spectral predictor into an explicit recurrent LDS. Theorem 3.1 decomposes the average squared prediction error into the OSF learning term, governed by the Luenberger complexity Q*, and a distillation term lambda_max(F)^2 exp(-2k/log T). The paper reports experiments on linear LDS benchmarks and MuJoCo behavior cloning, where the distilled model is compared with directly trained LDS baselines. The central claim is that this is the first end-to-end provable method for extracting a best-in-hindsight LDS representation of a nonlinear system through convex learning followed by provable distillation.

Significance. If the theorem's non-vacuousness were fully established, the paper would make a valuable contribution: it would give a dimension-free, convex, end-to-end method for obtaining an explicit recurrent LDS from nonlinear observations, with a clear separation between the learning error and the representation error. The proof is simple and clean, and the explicit recurrent realization is a useful algorithmic contribution. The paper also runs careful experiments, including a distillation-fidelity check in Appendix E showing that the trained STU and its SpectraLDS conversion have nearly identical test MSE. However, the advertised 'provable' guarantee is currently conditional on an unproven empirical bound on lambda_max(F), and the experimental section tests a simplified version of the full algorithm.

major comments (3)
  1. [Theorem 3.1 and Appendix A, Step 2 (Eq. A.3)] The central non-vacuousness of the distillation term rests on the product h*lambda_max(F) being bounded, but the proof only cites an empirical observation from [17, Appendix A.2]. Since lambda_max(F) is the largest singular value of a Moore-Penrose pseudoinverse and no lower bound on sigma_min of the geometric response matrix is supplied, nothing in the manuscript rules out lambda_max(F) growing with k or T, in which case lambda_max(F)^2 exp(-2k/log T) would not be small. Section 5 itself concedes that the current bound is dominated by the pseudoinverse norm and that sharper guarantees on lambda_max(F) are open. To make the theorem a genuine guarantee, the authors should either prove a quantitative bound on lambda_max(F) (or on the relevant product) or state such a bound as an explicit assumption in Theorem 3.1, and accordingly soften the claim of an unconditional, exponentially small distillation term.
  2. [Section 4 and Appendices C-D] The experiments do not directly test the theorem's guarantee. Section 4 states that all experiments use the STU/OSF predictor without the recurrent connections J_r and P_r (i.e., n=0), whereas Algorithm 1 and Theorem 3.1 include these terms. The MuJoCo evaluation is closed-loop rollout reward, which Section 5 explicitly says is not covered by the one-step teacher-forced bound, and the linear-LDS experiments are teacher-forced but also use n=0. Appendix E verifies distillation fidelity, but it does not close this gap. The empirical section should be framed as validation of a special case of the pipeline, or experiments with n>0 should be added.
  3. [Abstract, Section 1.1, and Section 5] The distillation term is repeatedly described as exponentially small, but with the stated choice k = Theta(log^2 T) it equals exp(-2k/log T) = T^{-Theta(1)}, which is only polynomially small in the horizon T. The decay is exponential in the number of filters k (equivalently, in the LDS state dimension divided by log T), not in T. Moreover, since the constant in Theta(log^2 T) is unspecified, the distillation term may even be larger than the OSF term T^{-1/2} for large T. Please qualify the language (e.g., 'exponential in k') and state the explicit dependence on the constant in the k = Theta(log^2 T) choice.
minor comments (4)
  1. [Section 4, first paragraph] The statement 'In these experiments, we consider the STU and OSF without the recurrent connections J_r and P_r (i.e., n=0)' should appear earlier and more prominently, since it means the experiments evaluate a restricted form of the algorithm presented in Section 3.
  2. [Table 1 caption] The caption says 'STU rows report the SpectraLDS-distilled recurrent predictor,' but the table's Model column also lists 'STU+ONS'; please clarify that the ONS row is also the distilled model.
  3. [Appendix A, Step 3] The proof is written for scalar outputs and says the vector-output case follows with operator norms; please add a few lines spelling out the norm definitions for the matrices M_j, N_j, since the main theorem is stated for general d_out.
  4. [Notation throughout Section 2 and Algorithm 1] The symbol m is used for the AR order in the theorem and for the history length in the experiments; please disambiguate, for instance by using n for the AR order in the main text consistently with Algorithm 1.

Circularity Check

1 steps flagged · score 4.0 of 10

The headline 'exponentially small' distillation term is supported only by an empirical λmax(F) bound cited from the authors' own SpectraLDS paper; the composition is not circular, but the provable-pipeline claim is not self-contained.

  1. self citation load bearing [Section 3, Theorem 3.1; Appendix A, Step 2, Eq. (A.3)]
    "SpectraLDS observes empirically that λmax(F) is bounded and decreasing in h [17, Appendix A.2]. ... As shown in [17, Appendix A.2], the product h λmax(F) is empirically bounded for the sampling distribution over αi used by Algorithm 2."

    The advertised exponentially-small distillation term λmax(F)^2 exp(-2k/log T) is small only under an unproven bound on λmax(F). The proof imports that bound from the authors' own prior work [17] instead of proving it: Step 2 bounds η_{k,h}(L) by h λmax(F) exp(-k/log L), and then asserts the h λmax(F) product is 'empirically bounded' in [17, Appendix A.2], a paper sharing authors with this one. The conclusion itself concedes that sharper guarantees on λmax(F) are open and that the current bound is dominated by the pseudoinverse norm. Thus the central 'provable' guarantee rests on a self-cited empirical observation, making the exponential-smallness claim non-self-contained. This is not a fitted-value reduction by construction, so it is partial rather than full circularity.

full rationale

The derivation chain of Theorem 3.1 is a composition of two prior results: the OSF learning bound from [1] (Eq. A.1) and the SpectraLDS filter-approximation bound from [17] (Eq. A.3), combined by a triangle inequality. That composition is logically sound and does not define the conclusion in terms of its inputs; the distillation-error term is genuinely controlled by the filter approximation error η_{k,h}(L). The single load-bearing self-reference is the treatment of the empirical bound on λ_max(F) as if it established the smallness of the distillation term. [17] is overlapping-author prior work, and its Appendix A.2 observation is cited but not proved; the paper's conclusion explicitly leaves sharper λ_max(F) guarantees open. The observability and discretized-lifting assumption (Appendix B) is stated as an explicit hypothesis, so it is a condition rather than hidden circularity. The empirical sections compare the distilled LDS against baselines and report distillation fidelity, so the experimental component is not circular. Because the theorem retains independent content but uses a self-cited empirical bound to support the headline exponential-smallness claim, a score of 4 is appropriate.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central theorem imports both the OSF bound and the SpectraLDS approximation bound from the authors' prior work, and it additionally relies on an empirical observation that h * lambda_max(F) is bounded to claim exponential decay. The observability of the discretized lifting is a strong structural assumption that is likely violated for many real-world partial-observation systems. No new entities are introduced beyond the mathematical constructions already present in the cited papers.

free parameters (1)
  • lambda_max(F) = not reported; empirically observed as bounded
    The distillation term's exponential decay depends on the largest singular value of the SpectraLDS pseudoinverse matrix. The paper cites an empirical observation from [17, Appendix A.2] rather than a theoretical bound, so the strength of the guarantee is conditional on this quantity.
assumptions (5)
  • domain assumption The discretized linear lifting (A', C') is observable
    Appendix B assumes the lifted grid-based LDS is observable so that a finite Luenberger observer exists and Q* is finite. This can fail for systems with non-injective observations or coarse discretization.
  • standard math The OSF bound of [1, Theorem 3.1] is valid
    The proof imports the OSF regret bound as a black box; it is not re-derived in this paper.
  • standard math The SpectraLDS approximation bound of [17, Theorem 1] is valid
    The proof imports the filter-approximation guarantee from [17] as a black box.
  • ad hoc to paper Empirical boundedness of h * lambda_max(F)
    The paper relies on an empirical observation from [17, Appendix A.2] that the product h * lambda_max(F) is bounded. No theorem is given, so the claimed exponential decay is conditional.
  • domain assumption Bounded inputs, states, outputs and 1-Lipschitz f, g
    Assumptions 1 and 2 of Theorem 3.1, inherited from [1]. Standard for the regret analysis but may not hold for many real-world control systems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spectral Distillation: From Nonlinear Dynamics to Linear State-Space Models." pith.science (2026). https://pith.science/paper/2AUY6NRW

@misc{pith2026260805416,
  author       = {Pith},
  title        = {Pith review of: Spectral Distillation: From Nonlinear Dynamics to Linear State-Space Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2AUY6NRW}},
  note         = {Machine review of arXiv:2608.05416}
}
read the original abstract

Can nonlinear dynamical systems be learned through a compact linear state-space representation, without directly solving a non-convex system-identification problem? We give a provable pipeline for doing so. Starting from observations of an unknown nonlinear dynamical system, we first learn an implicit spectral predictor using Observation Spectral Filtering (OSF), a convex method that competes with the best linear observer for the system. We then apply spectral-to-LDS distillation to convert this predictor into an explicit recurrent linear dynamical system. Our main theorem shows that the average prediction error of the distilled LDS decomposes into an exponentially-small distillation term and the OSF learning term governed by the Luenberger complexity of the best observer. The guarantee is dimension-free: it depends on observer complexity rather than on the latent dimension needed to represent the nonlinear system. To our knowledge, this yields the first end-to-end provable method for extracting a best-in-hindsight LDS representation of nonlinear dynamics through convex learning followed by provable distillation. Experiments on linear LDS benchmarks and MuJoCo behavior cloning show that the train-then-distill pipeline produces compact LDS predictors that match or outperform directly trained baselines.

Figures

Figures reproduced from arXiv: 2608.05416 by the authors.

Figure 1
Figure 1. Test MSE versus epoch on the long-memory LDS task. Curves are geometric means over [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Test MSE versus epoch on linear LDS prediction at spectral radius [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Evaluation reward during behavior cloning on six MuJoCo tasks. Curves show mean and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Test MSE versus epoch for the STU predictor (dashed) and the SpectraLDS-distilled [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Test MSE versus epoch for STU (blue) and tensor-dot STU (red), both with [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 20 canonical work pages

  1. [17]

    D. Shah, S. Fortgang, S. Druchyna, and E. Hazan. SpectraLDS: Provable distillation for linear dynamical systems.arXiv preprint arXiv:2505.17868, 2025

    work page Pith review arXiv 2025

  2. [1]

    Dogariu, A

    E. Dogariu, A. Brahmbhatt, and E. Hazan. Universal learning of nonlinear dynamics.arXiv preprint arXiv:2508.11990, 2025

    arXiv 2025

  3. [2]

    Gu and T

    A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    arXiv 2023

  4. [3]

    A. Gu, K. Goel, and C. R ´e. Efficiently modeling long sequences with structured state spaces. InInternational Conference on Learning Representations, 2022

    work page 2022

  5. [4]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018

    work page 2018

  6. [5]

    Hardt, T

    M. Hardt, T. Ma, and B. Recht. Gradient descent learns linear dynamical systems. InJournal of Machine Learning Research, volume 19, pages 1–44, 2018

    work page 2018

  7. [6]

    Hazan, A

    E. Hazan, A. Agarwal, and S. Kale. Logarithmic regret algorithms for online convex optimiza- tion.Machine Learning, 69(2):169–192, 2007

    work page 2007

  8. [7]

    Hazan, H

    E. Hazan, H. Lee, K. Singh, C. Zhang, and Y . Zhang. Spectral filtering for general linear dynamical systems. InAdvances in Neural Information Processing Systems, volume 31, pages 4634–4643, 2018

    work page 2018

Show all 24 references
  1. [8]

    Hazan, K

    E. Hazan, K. Singh, and C. Zhang. Learning linear dynamical systems via spectral filtering. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  2. [9]

    Kostic, P

    V . Kostic, P. Novelli, A. Maurer, C. Ciliberto, L. Rosasco, and M. Pontil. Learning dynamical systems via koopman operator regression in reproducing kernel hilbert spaces. InAdvances in Neural Information Processing Systems, 2022

    2022

  3. [10]

    Y . I. Liu, W. Nguyen, Y . Devre, E. Dogariu, A. Majumdar, and E. Hazan. Flash STU: Fast spectral transform units.arXiv preprint arXiv:2409.10489, 2024

    2024

  4. [11]

    Ljung.System Identification: Theory for the User

    L. Ljung.System Identification: Theory for the User. Prentice Hall, 2nd edition, 1999

    1999

  5. [12]

    Lusch, J

    B. Lusch, J. N. Kutz, and S. L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics.Nature Communications, 9(1):4950, 2018

    2018

  6. [13]

    I. Mezi ´c. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41(1-3):309–325, 2005

    2005

  7. [14]

    Orvieto, S

    A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resur- recting recurrent neural networks for long sequences. InInternational Conference on Machine Learning, 2023

    2023

  8. [15]

    Oymak and N

    S. Oymak and N. Ozay. Non-asymptotic identification of lti systems from a single trajectory. American Control Conference, pages 5655–5661, 2019

    2019

  9. [16]

    Sarkar and A

    T. Sarkar and A. Rakhlin. Near optimal finite time identification of arbitrary linear dynamical systems. InInternational Conference on Machine Learning, pages 5610–5618, 2019

    2019

  10. [18]

    Simchowitz, H

    M. Simchowitz, H. Mania, S. Tu, M. I. Jordan, and B. Recht. Learning without mixing: Towards a sharp analysis of linear system identification. InConference on Learning Theory, pages 439–473, 2018

    2018

  11. [19]

    J. T. H. Smith, A. Warrington, and S. Linderman. Simplified state space layers for sequence modeling. InInternational Conference on Learning Representations, 2023

    2023

  12. [20]

    Todorov, T

    E. Todorov, T. Erez, and Y . Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–

    2012

  13. [21]

    Tsiamis and G

    A. Tsiamis and G. J. Pappas. Finite sample analysis of stochastic system identification. In IEEE Conference on Decision and Control, pages 3648–3654, 2019

    2019

  14. [22]

    M. O. Williams, I. G. Kevrekidis, and C. W. Rowley. A data-driven approximation of the koopman operator: Extending dynamic mode decomposition.Journal of Nonlinear Science, 25(6):1307–1346, 2015

    2015

  15. [23]

    O. G. Younis, R. Perez-Vicente, J. U. Balis, W. Dudley, A. Davey, and J. K. Terry. Minari, Sept. 2024. A Proof of Theorem 3.1 The proof has two ingredients. First, the OSF guarantee gives a mean-squared prediction bound for the learned spectral predictor. Second, replacing the...

    2024

  16. [24]

    D.2 Teachers and trajectory data For each of the six environments we distill a single Soft Actor-Critic [4] teacher policy trained by the Minari team [23]

    of the MuJoCo environments [20]. D.2 Teachers and trajectory data For each of the six environments we distill a single Soft Actor-Critic [4] teacher policy trained by the Minari team [23]. Each teacher is rolled out forT= 2,000,000time steps in its environment using a fixed se...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.