Pith. sign in

REVIEW 7 cited by

On the Parameterization and Initialization of Diagonal State Space Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2206.11893 v2 pith:YASV6WIF submitted 2022-06-23 cs.LG

Albert Gu , Ankit Gupta , Karan Goel , Christopher Ré This is my paper · ORCID
classification cs.LG
keywords diagonalstatematrixmodelsinitializationmodelspacealmost
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

State space models (SSM) have recently been shown to be very effective as a deep learning layer as a promising alternative to sequence models such as RNNs, CNNs, or Transformers. The first version to show this potential was the S4 model, which is particularly effective on tasks involving long-range dependencies by using a prescribed state matrix called the HiPPO matrix. While this has an interpretable mathematical mechanism for modeling long dependencies, it introduces a custom representation and algorithm that can be difficult to implement. On the other hand, a recent variant of S4 called DSS showed that restricting the state matrix to be fully diagonal can still preserve the performance of the original model when using a specific initialization based on approximating S4's matrix. This work seeks to systematically understand how to parameterize and initialize such diagonal state space models. While it follows from classical results that almost all SSMs have an equivalent diagonal form, we show that the initialization is critical for performance. We explain why DSS works mathematically, by showing that the diagonal restriction of S4's matrix surprisingly recovers the same kernel in the limit of infinite state dimension. We also systematically describe various design choices in parameterizing and computing diagonal SSMs, and perform a controlled empirical study ablating the effects of these choices. Our final model S4D is a simple diagonal version of S4 whose kernel computation requires just 2 lines of code and performs comparably to S4 in almost all settings, with state-of-the-art results for image, audio, and medical time-series domains, and averaging 85\% on the Long Range Arena benchmark.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 73 citations worldwide. Full citation record

  1. No Free Lunch in Flow Surrogates under Time-Varying Boundary Conditions: A Two-Regime Study

    math.NA 2026-07 conditional novelty 6.0 of 10

    No single flow-surrogate architecture transfers from a boundary-driven Stokes film to a self-sustained Kármán wake; time treatment decides the winner and pointwise RMSE ranks the wrong models.

  2. Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A DP-means allocate-on-novelty cache matches full-attention associative recall while storing only distinct items, and a minimal novelty gate recovers the rule end-to-end.

  3. Explicit Context Reasoning with Supervision for Visual Tracking

    cs.CV 2025-07 conditional novelty 6.0 of 10

    RSTrack supervises a Mamba-based state reasoning module with true target states, improving visual tracking accuracy on six benchmarks.

  4. Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams

    cs.LG 2026-07 conditional novelty 5.5 of 10

    A Dirichlet-process novelty cache plus recurrence matches full-attention coupling at about half the attended tokens on long redundant streams, with inspectable retained slots.

  5. Cumsum-Composable Phase Transport for Low-Cost Streaming Keyword Spotting

    cs.SD 2026-07 conditional novelty 5.0 of 10

    Finite-window unitary phase-transport layers whose recurrences collapse to cumulative sums match compact CNN baselines on Speech Commands and run with lower latency than a custom scan.

  6. Rivaling Transformers: Multi-Scale Structured State-Space Mixtures for Agentic 6G O-RAN

    cs.NI 2025-10 conditional novelty 5.0 of 10

    A 0.70M-parameter multi-scale state-space mixture predicts next-step RSRP on an O-RAN testbed with RMSE 0.29 dB and R2=0.993, running 3-10x faster than the tested Transformers.

  7. ACM-UNet: Adaptive Integration of CNNs and Mamba for Efficient Medical Image Segmentation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    ACM-UNet, a UNet variant combining pretrained CNN and Mamba backbones via lightweight adapters and a wavelet decoder module, reports 85.12% Dice on Synapse and 92.29% on ACDC.

Pith tools