REVIEW 2 cited by
State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
State-space models have gained popularity in sequence modelling due to their simple and efficient network structures. However, the absence of nonlinear activation along the temporal direction limits the model's capacity. In this paper, we prove that stacking state-space models with layer-wise nonlinear activation is sufficient to approximate any continuous sequence-to-sequence relationship. Our findings demonstrate that the addition of layer-wise nonlinear activation enhances the model's capacity to learn complex sequence patterns. Meanwhile, it can be seen both theoretically and empirically that the state-space models do not fundamentally resolve the issue of exponential decaying memory. Theoretical results are justified by numerical verifications.
Forward citations
Cited by 2 Pith papers
-
DSSMs: State Space Models with Explicit Memory via Delay Differential Equations
Delay State Space Models augment diagonal SSMs with explicit delayed feedback, stable discrete parameterization, and FFT training, improving delayed-retrieval tasks and matching or beating S4D on most standard sequenc...
-
Adjoint sharding for very long context training of state space models
The paper derives an adjoint-based gradient sharding algorithm for SSMs and claims up to 3X memory reduction, but provides no experimental evidence for the central empirical claims.
Discussion (0). Continue with ORCID to comment.