REVIEW 7 cited by
HiPPO: Recurrent Memory with Optimal Polynomial Projections
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
A central problem in learning from sequential data is representing cumulative history in an incremental fashion as more data is processed. We introduce a general framework (HiPPO) for the online compression of continuous signals and discrete time series by projection onto polynomial bases. Given a measure that specifies the importance of each time step in the past, HiPPO produces an optimal solution to a natural online function approximation problem. As special cases, our framework yields a short derivation of the recent Legendre Memory Unit (LMU) from first principles, and generalizes the ubiquitous gating mechanism of recurrent neural networks such as GRUs. This formal framework yields a new memory update mechanism (HiPPO-LegS) that scales through time to remember all history, avoiding priors on the timescale. HiPPO-LegS enjoys the theoretical benefits of timescale robustness, fast updates, and bounded gradients. By incorporating the memory dynamics into recurrent neural networks, HiPPO RNNs can empirically capture complex temporal dependencies. On the benchmark permuted MNIST dataset, HiPPO-LegS sets a new state-of-the-art accuracy of 98.3%. Finally, on a novel trajectory classification task testing robustness to out-of-distribution timescales and missing data, HiPPO-LegS outperforms RNN and neural ODE baselines by 25-40% accuracy.
Forward citations
Cited by 7 Pith papers
-
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...
-
SketchMamba: A Lightweight State-Space Model for Joint Progressive Sketch Classification and Stroke Auto-Completion
Dense per-step class loss on a 1.55M-parameter causal Mamba unifies progressive sketch recognition (94.93% final Top-1, 0.706 progressive AUC) and stroke auto-completion on 58-class Quick, Draw!.
-
Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code
A survey of diffusion, world-model, and state-space architectures for code that frames them as routes past autoregressive bottlenecks toward System-2 coding agents.
-
Lag Operator SSMs: A Geometric Framework for Structured State Space Modeling
The authors derive discrete-time SSMs from a lag-operator inner product and show that a Legendre/exponential-warp instance reduces to HiPPO-LegS.
-
Rivaling Transformers: Multi-Scale Structured State-Space Mixtures for Agentic 6G O-RAN
A 0.70M-parameter multi-scale state-space mixture predicts next-step RSRP on an O-RAN testbed with RMSE 0.29 dB and R2=0.993, running 3-10x faster than the tested Transformers.
-
Elucidating the Design Space of Decay in Linear Attention
A controlled study of decay in linear attention finds median decay near 0.8 works best, vector decay generally beats scalar decay, and RoPE/TPE give little benefit for models with sub-unity decay.
-
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model
SSMLoRA inserts sparse low-rank adapters connected by a layer-wise state-space recurrence, reporting LoRA-comparable GLUE performance at roughly half the parameters, though with design limitations.
Discussion (0). Continue with ORCID to comment.