Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

This paper claims that the memory behaviors of modern state space models can be rebuilt as explicit, interpretable orthogonal-polynomial mechanisms.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:02 UTC pith:DVF3VRGT

load-bearing objection A competent, cleanly written paper that adds five interpretable HiPPO extensions, but the abstract's claim about realizing modern SSM capabilities is too broad given the scalar time-warp limitation of Salience HiPPO and the thin synthetic evidence. the 4 major comments →

arxiv 2602.21340 v2 pith:DVF3VRGT submitted 2026-02-24 cs.LG

HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models

Jack Goffinet , Casey Hanks , David E. Carlson This is my paper
classification cs.LG
keywords HiPPOstate space modelsorthogonal polynomialsselective memoryassociative memorymultiscale representationVolterra seriesinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern sequence models keep a compressed state that is updated by learned, opaque dynamics; how they select, store, and recall history is hidden inside those dynamics. This paper tries to show that the same capabilities—input-dependent memory weighting, associative recall, multiscale time coverage, nonlinear interactions, and forecast-dependent memory geometry—can be realized explicitly within the HiPPO framework, where history is represented as coefficients of an orthogonal polynomial basis with a known measure. To that end it introduces five extensions (Volterra, Salience, Associative Memory, Multiscale, and Forecasting HiPPO), each exposing a specific mechanism, and demonstrates each on synthetic tasks. If correct, the paper establishes that the 'implicit' machinery of modern SSMs is not necessary for these capabilities: explicit, analyzable polynomial memory structures suffice, which matters for scientific and streaming applications where understanding what is remembered is as important as accuracy.

Core claim

The central claim is that capabilities typically associated with modern state space models can be realized through explicit, interpretable polynomial memory structures. HiPPO represents the past by projecting a signal onto orthogonal polynomials under a measure, with coefficients updated by a structured linear ODE. The paper shows five modifications: Volterra HiPPO puts nonlinear interactions into polynomial kernels in the readout; Salience HiPPO reshapes the history measure via a scalar time warp so memory is allocated to informative inputs; Associative Memory HiPPO adds a separate OP-based key-value bank with continuous addresses; Multiscale HiPPO embeds a continuum of timescales as a scal

What carries the argument

The load-bearing object is the HiPPO history representation: a signal's recent past is encoded as coefficients s(t) of an orthogonal polynomial basis defined by a measure ω(τ), updated online via a linear ODE ˙s = A s + b f. The paper's five extensions modify this substrate—the readout (Volterra kernels), the measure via a scalar time warp (Salience), the state space augmented with a separate OP key-value bank (Associative Memory), the state itself parameterized over inverse timescale (Multiscale), and the metric on histories induced by a forecasting map (Forecasting). The orthogonal polynomial basis is what keeps each mechanism explicit: coefficients correspond to named functionals of the p

Load-bearing premise

The load-bearing premise is that a single scalar salience signal multiplying the entire HiPPO dynamics can express the full range of selective memory that modern SSMs achieve with high-dimensional, input-dependent state updates; if real selectivity requires richer, coordinate-dependent gating, the Salience mechanism cannot capture it.

What would settle it

Give the model a task where two input channels must be remembered with different priorities at the same time step (e.g., attend to channel A while ignoring channel B at the same timestamp). Since Salience HiPPO scales all channels by the same scalar g(t), it cannot represent such channel-specific selectivity; a model with per-channel input-dependent gates would succeed. If the paper's mechanism fails this test while a gated SSM succeeds, the claim that scalar time-warping realizes modern selective memory is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claims hold, selective and associative memory behaviors in SSMs can be studied and controlled directly through the history measure and the OP basis, instead of being inferred from learned state transitions.
  • Salience HiPPO provides an interpretable recipe for input-dependent memory: a learned scalar warps time, which is equivalent to adapting the memory measure; this could enable principled design of attention-like gating in streaming settings.
  • Multiscale HiPPO shows a single linear system can represent a continuum of timescales with a fixed coefficient budget, so a model need not commit to a horizon a priori.
  • Forecasting HiPPO shows that the training objective itself selects memory geometry: different horizons induce different predictive memories, which can be visualized as leading eigenfunctions of the induced metric.
  • The Volterra HiPPO readout shows that nonlinear system identification can be done with interpretable kernels, with faster convergence than an MLP readout in the paper's experiment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The scalar time-warping equivalence in Salience HiPPO suggests a precise diagnostic for how 'selective' any SSM actually is: if a model's selectivity can be mimicked by a single warped time axis, it is not using richer gating; future work could test this by fitting a Salience HiPPO to a trained selective SSM's input-output map.
  • Inference: The OP associative memory's reproducing-kernel interference structure is a concrete model of crosstalk between stored associations; this could be used to predict memory failures in tasks with many overlapping write addresses, a testable quantitative prediction.
  • Inference: The Multiscale construction's decoupling via spectral decomposition of the scale-coupling matrix implies that its expressivity is equivalent to a bank of fixed-timescale HiPPO systems; so its value is queryability and parameter sharing, not new dynamics—a claim the paper itself nearly states, and that could be probed by comparing to a tuned bank with the same total dimension.
  • Inference: The paper leaves open whether these explicit mechanisms scale to real benchmarks; a natural extension would be to plug Volterra or Salience mechanisms into a modern SSM architecture and measure whether interpretability survives while performance holds.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper revisits the HiPPO framework and proposes five explicit memory mechanisms: Volterra HiPPO (polynomial nonlinear readouts), Salience HiPPO (scalar time-warping of the history measure), Associative Memory HiPPO (OP-based key-value memory with minimum-norm writes), Multiscale HiPPO (polynomial embedding of inverse timescale), and Forecasting HiPPO (predictive memory geometry Q=T^T W T). Each extension is paired with a synthetic experiment and interpretability diagnostics. The central claim is that capabilities usually implicit in learned SSMs—adaptive memory allocation, associative recall, multiscale coverage, and forecast-induced memory geometry—can be realized explicitly in polynomial memory structures.

Significance. If the claims hold, the paper provides a useful, transparent toolbox for interpreting memory in state space models, and the derivations are mostly elementary and checkable. The Volterra substitution, the time-warp identity, the associative-memory write/read construction, and the Q=T^T W T forecasting geometry are clean and clearly explained. I also credit the authors for explicitly disclaiming expressivity gains for Multiscale HiPPO (App. D.4). However, the breadth of the central claim exceeds what the five proof-of-concept experiments establish, especially for Salience HiPPO, where the mechanism is a scalar time reparameterization and is only tested on a task requiring temporal, not feature-selective, prioritization. The contribution is still valuable as an explicit-mechanism framework, but the claim should be calibrated to the demonstrated scope.

major comments (4)
  1. [Abstract, §3.2, Eq. (2)] The claim that Salience HiPPO realizes adaptive memory allocation "typically associated with modern SSMs" is broader than the mechanism. Eq. (2) multiplies the entire state dynamics by a single scalar g(t), and Eqs. (3)–(4) show this is equivalent to a standard HiPPO system on a reparameterized time axis. Consequently, all memory prioritization is temporal: there is no mechanism to gate different channels or input features differently at the same timestep, unlike the per-channel/input-dependent updates in Mamba-type models. The selective-copying task (App. C.2) interleaves informative and uninformative tokens at different times, so scalar time warping suffices; it never tests a task where salience differs across features within a single timestep. I recommend either adding a per-channel or coordinate-dependent salience extension and a task that requires within-timestep selectivity, or exp
  2. [§3.2, Eqs. (3)–(4), App. C.2] The time-warping equivalence treats g as an exogenous function of time. In the experiment, g is produced by a learned network that conditions on the current HiPPO state, so g is state-dependent. Then φ(t)=∫g(s)ds depends on the trajectory itself, and Eq. (4) is not a model-level equivalence to a fixed HiPPO system; it is a post hoc reparameterization of each realized trajectory. This does not invalidate the adaptive-memory idea, but the paper should clarify that the "equivalent standard HiPPO" statement is only exact for exogenous g, or reformulate the discussion as a state-dependent time warp.
  3. [App. C.3, Eq. (C.3)] The paper states that the associative-memory write solves a constrained minimum-norm problem that achieves the desired value at the key address, so that m_j(x_key)=y_t[j]. However, the reported update Δ = α(y−ŷ)k with α = g_write/(||k||^2+ε) does not exactly satisfy that constraint for ε>0. The exact solution of the stated constrained problem is Δ = g_write(y−ŷ)k/||k||^2. As written, the update is approximate. If ε is a numerical regularizer, the text should say so; otherwise the "closed-form minimum-norm write and minimal interference" claim is stronger than what is implemented.
  4. [§3.2 / App. C.2, discretization] Eq. (2) is a continuous-time equation, but the experiments are discrete-time. For a time-varying g(t), the exact zero-order-hold discretization is not the usual constant-g ZOH update; for piecewise-constant g within a step it is s_{t+1}=exp(Δt g_t A)s_t + (g_t A)^{-1}(exp(Δt g_t A)-I)b f_t. The appendix does not specify how g_t enters the discrete update actually used for training and evaluation, nor whether an Euler or exact scheme was used. Since the time-warp interpretation and the induced-measure visualizations depend on continuous-time integration, this implementation detail should be documented and the exact discrete update stated.
minor comments (4)
  1. [§3.2, Eq. (4)] In Eq. (4), the input should be written as f(φ^{-1}(t1)) rather than f(t1) unless a new symbol for the warped-time input is introduced.
  2. [§3.1] The sentence defining the polynomial sum says "i0, . . . , ik" but the sum indexes i1,...,ik; this is a typo.
  3. [Table 1] The table entries "bnbk" should be typeset as products b_n b_k (and similarly for the Leg-S entries), and the sign convention for Leg-T should be checked against the standard HiPPO definitions.
  4. [App. C.4] The multiscale base timescale τ0 is introduced but its numerical value in the experiments is never stated. Since the query mapping g(L)=τ0/L depends on it, the value should be reported.

Circularity Check

0 steps flagged

No significant circularity: each zoo extension is constructed from explicit definitions and fitted parameters are used for interpretability, not as independent predictions.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs. Volterra HiPPO defines a polynomial readout of the HiPPO state and then fits the Volterra coefficients to a synthetic system; the fitted coefficients are visualized, not used as an independent prediction. Salience HiPPO's time-warp equivalence (Eqs. 2–4) is an exact change-of-variables identity, and the salience signal is learned, with the induced history measure computed post hoc from the trained g(t). Associative Memory HiPPO constructs an explicit OP key-value bank with a closed-form minimum-norm write update; solving the synthetic recall task is the intended behavior of that construction, not a separate derived result. Multiscale HiPPO substitutes a polynomial expansion into the HiPPO dynamics to derive coupled linear dynamics, and Appendix D.4 explicitly acknowledges that this is no more expressive than a bank of HiPPO systems—an honest statement that prevents overclaiming. Forecasting HiPPO fits a reduced-rank map T and then defines Q=T^T T as the induced history metric; this is an interpretability map of the fitted object, not a second, independently predicted result. The paper does not relabel fitted quantities as predictions, and there are no load-bearing self-citations: the cited HiPPO and SSM works are external and not authored by the present paper's authors. All experiments fit their own parameters and report the resulting behavior, which is consistent with the stated goal of illustration and interpretability rather than independent validation. Therefore, no circular step is identifiable under the specified criteria.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central claims rest on small hand-constructed synthetic tasks and the standard HiPPO mathematics. Free parameters are modest and mostly task constants; the main unquantified inputs are tau0, rank d, and the Leg-S scale sigma. The paper does not introduce exotic entities; its 'invented' objects are interpretability constructs. No code or data is shipped, so the burden of proof sits on the appendix protocols and synthetic reproductions.

free parameters (5)
  • Leg-S timescale factor sigma (alpha=0.375) = 0.375
    Chosen by hand in C.1 to match the Wray-Green system's memory scale; rescales A,b with A<-A/sigma, b<-b/sigma.
  • Wray-Green system constants a,m,k,tau_max,alpha = a=2, m=0.3, k=0.08, tau_max=50, alpha=0.004 (scaled 4e-3)
    Taken from the benchmark, not fitted, but they determine the target the model must learn and are hand-set.
  • Multiscale HiPPO base timescale tau0 = not explicitly stated numerically
    Scales A in the multiscale dynamics; the paper uses it in q(L)=clip(tau0/L,0,1) but does not report tau0.
  • Forecasting HiPPO horizons H = H=4 and H=32
    Chosen by hand to define short vs long-horizon forecasters; the whole 'predictive memory' comparison depends on these choices.
  • Rank d of reduced-rank regression = d not reported in main text or appendix
    The predictive memory plots depend on the rank of T_d; no value is given.
axioms (5)
  • domain assumption Volterra series converge / truncating to first few kernels is adequate (Sec. 3.1).
    The readout relies on the target system admitting a finite low-order Volterra representation; the Wray-Green benchmark is chosen because it is exactly quadratic, so the claim of a generally useful representation is not established.
  • domain assumption The HiPPO state truncation f(t-tau) ≈ Σ s_n(t)P_n(tau) is accurate enough to support the Volterra readout (Eq. 3.1, C.1).
    Uses N=64 with a fixed timescale; error from truncation is not quantified and could affect the inferred kernels.
  • domain assumption The reproducing-kernel / minimum-norm write update is a sufficient model of associative binding (Eq. C.3).
    The write rule is exact for the chosen loss, but the paper assumes local bump interference in the OP kernel captures the content-based similarity needed for recall tasks.
  • standard math Jordan/matrix-exponential machinery for G=exp(J) is numerically well-conditioned (App. D.3).
    The construction relies on polynomial spectral theory (Favard's theorem, Jacobi matrices); conditions are standard but the paper does not report numerical conditioning of exp(J).
  • domain assumption The RBF-mixture signal model and streaming statistics give a valid estimate of the optimal linear predictor (C.5).
    Stationarity and the use of running averages for Σxx,Σyy,Σyx are assumed; no diagnostics on convergence of these statistics are shown.
invented entities (1)
  • 'predictive memory' and Q=T^T T as 'objective-induced history geometry' no independent evidence
    purpose: Proposes that a forecasting objective defines a metric on histories whose eigenfunctions reveal which past features are retained.
    This is an interpretive construct computed from the fitted T, not a new physical entity or falsifiable prediction; it is a visualization device.

pith-pipeline@v1.3.0-alltime-deepseek · 19536 in / 8429 out tokens · 82472 ms · 2026-08-02T21:02:01.306883+00:00 · methodology

0 comments
read the original abstract

Representing the past in a compressed, efficient, and informative manner is a central problem for systems trained on sequential data. The HiPPO framework, originally proposed by Gu & Dao et al., provides a principled approach to sequential compression by projecting signals onto orthogonal polynomial (OP) bases via structured linear ordinary differential equations. Subsequent works have embedded these dynamics in state space models (SSMs), where HiPPO structure serves as an initialization. Nonlinear successors of these SSM methods such as Mamba are state-of-the-art for many tasks with long-range dependencies, but the mechanisms by which they represent and prioritize history remain largely implicit. In this work, we revisit the HiPPO framework with the goal of making these mechanisms explicit. We show how polynomial representations of history can be extended to support capabilities of modern SSMs such as adaptive memory allocation and associative memory, while retaining direct interpretability in the OP basis. We introduce a unified framework comprising five such extensions, which we collectively refer to as a "HiPPO zoo." Each extension exposes a specific modeling capability through an explicit, interpretable modification of the HiPPO framework. The resulting models adapt their memory online and train in streaming settings with efficient updates. We illustrate the behaviors and modeling advantages of these extensions through a range of synthetic sequence modeling tasks, demonstrating that capabilities typically associated with modern SSMs can be realized through explicit, interpretable polynomial memory structures.

Figures

Figures reproduced from arXiv: 2602.21340 by Casey Hanks, David E. Carlson, Jack Goffinet.

Figure 1
Figure 1. Figure 1: A: A visualization of the time warping interpretation of Salience HiPPO. Salience HiPPO is equivalent to a standard HiPPO system under an invertible time warp φ. Consequently, the static measures in warped time (left; green, red, and yellow) correspond to dynamic measures in real time (bottom; green, red, and yellow). B: Online learning of a continuous time invariant system. The ground truth system (middle… view at source ↗
Figure 2
Figure 2. Figure 2: Salience HiPPO applied to a selective copying task. Top: HiPPO is tasked with remembering only informative tokens (shown here projected as colors). White points are uninformative. When given gray input tokens HiPPO is tasked with repeating each informative token in order. Second from the top: Salience HiPPO measure functions place importance on informative points, and not on uninformative points. Second fr… view at source ↗
Figure 3
Figure 3. Figure 3: Associative Memory HiPPO uses OP associative memory to solve an associative recall task. Top: Task schematic Middle: The system learns to read and write from consistent locations. Bottom: OP memory before and after a write operation. During the episode, associations between tokens in the first set and subsequent tokens in the second set must be stored online. At the WRITE timestep, the model is queried wit… view at source ↗
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A: Forecasting HiPPO schematic B: Forecasting HiPPO with a reduced rank linear forecasting map reveals different “predictive memories” under different forecasting horizons. C: Objective-dependent history geometries in the top eigenfunctions of Q, the predictive history metric, show broader features for the long-horizon objective. distinction is one of degree rather than kind: Forecasting HiPPO makes the ef… view at source ↗
Figure 6
Figure 6. Figure 6: The log-timescale Multiscale HiPPO variant consistently outperforms the basic and Jeffreys-style variants on the variable-horizon reconstruction experiment from Section 3.4. Single-scale HiPPO systems with timescales 101 , . . . , 104 are shown in light to dark blue for comparison. Shaded regions denote ± SEM for 64 independently sampled trajectories of length T = 3 × 104 . D.2. Jeffreys-Style Scale Embedd… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SHiPPO: Recurrent Memory with Transported Polynomial Projections

    cs.LG 2026-07 conditional novelty 7.0

    SHiPPO transports HiPPO coefficient memories via right actions into Sylvester dynamics, and diagnostics show this recovers order-sensitive memory changes that high-rank writes cannot.

Reference graph

Works this paper leans on

6 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [1]

    We refer the reader to the survey of Totik and the book of Chihara for more information on orthogonal polynomials (Totik, 2005; Chihara, 1978). Orthonormal Legendre polynomials on [0,1] .Let Pn(x) denote the standard Legendre polynomials on [−1,1] , defined by the recurrence (n+ 1)P n+1(x) = (2n+ 1)xP n(x)−nP n−1(x), P 0(x) = 1, P1(x) =x, and orthogonal w...

    2005

  2. [6]

    The solution follows from a Lagrange multiplier and yields the closed-form update Ct+1[j] =C t[j] +α t yt[j]−ˆyt[j] kt, α t = gwrite ∥kt∥2 2 +ε

    and a full write (gwrite = 1). The solution follows from a Lagrange multiplier and yields the closed-form update Ct+1[j] =C t[j] +α t yt[j]−ˆyt[j] kt, α t = gwrite ∥kt∥2 2 +ε . Because the basis is orthonormal, ∥kt∥2 2 =⟨k t, kt⟩ is well-conditioned. Interpreted in function space, Eq. (C.3) adds a localized bump proportional to the truncated reproducing k...

    2000

  3. [7]

    Leg-T corresponds to the truncated Legendre HiPPO system defined on a finite window, while Leg-S corresponds to the exponentially weighted (sliding) Legendre HiPPO system

    The nonlinear benchmark introduced by Wray & Green (1994) is a purely second-order V olterra system with a separable 11 HiPPO Zoo ω(τ)P n(τ)b n Ank Leg-T U[0,1]L n(τ) √2n+ 1    bnbk(−1)n−k+1, n > k bnbk, n=k 0, n < k Leg-S exp(−τ)L n(e−τ ) √2n+ 1    0, n < k −bnbk n+ 1 2n+ 1 , n=k −bnbk, n > k Table 1.Orthogonal Polynomial HiPPO systems used i...

    1994

  4. [1978]

    ISBN 0-677-04150-0. Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060,

    Pith/arXiv arXiv

  5. [2022]

    Flow- state: Sampling rate invariant time series forecasting

    Graf, L., Ortner, T., Wo´Ls ¸niak, S., Pantazi, A., et al. Flow- state: Sampling rate invariant time series forecasting. arXiv preprint arXiv:2508.05287,

    arXiv

  6. [2023]

    Orthogonal polynomials.arXiv preprint math/0512424,

    Totik, V . Orthogonal polynomials.arXiv preprint math/0512424,

    Pith/arXiv arXiv