Pith. sign in

REVIEW 4 major objections 5 minor 27 references

W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that replacing HiPPO initialization with WaLRUS wavelet-frame initialization in S4 improves long-range memory and classification accuracy, even when the state matrices are frozen.

desk verdict Promising WaLRUS initialization for S4 with consistent wins on delay and classification tasks, but the load-bearing truncation theorem is unproved and Neff undefined, so the efficient implementation isn't yet established. read the letter →

arxiv 2506.07920 v1 pith:YRYARHDF submitted 2025-06-09 cs.LG eess.ASeess.IVeess.SP

Hossein Babaei , Mel White , Richard G. Baraniuk This is my paper
classification cs.LGeess.ASeess.IVeess.SP
keywords statespacemodelsWaLRUSwaveletframesHiPPOinitializationS4long-rangesequencemodelingmultiresolutionanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

State space models (SSMs) remember input history through a linear recurrence, and their usefulness hinges on how the state matrix is initialized. This paper argues that initializing an S4 model with WaLRUS—a state matrix built from redundant wavelet frames—gives the model better long-range memory than the standard HiPPO initialization. In delay-reconstruction tests the wavelet core achieves about 5.3 times lower error, and in full S4 models with (A, B) frozen, W4S4 beats HiPPO-based S4 models whose (A, B) are trainable, reaching the highest reported accuracy on the text and speech benchmarks with fewer parameters. The paper's central claim is that a high-quality structured initialization can matter more than training the state dynamics from a worse starting point.

What carries the argument

The central object is the WaLRUS state matrix, a SaFARi-built SSM dynamics derived from redundant wavelet frames—overcomplete collections of time-frequency atoms that represent a signal at multiple resolutions. Its load-bearing property is that it admits a stable, exact diagonalization, and its eigenvalues plateau: only the first Neff exceed 1, and the rest equal 1. Theorem 1 uses this plateau to justify truncating the state to Neff dimensions before kernel computation, allowing W4S4 to avoid Diagonal-Plus-Low-Rank approximations while the complex-conjugate pairing of the real-valued spectrum keeps the memory footprint unchanged.

What would settle it

Compute the full eigenvalue spectrum of the diagonalized WaLRUS matrix for a redundant wavelet frame of size N and test whether every eigenvalue beyond the first Neff equals 1 to machine precision; if more than Neff eigenvalues deviate from 1 by more than roundoff, truncation changes the kernel and the efficiency justification collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that WaLRUS, constructed through the SaFARi framework from redundant wavelet frames, supplies a better prior for SSM state dynamics than HiPPO's polynomial projections. Because WaLRUS has a stable, exact diagonalization, the diagonal-plus-low-rank approximation that HiPPO-based S4 requires is unnecessary, and the state can be truncated to Neff dimensions using the spectral property stated in Theorem 1: only the first Neff eigenvalues exceed 1 and the remaining eigenvalues equal 1. When this initialization is embedded in S4 with (A, B) frozen, W4S4 outperforms HiPPO-initialized S4 variants on every benchmark tested, including 88.55% versus 86.82% on text classification with 215K parameters instead of 800K, and 94.37% versus 93.60% on autoregressive speech-command recognition.

Load-bearing premise

The load-bearing premise is Theorem 1's claim that only the first Neff eigenvalues of the diagonalized WaLRUS matrix exceed 1 and all remaining eigenvalues are exactly 1, so the state can be truncated to Neff dimensions without losing reconstruction fidelity; the theorem is stated in Section 3.2 and restated in Appendix 7.1, but no proof appears there.

Editorial extensions

If this is right

  • W4S4 with frozen (A, B) outperforms HiPPO-initialized S4 models with trainable (A, B) on text classification (88.55% vs 86.82%) and on autoregressive speech-command recognition (94.37% vs 93.60%), so initialization alone can carry much of the performance.
  • The WaLRUS core retains delayed input information about 5.3 times better than HiPPO-LegS in the minimal delay-reconstruction setting, showing the advantage exists before any deep network is added.
  • Because WaLRUS diagonalizes without Diagonal-Plus-Low-Rank approximation, W4S4 kernel computation is cheaper, which can translate into faster or lower-memory training for long sequences.
  • These results support the view that SSM initialization acts as a functional prior: the wavelet state supplies a useful set of temporal filters at initialization, and gradient descent has less work to do.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the spectral-plateau truncation argument should generalize to other SaFARi species built from redundant frames, since the plateau likely follows from frame redundancy rather than from wavelets specifically.
  • Our inference: letting A and B be trainable after WaLRUS initialization, which the paper leaves as future work, may combine the strong prior with learned adaptation and improve accuracy further.
  • Our inference: because the WaLRUS core wins even in a single-layer delay task, W4S4 is a natural fit for streaming or online settings where fast recurrent inference and long memory are both required.
  • Our inference: the results suggest multiresolution time-frequency localization is the property behind long-range memory; a direct test would compare W4S4 against another frame-based initialization with similar redundancy but different localization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes W4S4, an SSM initialization built from WaLRUS wavelet-frame matrices within the SaFARi framework, intended as a replacement for HiPPO initialization in deep SSM architectures such as S4. The central technical claim is that the diagonalized WaLRUS state matrix has only Neff significant eigenvalues, allowing the state space to be truncated to Neff dimensions without loss of reconstruction fidelity (Theorem 1). The paper reports controlled delay-reconstruction ablations and text/speech classification experiments, claiming that W4S4 outperforms HiPPO-initialized S4 models even when (A, B) are frozen, and it advertises parameter efficiency and better memory retention.

Significance. If fully supported, the contribution would be notable: a wavelet-frame-based initialization with stable diagonalization, exact kernel computation, and state truncation could improve both the accuracy and parameter efficiency of SSM-based sequence models. The paper's decision to freeze (A, B) in classification experiments is a clean and commendable way to isolate the effect of initialization, and the delay-reconstruction ablations provide a useful controlled setup. However, the current manuscript does not yet support these claims: the load-bearing Theorem 1 is asserted without proof, Neff is never defined, several datasets promised in Section 5.1 are never reported on, and the parameter-count comparisons are inconsistent across tables. The authors themselves state in the Future Work section that 'more empirical evidence should be gradually added to this manuscript,' which is a candid acknowledgment that the evidence base is incomplete.

major comments (4)
  1. [3.2/7.1] Theorem 1 is load-bearing but is never proved. It is stated in Section 3.2 with 'See Appendix for proof,' and restated in Appendix 7.1, but the appendix contains only the statement and the implementation steps, not a proof. Furthermore, Neff is never defined in terms of the wavelet frame construction, and the WaLRUS matrix itself is not constructed in the paper but only cited to the authors' preprints [12, 13]. Since the implemented model uses the theorem to truncate the state space, the evaluated model is not precisely specified without this proof. Please provide a complete proof, a precise definition of Neff, and an explicit statement of whether the tail eigenvalues are exactly 1 or only approximately 1; if the latter, give quantitative error bounds on the induced kernel approximation.
  2. [5.1/5.3] Section 5.1 promises evaluation on Long-Range Arena, Speech Commands, sequential CIFAR, BIDMC Vital Signs, and four multivariate forecasting datasets, but the experimental sections report results only for a small subset: delay-task log-MSE in Table 1 and text/speech classification in Tables 2 and 3. No results are reported for LRA tasks (ListOps, Retrieval, Images, Pathfinder, PathX) or for Electricity, Traffic, Weather, and Solar-Energy forecasting. The abstract and conclusion claim consistent improvements across long-range sequence modeling, which is not supported by the presented evidence. Please add the missing experiments or explicitly restrict the claims to the datasets actually evaluated.
  3. [Table 1] Table 1 reports single log-MSE values without error bars, number of seeds, or statistical significance information. The 'Relative' column is undefined, and for entries like '× 2.45' it is unclear whether the ratio is computed on MSE or log-MSE. Several margins are small (Solar ×1.15, BIDMC ×1.05), so without variance information the claim of consistent improvement is not robust. Please report mean ± standard deviation over multiple runs and define exactly how the relative improvement is computed.
  4. [Tables 2/3 and Section 6] The parameter-efficiency claim is not consistently supported by the data. In Table 2, W4S4 has 215K parameters versus 800K for the baselines, but in Table 3, W4S4 has 260K parameters versus 200K for the baselines, so the claim that W4S4 uses 'fewer parameters' is false on the speech benchmark. In addition, the comparisons across rows differ in architectural details that are not specified for W4S4, such as number of layers, hidden size, and number of SSM cores, and the baselines have learnable (A, B) while W4S4 does not. To support the conclusion in Section 6 that W4S4 achieves higher accuracy with fewer parameters and fixed (A, B), please provide a controlled comparison with matched architectural capacity and give the full configuration of W4S4.
minor comments (5)
  1. [2.3] The reference for 'Attention Is All You Need' appears as a placeholder '[? ]'; please replace it with the full citation.
  2. [4] The text says WaLRUS achieves '0.725 smaller log-MSE' and '×5.31', while also giving MSE values of 7.5e-4 and 4e-3; the relationship between these numbers and the figures should be clarified.
  3. [5.2/5.3] Section 5.2 states that B is fixed as an all-ones vector following [16], while Section 5.3 says that '(A, B) frozen'; please make the training protocol consistent and specify exactly which parameters are frozen in each experiment.
  4. [7.2] Appendix 7.2 contains the typo 'auroregressive' instead of 'autoregressive', and 'SC-ar' is used in Table 3 without being defined at first use.
  5. [Table 2] The heading 'Learnable (A, B)' uses checkmarks and crosses; please clarify that a checkmark means the (A, B) parameters are trained and a cross means they are frozen.

Circularity Check

1 steps flagged · score 4.0 of 10

Central truncation theorem is load-bearing and rests on the authors' unproved WaLRUS preprint, but the empirical W4S4-vs-HiPPO comparisons are independent.

  1. self citation load bearing [Section 3.2 (Theorem 1); Appendix 7.1]
    "Theorem 1. (See Appendix for proof.) Let A be the diagonalized state matrix of WaLRUS constructed from a redundant wavelet frame. Then: • Only the first Neff eigenvalues of A are greater than 1; the rest are exactly 1. • Only the first Neff elements of the state vector ex (in the diagonal basis) contribute to the reconstruction of the input signal. • The kernel K can be computed using only the top Neff rows of eK and the corresponding rows of the eigenvector matrix V ."

    The efficient form of W4S4 used in all experiments depends on truncating the WaLRUS state space to Neff dimensions, and that truncation is justified solely by Theorem 1. The theorem is asserted in Section 3.2 and then merely restated verbatim in Appendix 7.1, with no proof provided and with Neff never defined. Its content is exactly the 'stable diagonalization' and 'fast kernel computation without requiring low-rank approximations' property imported from the authors' own WaLRUS preprint [13]. Thus the load-bearing step reduces to an unverified self-citation rather than to a derivation contained in this paper. This is a substantive gap, though not a definitional equivalence: the reported accuracy numbers remain independent empirical content.

full rationale

The paper's main empirical claim—W4S4 with frozen (A,B) outperforms HiPPO-initialized S4 on delay-reconstruction, text, and speech benchmarks—is supported by experiments on external benchmarks and is not fitted to the target; no fitted parameter is renamed as a prediction, and the reported numbers are not forced by construction. However, the method's construction is not self-contained: WaLRUS is introduced by citation to two same-author preprints [12,13], and Theorem 1 in Section 3.2, which licenses the truncation enabling the efficient W4S4 implementation, is stated twice but never proved (Appendix 7.1 only repeats the statement) and relies on an undefined Neff. Because the parameter counts reported in Tables 2 and 3 (215K/260K) assume this truncation, the correctness and identity of the evaluated model rest on a load-bearing self-citation that is not independently verified in this manuscript. This raises the circularity score to 4: the central claim still has independent empirical content, but an important load-bearing step is imported from the authors' prior work without the promised proof.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central construction is inherited from the authors' own SaFARi and WaLRUS preprints, and the key truncation theorem is asserted without proof. The only new element is a reparameterization from prior work, so the ledger is dominated by self-referential assumptions.

free parameters (2)
  • Neff = not specified
    Truncation threshold for WaLRUS state; chosen by hand and not justified by a proof in this paper (Theorem 1 is stated without proof).
  • theta (1/dt) = swept in [1e-4, 1e-2] in delay ablation
    Time-scale hyperparameter in the time-invariant SSM; optimal value selected per model and per delay, which gives each model its best possible comparison but makes the comparison sensitive to tuning.
assumptions (3)
  • domain assumption SaFARi framework constructs valid SSMs from arbitrary algebraic frames
    The W4S4 construction depends on the SaFARi framework (ref [12], by the same authors) without re-deriving it.
  • ad hoc to paper WaLRUS matrices admit stable diagonalization with only Neff significant eigenvalues
    Theorem 1 in Section 3.2 and Appendix 7.1 is asserted with a promised proof that does not appear; the truncation procedure relies on it.
  • standard math Time-invariant reparameterization from How to train your HiPPO (ref [17]) preserves memory properties
    Used to define W4S4 from the scaled WaLRUS; standard in the SSM literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling." pith.science (2026). https://pith.science/paper/YRYARHDF

@misc{pith2026250607920,
  author       = {Pith},
  title        = {Pith review of: W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YRYARHDF}},
  note         = {Machine review of arXiv:2506.07920}
}
read the original abstract

State Space Models (SSMs) have emerged as powerful components for sequence modeling, enabling efficient handling of long-range dependencies via linear recurrence and convolutional computation. However, their effectiveness depends heavily on the choice and initialization of the state matrix. In this work, we build on the SaFARi framework and existing WaLRUS SSMs to introduce a new variant, W4S4 (WaLRUS for S4), a new class of SSMs constructed from redundant wavelet frames. WaLRUS admits a stable diagonalization and supports fast kernel computation without requiring low-rank approximations, making it both theoretically grounded and computationally efficient. We show that WaLRUS retains information over long horizons significantly better than HiPPO-based SSMs, both in isolation and when integrated into deep architectures such as S4. Our experiments demonstrate consistent improvements across delay reconstruction tasks, classification benchmarks, and long-range sequence modeling, confirming that high-quality, structured initialization enabled by wavelet-based state dynamic offers substantial advantages over existing alternatives. WaLRUS provides a scalable and versatile foundation for the next generation of deep SSM-based models.

Figures

Figures reproduced from arXiv: 2506.07920 by the authors.

Figure 1
Figure 1. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 1
Figure 1. Log-MSE comparison between Scaled-WaLRUS and HiPPO-LegS for various values of [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Log-MSE across different delay lengths with optimal [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 17 canonical work pages

  1. [1]

    Transformers in time series: A survey

    Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125, 2022

    arXiv 2022

  2. [2]

    Cheung, Ahmed Imtiaz Hu- mayun, Daniel LeJeune, Naiming Liu, Lorenzo Luzi, Jasper Tan, Zichao Wang, and Richard G

    Sina Alemohammad, Hossein Babaei, Randall Balestriero, Matt Y . Cheung, Ahmed Imtiaz Hu- mayun, Daniel LeJeune, Naiming Liu, Lorenzo Luzi, Jasper Tan, Zichao Wang, and Richard G. Baraniuk. Wearing a mask: Compressed representations of variable-length sequences using recurrent neural tangent kernels. In ICASSP 2021 - 2021 IEEE International Conference on A...

    work page 2021

  3. [3]

    Machine learning advances for time series forecasting

    Ricardo P Masini, Marcelo C Medeiros, and Eduardo F Mendes. Machine learning advances for time series forecasting. Journal of economic surveys, 37(1):76–111, 2023

    work page 2023

  4. [4]

    Scinet: Time series modeling and forecasting with sample convolution and interaction.Advances in Neural Information Processing Systems , 35:5816–5828, 2022

    Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction.Advances in Neural Information Processing Systems , 35:5816–5828, 2022

    work page 2022

  5. [5]

    Time series forecasting using a hybrid arima and neural network model

    G Peter Zhang. Time series forecasting using a hybrid arima and neural network model. Neurocomputing, 50:159–175, 2003

    work page 2003

  6. [6]

    Tsmixer: An all-mlp architecture for time series forecasting

    Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting. arXiv preprint arXiv:2303.06053, 2023

    arXiv 2023

  7. [7]

    Rethinking full connectivity in recurrent neural networks

    Matthijs Van Keirsbilck, Alexander Keller, and Xiaodong Yang. Rethinking full connectivity in recurrent neural networks. arXiv preprint arXiv:1905.12340, 2019

    arXiv 1905

  8. [8]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310–1318. Pmlr, 2013

    2013

Show all 27 references
  1. [9]

    Understanding the effective receptive field in deep convolutional neural networks

    Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016

    2016

  2. [10]

    HiPPO: Recurrent memory with optimal polynomial projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. HiPPO: Recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems , 2020

    2020

  3. [11]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations , 2022

    2022

  4. [12]

    Safari: State-space models for frame-agnostic representation

    Hossein Babaei, Mel White, Sina Alemohammad, and Richard G Baraniuk. Safari: State-space models for frame-agnostic representation. arXiv preprint arXiv:2505.08977, 2025

    2025 arXiv

  5. [13]

    Baraniuk

    Hossein Babaei, Mel White, Sina Alemohammad, and Richard G. Baraniuk. Walrus: Wavelets for long range representation using state space methods. arXiv preprint arXiv:2505.12161 , 2025

    2025 arXiv

  6. [14]

    Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges

    Badri Narayana Patro and Vijay Srinivas Agneeswaran. Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges. arXiv preprint arXiv:2404.16112, 2024

    2024 arXiv

  7. [15]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems , 34:572–585, 2021. 11

    2021

  8. [16]

    On the parameterization and initialization of diagonal state space models

    Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971–35983, 2022

    2022

  9. [17]

    How to train your hippo: State space models with generalized orthogonal basis projections

    Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. How to train your hippo: State space models with generalized orthogonal basis projections. arXiv preprint arXiv:2206.12037, 2022

    2022 arXiv

  10. [18]

    Long range arena: A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020

    2011 arXiv

  11. [19]

    Speech commands: A dataset for limited-vocabulary speech recognition

    Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018

    2018 arXiv

  12. [20]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, ON, Canada, 2009

    2009

  13. [21]

    Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals

    Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic si...

    2000

  14. [22]

    Toward a robust estimation of respiratory rate from pulse oximeters

    Marco AF Pimentel, Alistair EW Johnson, Peter H Charlton, Drew Birrenkott, Peter J Watkinson, Lionel Tarassenko, and David A Clifton. Toward a robust estimation of respiratory rate from pulse oximeters. IEEE Transactions on Biomedical Engineering , 64(8):1914–1923, 2016

    1914

  15. [23]

    Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting

    Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419–22430, 2021

    2021

  16. [24]

    Modeling long-and short-term temporal patterns with deep neural networks

    Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95–104, 2018. 7 Appendix 7.1 Theoretic...

    2018

  17. [25]

    Diagonalize A and compute only the top Neff eigenvalues and rows of eB

  18. [26]

    Compute the kernel eK using this reduced system

  19. [27]

    Absorb V [0:Neff , :] into the output matrix C and proceed with standard SSM computations. 7.2 Datasets To demonstrate the improvement that W4S4 provides, we use a popular and diverse suite of datasets spanning vision, biomedical signals, speech, and multivariate time series f...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.