Pith. sign in

REVIEW 4 major objections 4 minor 50 references

On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single selective state-space layer realizes polynomial degree L+3, forcing linear attention to need O(log L) layers, and four Mamba layers can represent any bounded-degree multivariate polynomial.

desk verdict A clean polynomial-degree separation for simplified S6 vs linear attention, but the four-layer universality proof has an internal algebraic contradiction that needs fixing before the broad expressivity claim is credible. read the letter →

arxiv 2502.02209 v1 pith:HNKMKBT4 submitted 2025-02-04 cs.LG

Edo Cohen-Karlik , Itamar Zimerman , Liane Galanti , Ido Atad , Amir Globerson , Lior Wolf This is my paper · ORCID
classification cs.LG MSC 68T0768Q3268Q17
keywords selectivestate-spacemodelsS6layerMambalinearattentionmultivariatepolynomialsexpressivitygeneralizationboundsequencemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that the selective state-space layer at the core of Mamba can represent functions that a linear transformer would need logarithmically many layers to match. By viewing each architecture as a multivariate polynomial in the input, it proves that a single S6 channel realizes polynomial degree $L+3$ for a sequence of length $L$, while one causal linear-attention head reaches degree at most 3. A model with four stacked Mamba layers suffices to represent every multivariate polynomial of bounded degree, where a linear attention model needs $O(\log L)$ layers. The paper also derives a generalization bound for S6 that does not grow with sequence length.

What carries the argument

The load-bearing identity is the unrolled selective recurrence seen as a multivariate polynomial. For the simplified variant, $y_L = \sum_{j=1}^L c_j (\prod_{k=j+1}^L x_k) x_j^2$ shows that each time step contributes a term proportional to the product of all later inputs, so the polynomial degree grows as $L+3$. This is contrasted with the attention identity $y_L = \sum_j c_j x_j^2 x_L$, where each head is limited to degree 3 and $N$ layers multiply the degree to at most $3^N$. The paper then uses a three-layer Mamba construction (position selection, duplication, and telescoping multiplication) as the engine for representing arbitrary monomials, and feeds those monomials through a fourth layer to assemble any bounded-degree multivariate polynomial.

What would settle it

Train a single S6 layer and a single causal linear-attention head with equal width on a fixed multivariate polynomial of degree $L+3$ over sequences of length $L \in \{5,10,20,40\}$; if the attention head reaches the same accuracy as S6 at any length, or if the S6 layer cannot represent the polynomial for large $L$, the claimed logarithmic depth gap fails.

Watch

Extended reading notes

Core claim

The central discovery is that the input-dependent transition matrix of a selective state-space layer accumulates polynomial degree linearly with sequence length. In the simplified polynomial variant, unrolling the recurrence $h_t = \bar{A}_t h_{t-1} + \bar{B}_t x_t$ with $\bar{A}_t = S_\Delta x_t$ gives $y_L = \sum_{j=1}^L c_j (\prod_{k=j+1}^L x_k) x_j^2$, a polynomial whose maximal degree is $L+3$. A single causal linear-attention head computes only degree-3 polynomials, because the query-key-value contraction is trilinear; after $N$ stacked layers the maximum degree is at most $3^N$, so matching degree $L+3$ requires $N \in O(\log L)$. Theorem 2 then broadens the gap from a single function to the whole class of bounded-degree multivariate polynomials: four Mamba layers with learnable positional encodings, zero padding, and a linear encoder can construct any such polynomial, using the layers to isolate a position, duplicate an entry, and multiply the copies into powers through a telescoping subtraction.

Load-bearing premise

The theory is proved for a simplified polynomial S6 variant in which softplus and exponentiation are replaced by polynomials, $A$ is set to zero in the unrolled identity, and the architecture is augmented with learnable positional encodings, zero padding, and a linear encoder; the paper assumes, with only empirical support, that these changes preserve the essential expressivity of the original Mamba layer.

Editorial extensions

If this is right

  • One S6 layer can represent counting-style and recurrent functions in a single pass; a linear transformer needs a stack of depth $O(\log L)$ to express the same function.
  • Four stacked Mamba layers are formally sufficient to cover the entire class of bounded-degree multivariate polynomials, a function space the paper argues is substantial rather than anecdotal.
  • The expressivity gain does not come at the price of generalization: the bound in Theorem 3 scales with $1/\sqrt{m}$ and depends on parameter norms and the contraction constant $K$, but not on the sequence length $L$.
  • For linear-attention language models, the result means selective state-space layers offer comparable or greater representational power at constant depth, with the computational benefit of subquadratic sequence processing.
  • If the standard softmax is approximated by a degree-$P$ polynomial, each softmax attention layer reaches degree at most $3P+1$, still independent of $L$, so the logarithmic gap persists in the softmax regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the polynomial-degree view predicts that tasks dominated by counting, copying, or multiplicative long-range dependencies will favor S6 layers over linear attention at equal parameter budgets, while tasks dominated by pairwise induction-like pattern matching will favor attention.
  • The telescoping construction used to build monomials suggests a concrete complexity measure for selective state-space layers: the number of distinct tokens a single channel couples grows linearly with $L$, which could be tested empirically by probing trained Mamba models for high-order interactions.
  • If the simplified polynomial variant is as close to the original S6 as the reported perplexity and accuracy numbers suggest, the paper's four-layer universality statement transfers to practical Mamba models, which would mean hybrid architectures mixing recurrent and attention blocks get a principled assignment of high-order and pairwise computation.
  • A length-agnostic bound of this form implies that selective state-space models trained on short contexts can be evaluated on much longer sequences without a principled generalization penalty, provided the empirical contraction constant stays below 1.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies the expressive power of selective state-space layers (S6/Mamba) through the lens of multivariate polynomial degree. It introduces a simplified polynomial variant of S6, argues that one S6 channel can realize polynomials whose degree grows linearly with sequence length, contrasts this with the degree-3 bound for a single causal linear attention head, and concludes that S6 layers require logarithmically fewer layers than linear attention to represent the same polynomials. It further claims that four Mamba layers can represent any bounded-degree multivariate polynomial, and it derives a norm-based generalization bound for the simplified S6 layer. The theoretical claims are supported by synthetic polynomial-learning experiments and by ImageNet-100 and WikiText-103 comparisons of the simplified variant against the original S6 layer.

Significance. If the main claims are established, this is a useful contribution to the theory of selective state-space models. The core unrolled-recurrence computation for the simplified polynomial S6 is transparent and checkable, and the synthetic experiments provide a concrete, falsifiable test of the claimed expressivity gap. The empirical demonstration that the simplified polynomial variant performs close to the original S6 on NLP and vision benchmarks is a genuine strength and goes beyond the usual purely theoretical treatment. However, the universality theorem that carries the paper's central separation is currently not proven as written: the construction in the appendix uses a parameter configuration that cannot produce the multiplicative behavior it invokes. The paper's own limitations section (Appendix D) also concedes that the simplified architecture is not formally connected to the original S6 layer. For these reasons the paper is not yet ready for acceptance, though the underlying idea seems repairable.

major comments (4)
  1. [Appendix B, proof of Theorem 2; Lemma 3, Step 3; Eq. (30)] The multiplicative step in the 4-layer construction is internally inconsistent. The proof states that the final SSM is applied 'with system matrices equal to 1 (Ai = Bi = Ci = 1)' and that subtracting two such SSM outputs yields the product Π_{j=1}^L s_j. With constant A_k = 1, the unrolled recurrence y_t = Σ_{j≤t} (Π_{k=j+1}^t A_k) B_j x_j reduces to an additive cumulative sum, so SSM(z) − SSM(z') can cancel only early additive terms and cannot produce the monomial x_j^P for P > 2. The displayed telescoping identity in Eq. (31) requires an input-dependent transition such as A_k = x_k, as in Eq. (8), not A_k = 1. Since Theorem 2 is the basis for the paper's main expressivity gap, this proof must be repaired: either set the transition to the input-dependent sequence z_k and verify the cancellation explicitly, or provide a different construction.
  2. [Appendix B, Lemma 2, Eqs. (22)-(24)] The proof that a single S6 channel can express any single causal linear attention head contains an indexing error. With A = 1 and S∆ = 0, the recurrence gives H_i = Σ_{j≤i} WK X_j X_j, so the output should be (WQ X_i)^T H_i = Σ_{j≤i} X_i^T WQ^T WK X_j X_j. The displayed Eq. (24) instead writes WK X_i X_j inside the sum, which is a different function and does not correspond to the stated attention pattern. As written, the reduction to causal linear attention is therefore not established; the authors should correct the formula and specify exactly how the value matrix WV is absorbed into the Mamba block.
  3. [Section 4.2, Theorem 2; Appendix D] The theorem's conclusion concerns a model with learnable PE, ZeroPad, and a linear encoder, with the SiLU activations removed, while the paper's headline claim concerns the original S6/Mamba layer. The transfer from the simplified model to the original architecture is supported only by the empirical results in Table 1 and Fig. 3; Appendix D explicitly acknowledges that the simplified architecture is not formally justified. Given that Theorem 2 is central to the expressivity separation, the authors should state precisely which claims are proven about the original S6 layer and which are proven only for the polynomial augmentation, or provide a formal argument that the omitted components cannot reduce the stated expressivity.
  4. [Section 4.2, Eq. (8); Appendix B.1, proof of Lemma 1] The maximal degree computation is inconsistent: the main text claims a maximal degree of L+3, while the appendix proof, using A = 0 and the displayed recurrence, concludes degree L+2. Moreover, in Eq. (7) the product Π_{k=j+1}^t x_k already includes x_t, making the separate x_t factor in the same equation ambiguous. The O(log L) depth separation survives either degree count, but the precise statement of Lemma 1 and the displayed formulas should be corrected and made internally consistent.
minor comments (4)
  1. [Section 4.2, Lemma 3] The lemma states that P ∈ R, but the construction only makes sense for nonnegative integers P, since it duplicates x_j exactly P − 2 times and forms the power x_j^P. Please restrict P to integers, or clarify the intended generalization.
  2. [Section 5.1] The sentence 'the polynomial variant achieved a perplexity score of 26.42, 0.69 points lower than its original baseline score of 25.73' is directionally wrong: 26.42 is higher than 25.73, so the polynomial variant is worse, not better.
  3. [Appendix B, Theorem 5] The final displayed inequality appears to be missing a '+' sign between the Rademacher term and the logarithmic term; please fix the typo.
  4. [Section 3 and Section 4.2] In Eq. (16) and surrounding text, the notation switches between Y_i, Y_k, and y_t, and the roles of C_i and the output equation are inconsistent; please unify the notation for the S6 layer.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the expressivity separation is derived from the unrolled S6 recurrence and attention polynomial degree, not from fitted inputs or load-bearing self-citations.

full rationale

The paper's expressivity claims are derived from the model equations rather than fitted to data or imported from self-citations. Lemma 1 unrolls the simplified S6 recurrence (Eq. 7) to obtain a polynomial of degree L+3, and independently characterizes a single causal linear-attention head as having degree at most 3; the O(log L) depth separation follows from polynomial composition. Lemma 2 constructs an S6 channel that reproduces the causal linear-attention map, and Theorem 2/Lemma 3 give an explicit channel-construction for monomials. None of these steps fits a parameter to the target quantity and then reports the fit as a prediction. The paper's self-citations (Ali et al. 2024; Zimerman & Wolf 2023; Zimerman et al. 2023) appear only in related work or discussion and are not load-bearing. The main caveat is a proof inconsistency, not circularity: Theorem 2's proof states 'applying the same SSM with system matrices equal to 1 (Ai = Bi = Ci = 1)' and then claims U^4_i = Product_j s_j, whereas with A=B=C=1 the unrolled recurrence is a cumulative sum rather than a product; the multiplicative step in Lemma 3 instead requires the input-dependent A term of Eq. 7. This is a correctness gap in the universality proof, and Appendix D honestly acknowledges the broader simplification gap, but it does not amount to the target result being assumed as input. The derivations are self-contained, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim does not depend on fitted numerical parameters. The paper's own simplified S6 variant is a modeling choice rather than an invented physical entity. The main axioms are the architectural simplifications and the comparison-class restrictions, plus the contraction lemma application that is not fully justified.

assumptions (4)
  • ad hoc to paper The simplified S6 variant of Eq. 5, with p_A assumed linear and A=0 in Lemma 1, faithfully represents the expressivity of the actual selective S6 layer.
    Introduced in Sec. 4.1 and used throughout the proof of Lemma 1 and Theorem 1. The gap to the original S6 is only empirically justified in Table 1, not proven.
  • domain assumption Causal, single-head, softmax-free linear self-attention with polynomial activations and no FFN is the relevant comparison class.
    Stated in Sec. 4.2 and defended in Sec. 6 by citing linear-attention LLMs. The expressivity gap is not proven for standard softmax Transformers.
  • domain assumption The theoretical analysis may omit SiLU, Conv1D, and gating, and Theorem 2 may add learnable PE, ZeroPad, and a linear encoder.
    The proofs in Sec. 4.2 and Appendix B use a simplified Mamba block. These auxiliary components are not part of the original Mamba architecture as described in Eq. 1.
  • ad hoc to paper The contraction lemma (Lemma 5) applies to the composed functions in the Rademacher complexity proof, including exp functions.
    Appendix B.2, Eq. 38-40 invokes Lemma 5 with σij(z)=exp(z)·constants. exp is not globally Lipschitz or positive-homogeneous on the relevant domain; the domain truncation is not stated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach." pith.science (2026). https://pith.science/paper/HNKMKBT4

@misc{pith2026250202209,
  author       = {Pith},
  title        = {Pith review of: On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HNKMKBT4}},
  note         = {Machine review of arXiv:2502.02209}
}
read the original abstract

Recent advances in efficient sequence modeling have introduced selective state-space layers, a key component of the Mamba architecture, which have demonstrated remarkable success in a wide range of NLP and vision tasks. While Mamba's empirical performance has matched or surpassed SoTA transformers on such diverse benchmarks, the theoretical foundations underlying its powerful representational capabilities remain less explored. In this work, we investigate the expressivity of selective state-space layers using multivariate polynomials, and prove that they surpass linear transformers in expressiveness. Consequently, our findings reveal that Mamba offers superior representational power over linear attention-based models for long sequences, while not sacrificing their generalization. Our theoretical insights are validated by a comprehensive set of empirical experiments on various datasets.

Figures

Figures reproduced from arXiv: 2502.02209 by the authors.

Figure 1
Figure 1. Expressivity via Polynomial Degree: Our characteriza￾tion of SSMs, S6 layers, and causal self-attention via multivariate polynomials allows us to identify the expressiveness gap between these layers through maximal polynomial degree. pabilities of Mamba models compared to other architectures, our work introduces a different trend. We demonstrate the superior expressive power of S6 layers, using a theoretical framewo… view at source ↗
Figure 2
Figure 2. Visualization of 3-stacked Mamba layers expressing monomials of a univariate polynomial, as formulated in Lemma 3. To simplify the visualization, the Conv1D layer has been omitted. proof, which hinges on the following two key capability of the Mamba architecture: (i) Per-position selection: By utilizing Mamba’s auxil￾iary components, including the gating branch, linear layers, and learnable PE, each Mamba layer can … view at source ↗
Figure 3
Figure 3. Model justifications & ablations: In the left panel, we present the top-1 accuracy score for image classification via the ImageNet-100 benchmark, while the right panel displays the per￾plexity score for language modeling using the WikiText-103. The y-axis represents the model’s score across different epochs. In both figures, the blue curve represents the baseline, the yellow curve corresponds to Eq.5, the green curv… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 20 canonical work pages

  1. [1]

    Ahamed, M. A. and Cheng, Q. Mambatab: A simple yet effective approach for handling tabular data. arXiv preprint arXiv:2401.08867, 2024

    arXiv 2024

  2. [2]

    The hidden attention of mamba models

    Ali, A., Zimerman, I., and Wolf, L. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590, 2024

    arXiv 2024

  3. [3]

    and Li, Y

    Allen-Zhu, Z. and Li, Y. Can sgd learn recurrent neural networks with provable generalization? Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  4. [4]

    A 2-dimensional state space layer for spatial inductive bias

    Baron, E., Zimerman, I., and Wolf, L. A 2-dimensional state space layer for spatial inductive bias. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=BGkqypmGvm

    work page 2024

  5. [5]

    M., Orvieto, A., Walker, B., Salvi, C., and Lyons, T

    Cirone, N. M., Orvieto, A., Walker, B., Salvi, C., and Lyons, T. Theoretical foundations of deep selective state-space models. arXiv preprint arXiv:2402.19047, 2024

    arXiv 2024

  6. [6]

    B., Cohen, N., and Globerson, A

    Cohen-Karlik, E., David, A. B., Cohen, N., and Globerson, A. On the implicit bias of gradient descent for temporal extrapolation. In International Conference on Artificial Intelligence and Statistics, pp.\ 10966--10981. PMLR, 2022 a

    work page 2022

  7. [7]

    Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets

    Cohen-Karlik, E., Menuhin-Gruman, I., Giryes, R., Cohen, N., and Globerson, A. Learning low dimensional state spaces with overparameterized recurrent neural nets. arXiv preprint arXiv:2210.14064, 2022 b

    work page Pith review arXiv 2022

  8. [8]

    L., Fernando, A., Botev, A., Cristian-Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., et al

    De, S., Smith, S. L., Fernando, A., Botev, A., Cristian-Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427, 2024

    arXiv 2024

Show all 50 references
  1. [9]

    Emami, M., Sahraee-Ardakan, M., Pandit, P., Rangan, S., and Fletcher, A. K. Implicit bias of linear rnns. In International Conference on Machine Learning, pp.\ 2982--2992. PMLR, 2021

    2021

  2. [10]

    Norm-based generalization bounds for sparse neural networks

    Galanti, T., Xu, M., Galanti, L., and Poggio, T. Norm-based generalization bounds for sparse neural networks. Advances in Neural Information Processing Systems, 36, 2024

    2024

  3. [11]

    Size-independent sample complexity of neural networks

    Golowich, N., Rakhlin, A., and Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pp.\ 297--299. PMLR, 2018

    2018

  4. [12]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  5. [13]

    Efficiently modeling long sequences with structured state spaces

    Gu, A., Goel, K., and R \'e , C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021 a

    2021 arXiv

  6. [14]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and R \'e , C. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34: 0 572--585, 2021 b

    2021

  7. [15]

    Diagonal state spaces are as effective as structured state spaces

    Gupta, A., Gu, A., and Berant, J. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35: 0 22982--22994, 2022 a

    2022

  8. [16]

    Simplifying and understanding state space models with diagonal linear rnns

    Gupta, A., Mehta, H., and Berant, J. Simplifying and understanding state space models with diagonal linear rnns. arXiv preprint arXiv:2212.00768, 2022 b

    2022 arXiv

  9. [17]

    Gradient descent learns linear dynamical systems

    Hardt, M., Ma, T., and Recht, B. Gradient descent learns linear dynamical systems. Journal of Machine Learning Research, 19 0 (29): 0 1--44, 2018

    2018

  10. [18]

    Transformer quality in linear time

    Hua, W., Dai, Z., Liu, H., and Le, Q. Transformer quality in linear time. In International Conference on Machine Learning, pp.\ 9099--9117. PMLR, 2022

    2022

  11. [19]

    M., and Malach, E

    Jelassi, S., Brandfonbrener, D., Kakade, S. M., and Malach, E. Repeat after me: Transformers are better than state space models at copying. arXiv preprint arXiv:2402.01032, 2024

    2024 arXiv

  12. [20]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  13. [21]

    and Talagrand, M

    Ledoux, M. and Talagrand, M. Probability in Banach spaces. Springer Berlin, Heidelberg, 1991

    1991

  14. [22]

    Minimax-01: Scaling foundation models with lightning attention

    Li, A., Gong, B., Yang, B., Shan, B., Liu, C., Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313, 2025 a

    2025 arXiv

  15. [23]

    Videomamba: State space model for efficient video understanding

    Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., and Qiao, Y. Videomamba: State space model for efficient video understanding. In European Conference on Computer Vision, pp.\ 237--255. Springer, 2025 b

    2025

  16. [24]

    Mamba-nd: Selective state space modeling for multi-dimensional data

    Li, S., Singh, H., and Grover, A. Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprint arXiv:2402.05892, 2024

    2024 arXiv

  17. [25]

    Pointmamba: A simple state space model for point cloud analysis

    Liang, D., Zhou, X., Wang, X., Zhu, X., Xu, W., Zou, Z., Ye, X., and Bai, X. Pointmamba: A simple state space model for point cloud analysis. arXiv preprint arXiv:2402.10739, 2024

    2024 arXiv

  18. [26]

    Jamba: A hybrid transformer-mamba language model

    Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev-Shwartz, S., et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  19. [27]

    and Li, Q

    Liu, F. and Li, Q. From generalization analysis to optimization designs for state space models. arXiv preprint arXiv:2405.02670, 2024

    2024 arXiv

  20. [28]

    Swin-umamba: Mamba-based unet with imagenet-based pretraining

    Liu, J., Yang, H., Zhou, H.-Y., Xi, Y., Yu, L., Yu, Y., Liang, Y., Shi, G., Zhang, S., Zheng, H., et al. Swin-umamba: Mamba-based unet with imagenet-based pretraining. arXiv preprint arXiv:2402.03302, 2024 a

    2024 arXiv

  21. [29]

    Vmamba: Visual state space model

    Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024 b

    2024 arXiv

  22. [30]

    Soft: Softmax-free transformer with linear complexity

    Lu, J., Yao, J., Zhang, J., Zhu, X., Xu, H., Gao, W., Xu, C., Xiang, T., and Zhang, L. Soft: Softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34: 0 21297--21309, 2021

    2021

  23. [31]

    Y., and Nie, L

    Lv, Q., Deng, X., Chen, G., Wang, M. Y., and Nie, L. Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl. arXiv preprint arXiv:2406.05427, 2024

    2024 arXiv

  24. [32]

    Mega: moving average equipped gated attention

    Ma, X., Zhou, C., Kong, X., He, J., Gui, L., Neubig, G., May, J., and Zettlemoyer, L. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209.10655, 2022

    2022 arXiv

  25. [33]

    The illusion of state in state-space models

    Merrill, W., Petty, J., and Sabharwal, A. The illusion of state in state-space models. arXiv preprint arXiv:2404.08819, 2024

    2024 arXiv

  26. [34]

    Transnormerllm: A faster and better large language model with improved transnormer

    Qin, Z., Li, D., Sun, W., Sun, W., Shen, X., Han, X., Wei, Y., Lv, B., Luo, X., Qiao, Y., et al. Transnormerllm: A faster and better large language model with improved transnormer. 2023

    2023

  27. [35]

    Theory, analysis, and best practices for sigmoid self-attention

    Ramapuram, J., Danieli, F., Dhekane, E., Weers, F., Busbridge, D., Ablin, P., Likhomanenko, T., Digani, J., Gu, Z., Shidani, A., et al. Theory, analysis, and best practices for sigmoid self-attention. arXiv preprint arXiv:2409.04431, 2024

    2024 arXiv

  28. [36]

    The expressive capacity of state space models: A formal language perspective

    Sarrof, Y., Veitsman, Y., and Hahn, M. The expressive capacity of state space models: A formal language perspective. arXiv preprint arXiv:2405.17394, 2024

    2024

  29. [37]

    Scaling laws for linear complexity language models

    Shen, X., Li, D., Leng, R., Qin, Z., Sun, W., and Zhong, Y. Scaling laws for linear complexity language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 16377--16426, 2024

    2024

  30. [38]

    T., Warrington, A., and Linderman, S

    Smith, J. T., Warrington, A., and Linderman, S. W. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933, 2022

    2022 arXiv

  31. [39]

    Stéphane Boucheron, Olivier Bousquet, G. L. Theory of classification: a survey of some recent advances. ESAIM: Probability and Statistics, 9: 0 323--375, 3 2010. URL http://eudml.org/doc/104340

    2010

  32. [40]

    Retentive network: A successor to transformer for large language models

    Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  33. [41]

    Training data-efficient image transformers & distillation through attention

    Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J \'e gou, H. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp.\ 10347--10357. PMLR, 2021

    2021

  34. [42]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  35. [43]

    An empirical study of mamba-based language models

    Waleffe, R., Byeon, W., Riach, D., Norick, B., Korthikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024

    2024 arXiv

  36. [44]

    Graph-mamba: Towards long-range graph sequence modeling with selective state spaces

    Wang, C., Tsepa, O., Ma, J., and Wang, B. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint arXiv:2402.00789, 2024

    2024 arXiv

  37. [45]

    Replacing softmax with relu in vision transformers

    Wortsman, M., Lee, J., Gilmer, J., and Kornblith, S. Replacing softmax with relu in vision transformers. arXiv preprint arXiv:2309.08586, 2023

    2023 arXiv

  38. [46]

    Secure transformer inference made non-interactive

    Zhang, J., Liu, J., Yang, X., Wang, Y., Chen, K., Hou, X., Ren, K., and Yang, X. Secure transformer inference made non-interactive. Cryptology ePrint Archive, 2024

    2024

  39. [47]

    and Wolf, L

    Zimerman, I. and Wolf, L. On the long range abilities of transformers. arXiv preprint arXiv:2311.16620, 2023

    2023 arXiv

  40. [48]

    Converting transformers to polynomial form for secure inference over homomorphic encryption

    Zimerman, I., Baruch, M., Drucker, N., Ezov, G., Soceanu, O., and Wolf, L. Converting transformers to polynomial form for secure inference over homomorphic encryption. arXiv preprint arXiv:2311.08610, 2023

    2023 arXiv

  41. [49]

    E., Chahed, I., Belkada, Y., Kunsch, G., and Hacid, H

    Zuo, J., Velikanov, M., Rhaiem, D. E., Chahed, I., Belkada, Y., Kunsch, G., and Hacid, H. Falcon mamba: The first competitive attention-free 7b language model. arXiv preprint arXiv:2410.05355, 2024

    2024 arXiv

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.