pith. sign in

arxiv: 2606.01135 · v1 · pith:OQYDOJKEnew · submitted 2026-05-31 · 💻 cs.NE · cs.SD

Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition

Tauseef Ahmed , Tao Sun , Jeronimo Castrillon , Kanishkan Vadivel , Guangzhi Tang This is my paper

Pith reviewed 2026-06-28 16:19 UTC · model grok-4.3

classification 💻 cs.NE cs.SD
keywords neuromorphic computingautomatic speech recognitionspiking neural networksevent-driven neural networksMamba modelsactivation sparsityLibriSpeech
0
0 comments X

The pith

SpeechMamba adapted to neuromorphic form reaches over 60 percent activation sparsity with little accuracy loss on speech tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to adapt the SpeechMamba model for neuromorphic hardware by creating event-driven and spiking versions that exploit activation sparsity. A sympathetic reader would care because standard ASR models require too much power for always-on use on phones or smart devices. The event-driven version uses a FATReLU activation to hit more than 60 percent sparsity while losing less than one percent accuracy on the LibriSpeech benchmark. The spiking version reaches over 70 percent sparsity and needs 30 percent fewer parameters than other spiking networks. The authors also supply a cycle-accurate simulator that finds further efficiency improvements of over 10 percent by guiding algorithm and hardware choices together.

Core claim

The paper claims that an event-driven SpeechMamba model with FATReLU activation can achieve over 60% activation sparsity with less than 1% accuracy degradation on LibriSpeech, and that a spiking SpeechMamba can attain over 70% sparsity while using 30% fewer parameters than comparable spiking neural networks, supported by results from a cycle-accurate event-driven simulator.

What carries the argument

The FATReLU activation function in the event-driven SpeechMamba, which promotes sparsity by producing zero outputs for many inputs, combined with the conversion to spiking neural network dynamics in the spiking variant.

If this is right

  • ASR models can operate with far lower energy on edge devices due to the high sparsity.
  • The simulator enables identification of computational bottlenecks for additional efficiency gains beyond 10%.
  • Spiking versions reduce model size compared to other SNN approaches for the same task.
  • High sparsity levels support real-time processing with reduced latency on resource-constrained hardware.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • These techniques could be tested on other sequence modeling architectures for similar efficiency gains.
  • Physical hardware measurements would be needed to confirm the energy benefits beyond simulation.
  • The approach opens the possibility of combining with other neuromorphic sensors for integrated low-power systems.

Load-bearing premise

Sparsity measured in the cycle-accurate simulator will produce corresponding reductions in energy consumption on real neuromorphic hardware.

What would settle it

Deploying the models on physical neuromorphic chips and measuring that energy use remains close to that of the original dense model despite the reported sparsity.

Figures

Figures reproduced from arXiv: 2606.01135 by Guangzhi Tang, Jeronimo Castrillon, Kanishkan Vadivel, Tao Sun, Tauseef Ahmed.

Figure 1
Figure 1. Figure 1: Encoder and decoder blocks of the E-SpeechMamba architecture. Red dots indicate inserted FATReLU activation points. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Block diagram of Mamba block in E-SpeechMamba [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Activation sparsity within each encoder and decoder [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Simulated CPU cycles distributions across different [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
read the original abstract

Deep learning has greatly advanced automatic speech recognition (ASR), enabling widespread deployment on edge devices such as smartphones and smart home systems. However, the computational and energy demands of deep neural networks pose significant challenges for such resource-constrained deployments, introducing latency and limiting real-time interaction. Neuromorphic computing offers a promising solution by introducing activation sparsity through spiking neural networks (SNNs) and event-driven neural networks, converting dense operations into sparse computations. However, a study that evaluates the hardware benefits of different neuromorphic strategies remains lacking for ASR. This paper explores spiking and event-driven neuromorphic neural networks to improve activation sparsity in the state-of-the-art SpeechMamba model for ASR. We introduce an event-driven SpeechMamba with FATReLU activation, achieving over 60% activation sparsity with less than 1% accuracy degradation on LibriSpeech. We also propose a spiking SpeechMamba that attains over 70% sparsity while using 30% fewer parameters than comparable SNNs. Finally, we develop a cycle-accurate event-driven simulator enabling flexible algorithm-hardware co-exploration, which helps us identify computational bottlenecks and yields over 10% additional efficiency improvements.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces event-driven and spiking neuromorphic variants of the SpeechMamba architecture for automatic speech recognition. It claims an event-driven SpeechMamba with FATReLU activation achieves >60% activation sparsity with <1% accuracy degradation on LibriSpeech, a spiking version reaches >70% sparsity with 30% fewer parameters than comparable SNNs, and a custom cycle-accurate event-driven simulator identifies bottlenecks and yields >10% additional efficiency gains.

Significance. If the reported sparsity levels, accuracy retention, and simulator-derived efficiency improvements are reproducible and translate to hardware, the work would provide a concrete demonstration of neuromorphic techniques applied to state-space models for edge ASR, with the FATReLU activation and co-exploration simulator as potentially reusable contributions. The empirical focus on a public dataset is a strength, though the absence of hardware measurements limits the immediate applicability to real neuromorphic deployments.

major comments (2)
  1. [Abstract] Abstract: the concrete claims of >60% sparsity with <1% accuracy loss and >70% sparsity with 30% parameter reduction are presented without any description of training protocols, baseline models (e.g., standard SpeechMamba or other SNNs), number of runs, or error bars, rendering the central empirical results impossible to evaluate for statistical reliability.
  2. [Simulator results section] Simulator results section: the reported >10% additional efficiency from the cycle-accurate event-driven simulator rests on the unvalidated assumption that modeled event-driven and spiking costs (membrane updates, event routing, SSM state carry-over) match physical neuromorphic hardware; no cross-validation or comparison to real-chip measurements is described, which is load-bearing for the hardware-benefit claims.
minor comments (1)
  1. [Method] The definition and implementation details of the FATReLU activation (threshold parameter, event generation logic) are referenced but not fully specified in the provided text, which would aid reproducibility.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive feedback. We address each major comment point-by-point below, providing the strongest honest defense of the manuscript while acknowledging limitations where they exist.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the concrete claims of >60% sparsity with <1% accuracy loss and >70% sparsity with 30% parameter reduction are presented without any description of training protocols, baseline models (e.g., standard SpeechMamba or other SNNs), number of runs, or error bars, rendering the central empirical results impossible to evaluate for statistical reliability.

    Authors: The abstract serves as a concise high-level summary of the key empirical outcomes. Full details on training protocols (surrogate gradient learning for the spiking variant and standard optimization for the event-driven FATReLU model) appear in Section 3. Baseline models, including comparisons to vanilla SpeechMamba and prior SNNs, are defined and evaluated in Section 4. All reported sparsity and accuracy figures are averaged over five independent runs with standard deviation error bars shown in Tables 2–4. These elements allow statistical evaluation from the complete manuscript. We can revise the abstract to include a brief parenthetical reference to the experimental protocol if space permits. revision: partial

  2. Referee: [Simulator results section] Simulator results section: the reported >10% additional efficiency from the cycle-accurate event-driven simulator rests on the unvalidated assumption that modeled event-driven and spiking costs (membrane updates, event routing, SSM state carry-over) match physical neuromorphic hardware; no cross-validation or comparison to real-chip measurements is described, which is load-bearing for the hardware-benefit claims.

    Authors: The cycle-accurate simulator models event-driven costs using established parameters from the neuromorphic literature (membrane potential updates, sparse event routing, and SSM state propagation). The >10% efficiency improvement arises from simulator-guided identification of routing and state-update bottlenecks, enabling targeted algorithmic changes. We agree that the absolute hardware gains remain modeled rather than directly measured on physical chips; no cross-validation against real neuromorphic hardware is provided because such platforms were unavailable during the study. The simulator nonetheless supplies a reproducible tool for co-exploration, as stated in the contributions. revision: no

standing simulated objections not resolved
  • Absence of direct measurements on physical neuromorphic hardware to validate simulator predictions.

Circularity Check

0 steps flagged

No circularity; empirical measurements only

full rationale

The paper proposes event-driven and spiking variants of SpeechMamba, reports measured activation sparsity (>60% and >70%), accuracy degradation (<1%), parameter reduction (30%), and simulator efficiency gains (>10%) on LibriSpeech. These are direct experimental outcomes from model modifications and cycle-accurate simulation, with no derivation chain, equations, or first-principles results that reduce to inputs by construction. No self-definitional steps, fitted-input predictions, load-bearing self-citations, uniqueness theorems, or ansatz smuggling appear. The work is self-contained against external benchmarks (public dataset, simulator), consistent with the reader's assessment of score ~2.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 2 invented entities

The work rests on standard assumptions of deep learning training (gradient descent on cross-entropy loss) and the representativeness of LibriSpeech for real-world ASR; no new mathematical axioms or invented physical entities are introduced beyond the FATReLU activation function and the simulator.

free parameters (1)
  • FATReLU threshold
    Threshold value chosen to achieve target sparsity while preserving accuracy; value not stated in abstract.
axioms (1)
  • domain assumption Standard backpropagation training converges to a usable local minimum for the modified activations
    Implicit in all reported accuracy numbers
invented entities (2)
  • FATReLU activation no independent evidence
    purpose: Produce event-driven sparsity in Mamba blocks
    New activation function introduced to replace standard ReLU or GELU
  • cycle-accurate event-driven simulator no independent evidence
    purpose: Model hardware bottlenecks and measure efficiency gains
    New software tool developed for algorithm-hardware co-exploration

pith-pipeline@v0.9.1-grok · 5751 in / 1513 out tokens · 33372 ms · 2026-06-28T16:19:46.359608+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 6 canonical work pages · 2 internal anchors

  1. [1]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wuet al., “Conformer: Convolution-augmented transformer for speech recognition,” inProc. Interspeech 2020, 2020, pp. 5036–5040

    2020

  2. [2]

    Speech-mamba: Long-context speech recogni- tion with selective state spaces models,

    X. Gao and N. F. Chen, “Speech-mamba: Long-context speech recogni- tion with selective state spaces models,” in2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 1–8

    2024

  3. [3]

    Conformer-based speech recognition on ex- treme edge-computing devices,

    M. Xu, A. Jin, S. Wang, M. Su, T. Ng, H. Mason, S. Han, Z. Lei, Y . Deng, Z. Huanget al., “Conformer-based speech recognition on ex- treme edge-computing devices,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), 2024, pp. 131–139

    2024

  4. [4]

    Efficient processing of deep neural networks: A tutorial and survey,

    V . Sze, Y .-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient processing of deep neural networks: A tutorial and survey,”Proceedings of the IEEE, vol. 105, no. 12, pp. 2295–2329, 2017

    2017

  5. [5]

    Training spiking neural networks using lessons from deep learning,

    J. K. Eshraghian, M. Ward, E. O. Neftci, X. Wang, G. Lenz, G. Dwivedi, M. Bennamoun, D. S. Jeong, and W. D. Lu, “Training spiking neural networks using lessons from deep learning,”Proceedings of the IEEE, vol. 111, no. 9, pp. 1016–1054, 2023

    2023

  6. [6]

    Optimizing event-based neural networks on digital neuromorphic architecture: a comprehensive design space exploration,

    Y . Xu, K. Shidqi, G.-J. van Schaik, R. Bilgic, A. Dobrita, S. Wang, R. Meijer, P. Nembhani, C. Arjmand, P. Martinelloet al., “Optimizing event-based neural networks on digital neuromorphic architecture: a comprehensive design space exploration,”Frontiers in Neuroscience, vol. 18, p. 1335422, 2024

    2024

  7. [7]

    Context-aware sparse spatiotemporal learning for event-based vision,

    S. Wang and G. Tang, “Context-aware sparse spatiotemporal learning for event-based vision,” in2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025, pp. 13 713–13 719

    2025

  8. [8]

    Explore activation sparsity in recurrent llms for energy- efficient neuromorphic computing,

    I. Knunyants, M. Tavakol, M. Sifalakis, Y . Xu, A. Yousefzadeh, and G. Tang, “Explore activation sparsity in recurrent llms for energy- efficient neuromorphic computing,” in2025 IEEE 7th International Conference on Artificial Intelligence Circuits and Systems (AICAS), 2025, pp. 1–5

    2025

  9. [9]

    Deep residual spiking neural network for keyword spotting in low-resource settings

    Q. Yang, Q. Liu, and H. Li, “Deep residual spiking neural network for keyword spotting in low-resource settings.” inInterspeech, 2022, pp. 3023–3027

    2022

  10. [10]

    Dpsnn: spiking neural network for low-latency streaming speech enhancement,

    T. Sun and S. Boht ´e, “Dpsnn: spiking neural network for low-latency streaming speech enhancement,”Neuromorphic Computing and Engi- neering, vol. 4, no. 4, p. 044008, 2024

    2024

  11. [11]

    Iml-spikeformer: Input-aware multilevel spiking transformer for speech processing,

    Z. Song, S. Zhang, Y . Chou, J. Wu, and H. Li, “Iml-spikeformer: Input-aware multilevel spiking transformer for speech processing,”IEEE Transactions on Neural Networks and Learning Systems, 2025

    2025

  12. [12]

    Randomize and match: Exploiting irregular sparsity for energy efficient processing in snns,

    F. Liu, Z. Wang, W. Zhao, Y . Chen, T. Yang, X. Yang, and L. Jiang, “Randomize and match: Exploiting irregular sparsity for energy efficient processing in snns,” in2022 IEEE 40th International Conference on Computer Design (ICCD). IEEE, 2022, pp. 451–454

    2022

  13. [13]

    Loihi: A neuromorphic manycore processor with on-chip learning,

    M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y . Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jainet al., “Loihi: A neuromorphic manycore processor with on-chip learning,”Ieee Micro, vol. 38, no. 1, pp. 82–99, 2018

    2018

  14. [14]

    Inducing and exploiting activation sparsity for fast inference on deep neural networks,

    M. Kurtz, J. Kopinsky, R. Gelashvili, A. Matveev, J. Carr, M. Goin, W. Leiserson, S. Moore, N. Shavit, and D. Alistarh, “Inducing and exploiting activation sparsity for fast inference on deep neural networks,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 5533–5543

    2020

  15. [15]

    Ibex: A small 32-bit RISC-V CPU core,

    lowRISC contributors, “Ibex: A small 32-bit RISC-V CPU core,” https: //github.com/lowRISC/ibex, 2024

    2024

  16. [16]

    Neural Machine Translation by Jointly Learning to Align and Translate

    D. Bahdanau, “Neural machine translation by jointly learning to align and translate,”arXiv preprint arXiv:1409.0473, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  17. [17]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” inFirst conference on language modeling, 2024

    2024

  18. [18]

    An Exploration of Mamba for Speech Self-Supervised Models

    T.-Q. Lin, H.-C. Kuo, T.-C. Wei, H.-C. Cheng, C.-W. Chen, H.-F. Hsiao, Y . Tsao, and H.-y. Lee, “An exploration of mamba for speech self- supervised models,”arXiv preprint arXiv:2506.12606, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [19]

    Spikformer: When spiking neural network meets transformer.arXiv preprint arXiv:2209.15425, 2022

    Z. Zhou, Y . Zhu, C. He, Y . Wang, S. Yan, Y . Tian, and L. Yuan, “Spikformer: When spiking neural network meets transformer,”arXiv preprint arXiv:2209.15425, 2022

    work page arXiv 2022

  20. [20]

    Spikmamba: When snn meets mamba in event-based human action recognition,

    J. Chen, Y . Yang, S. Deng, D. Teng, and L. Pan, “Spikmamba: When snn meets mamba in event-based human action recognition,” inProceedings of the 6th ACM International Conference on Multimedia in Asia, ser. MMAsia ’24. New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3696409.3700204

    work page doi:10.1145/3696409.3700204 2024

  21. [21]

    Seneca: building a fully digital neuromorphic processor, design trade-offs and challenges,

    G. Tang, K. Vadivel, Y . Xu, R. Bilgic, K. Shidqi, P. Detterer, S. Traferro, M. Konijnenburg, M. Sifalakis, G.-J. van Schaiket al., “Seneca: building a fully digital neuromorphic processor, design trade-offs and challenges,” Frontiers in Neuroscience, vol. 17, p. 1187252, 2023

    2023

  22. [22]

    Timeloop: A systematic approach to dnn accelerator evaluation,

    A. Parashar, P. Raina, Y . S. Shao, Y .-H. Chen, V . A. Ying, A. Mukkara, R. Venkatesan, B. Khailany, S. W. Keckler, and J. Emer, “Timeloop: A systematic approach to dnn accelerator evaluation,” in2019 IEEE inter- national symposium on performance analysis of systems and software (ISPASS). IEEE, 2019, pp. 304–315

    2019

  23. [23]

    Nnasim: An efficient event-driven simulator for dnn accelerators with accurate timing and area models,

    X. Yi, J. Yu, Z. Wu, X. Xiong, D. Xu, C. Chen, J. Tao, and F. Yang, “Nnasim: An efficient event-driven simulator for dnn accelerators with accurate timing and area models,” in2022 IEEE International Sympo- sium on Circuits and Systems (ISCAS), 2022, pp. 2806–2810

    2022

  24. [24]

    An event-based digital compute-in-memory accelerator with flexible operand resolution and layer-wise weight/output stationarity,

    N. Chauvaux, A. Kneip, C. Posch, K. Makinwa, and C. Frenkel, “An event-based digital compute-in-memory accelerator with flexible operand resolution and layer-wise weight/output stationarity,” in2025 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2025, pp. 1–5

    2025

  25. [25]

    Ample: Event-driven accelerator for mixed-precision inference of graph neural networks,

    P. Gimenes, A. Zhao, and G. A. Constantinides, “Ample: Event-driven accelerator for mixed-precision inference of graph neural networks,” in Proceedings of the 5th Workshop on Machine Learning and Systems, 2025, pp. 107–113

    2025

  26. [26]

    Pycarl: A pynn interface for hardware-software co-simulation of spiking neural network,

    A. Balaji, P. Adiraju, H. J. Kashyap, A. Das, J. L. Krichmar, N. D. Dutt, and F. Catthoor, “Pycarl: A pynn interface for hardware-software co-simulation of spiking neural network,” in2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020, pp. 1–10

    2020

  27. [27]

    Legendre memory units: Continuous-time representation in recurrent neural networks,

    A. V oelker, I. Kaji ´c, and C. Eliasmith, “Legendre memory units: Continuous-time representation in recurrent neural networks,”Advances in neural information processing systems, vol. 32, 2019

    2019

  28. [28]

    Event-based optical flow on neuromorphic processor: Ann vs. snn comparison based on activation sparsification,

    Y . Xu, G. Tang, A. Yousefzadeh, G. C. de Croon, and M. Sifalakis, “Event-based optical flow on neuromorphic processor: Ann vs. snn comparison based on activation sparsification,”Neural Networks, vol. 188, p. 107447, 2025

    2025

  29. [29]

    Spinnaker 2: A 10 million core processor system for brain simulation and machine learning-keynote presentation,

    C. Mayr, S. Hoeppner, and S. Furber, “Spinnaker 2: A 10 million core processor system for brain simulation and machine learning-keynote presentation,” inCommunicating Process Architectures 2017 & 2018. IOS Press, 2019, pp. 277–280

    2017

  30. [30]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  31. [31]

    Accurate and structured pruning for efficient automatic speech recognition,

    H. Jiang, L. L. Zhang, Y . Li, Y . Wu, S. Cao, T. Cao, Y . Yang, J. Li, M. Yang, and L. Qiu, “Accurate and structured pruning for efficient automatic speech recognition,”arXiv preprint arXiv:2305.19549, 2023

    work page arXiv 2023

  32. [32]

    Spike- driven transformer,

    M. Yao, J. Hu, Z. Zhou, L. Yuan, Y . Tian, B. Xu, and G. Li, “Spike- driven transformer,”Advances in neural information processing systems, vol. 36, pp. 64 043–64 058, 2023

    2023

  33. [33]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  34. [34]

    SpeechBrain: A general-purpose speech toolkit,

    M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong et al., “Speechbrain: A general-purpose speech toolkit,”arXiv preprint arXiv:2106.04624, 2021

    work page arXiv 2021