Bidirectional recurrent attention with Direction Dropout matches or exceeds multi-head attention accuracy in a Conformer-Transducer ASR system while increasing throughput by up to 44 percent.
Multi-Head State Space Model for Speech Recognition
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
State space models (SSMs) have recently shown promising results on small-scale sequence and language modelling tasks, rivalling and outperforming many attention-based approaches. In this paper, we propose a multi-head state space (MH-SSM) architecture equipped with special gating mechanisms, where parallel heads are taught to learn local and global temporal dynamics on sequence data. As a drop-in replacement for multi-head attention in transformer encoders, this new model significantly outperforms the transformer transducer on the LibriSpeech speech recognition corpus. Furthermore, we augment the transformer block with MH-SSMs layers, referred to as the Stateformer, achieving state-of-the-art performance on the LibriSpeech task, with word error rates of 1.76\%/4.37\% on the development and 1.91\%/4.36\% on the test sets without using an external language model.
citation-role summary
background 1
citation-polarity summary
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR
Bidirectional recurrent attention with Direction Dropout matches or exceeds multi-head attention accuracy in a Conformer-Transducer ASR system while increasing throughput by up to 44 percent.