Pith. sign in

REVIEW 4 major objections 5 minor 24 references

Early Attentive Sparsification Accelerates Neural Speech Transcription

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that dropping audio tokens with low mean attention scores early in a Whisper encoder, with no fine-tuning and no weight changes, delivers up to 1.6x faster GPU inference while keeping word error rate within 1% of the…

desk verdict A solid practical token-dropping result for Whisper ASR, but the accuracy claims rest entirely on a 300-example subset used for both search and reporting. read the letter →

arxiv 2506.15912 v1 pith:4FGGSO3R submitted 2025-06-18 cs.LG cs.CLcs.SDeess.AS

Zifei Xu , Sayeh Sharify , Hesham Mostafa , Tristan Webb , Wanzin Yazar , Xin Wang This is my paper · ORCID
classification cs.LGcs.CLcs.SDeess.AS
keywords automaticspeechrecognitiontokendroppingattentionsparsificationWhisperinferenceaccelerationworderrorraterealtimefactortransformerencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that speech tokens in transformer audio encoders are redundant enough to be safely dropped early in the encoding stage, and that a simple attention-based importance score can pick which tokens to drop. The proposed mechanism, early attentive sparsification (EAS), shortens the temporal sequence passed through the rest of the encoder and decoder, cutting compute without retraining. The authors report that across ten Whisper and Distil-Whisper models, the best configurations sparsify 40-60% of audio tokens at an early layer and achieve up to 1.6x runtime speedup on an A100 GPU while preserving at least 99% of baseline accuracy. This matters because it offers a parameter-free, fine-tuning-free lever for reducing ASR latency that is orthogonal to weight compression and kernel-level optimizations.

What carries the argument

Early attentive sparsification (EAS) is a token-dropping layer inserted at an encoder stage i. It computes an importance score for each audio token as the arithmetic mean, across attention heads and source positions, of the previous layer's post-softmax self-attention scores, then gathers the top k tokens where k = floor((1-s)T), reducing the temporal dimension from T to about (1-s)T. The reduced sequence then flows through the remaining encoder layers and the decoder, so the mechanism's two effects are a one-time scoring overhead and a persistent reduction in attention and feedforward compute. The empirical argument is carried by a grid search over (i, s) with a Pareto front of word-error-rate versus real-time-factor, constrained to at least 99% of baseline accuracy.

What would settle it

Run the reported (i, s) configurations and their baselines on all 2,703 examples of the LibriSpeech clean validation split and compute the accuracy ratio (1 - WER_sparsified) / (1 - WER_baseline) for each reported configuration; if any advertised configuration falls below 0.99 on the full split, the central claim is refuted.

Watch

Extended reading notes

Core claim

Audio tokens in Whisper-family encoders are compressible in the time domain: keeping the top 40-60% of tokens by mean self-attention importance, at one of the first few encoder layers, preserves at least 99% of the baseline accuracy (defined as 1 - WER) while reducing real-time factor by up to 1.6x on an Nvidia A100 GPU. The effect holds across model sizes from 37.8M to 1.54B parameters, with the largest speedups appearing in already-compressed models such as Distil-Whisper and whisper-large-v3-turbo, suggesting that EAS compounds with existing compression rather than duplicating it.

Load-bearing premise

The accuracy numbers that define the 1% constraint are computed on a 300-example subset of LibriSpeech clean validation, and the adequacy check covers only one configuration; if that subset's word error rates differ from the full validation split for the configurations actually selected, the under-1%-degradation claim could fail.

Editorial extensions

If this is right

  • EAS can be applied to a Whisper or Distil-Whisper model without any fine-tuning, so it is a drop-in runtime optimization for already-deployed checkpoints.
  • The best recipes cluster at 40-60% sparsity in the first few encoder layers, which implies the redundancy is in the early embedded audio representation rather than in later semantic layers.
  • Because only one encoder layer needs eager attention to supply the importance scores, EAS can be combined with FlashAttention-2 or PyTorch SDPA for the remaining layers and still inherit their speedups.
  • Distilled models gain more than their teachers, so EAS appears synergistic with knowledge-distillation-based compression for ASR.
  • High sparsity (80-90%) causes the decoder to generate repetitive text, meaning there is a practical upper bound on token dropping before accuracy degrades sharply.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not test is streaming ASR, where reducing the token count early could lower per-chunk latency on long utterances even more than the batch-mode wall-clock speedups reported here.
  • The observed repetitive-decoder failure at high sparsity suggests a confidence-aware policy could stop dropping once decoding begins to loop, which would extend the usable sparsity range.
  • Because the importance signal is just mean attention from one layer, the same search could be run on other encoder-decoder speech models or on multilingual and noisy-speech benchmarks to test whether the 40-60% sweet spot is a general property of learned audio encoders.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Early Attentive Sparsification (EAS), a training-free method that shortens the audio-token sequence inside Whisper-family ASR models by keeping only top-k frames according to a mean self-attention importance score at an early encoder stage. The authors perform a grid search over (sparsification layer, sparsity) across ten public Whisper and Distil-Whisper models, constrain admissible configurations to those preserving at least 99% of baseline accuracy (defined via 1 - WER), and report WER and real-time-factor (RTF) for the top configurations, with up to 1.6x speedup. Appendices address dataset-size adequacy, importance-score aggregation choices, runtime decomposition, and compatibility with FlashAttention-2 and PyTorch SDPA.

Significance. If the accuracy constraint survives evaluation on an independent or full validation set, EAS is a practically useful and refreshingly simple latency optimization: it requires no fine-tuning, it acts on sequence length rather than weights, and it appears complementary to kernel-level and weight-compression speedups. The qualitative finding that early 40-60% sparsity gives the best accuracy-latency tradeoff is consistent across ten models spanning three orders of magnitude in parameter count, which is the paper's strongest empirical contribution. The method is simple enough to reproduce, and the runtime decomposition and attention-implementation compatibility experiments strengthen the latency claims. The main limitation is that all WER numbers used both for configuration selection and for final reporting come from one 300-example subset, with no full-split validation of the chosen recipes.

major comments (4)
  1. [Section 3.1, Section 6.1, Table 1] The headline 'under 1% accuracy degradation' is not yet anchored to an independent evaluation. All WER values in Table 1 and in the architecture search come from the same 300-example subset that is used to select configurations via the constraint in Equation (3); selecting (i,s) on that subset conditions on the constraint being satisfied there, so the reported ratios are optimistic by construction. The margins are thin: whisper-base (1,0.6) is at ratio 0.990, and whisper-tiny (3,0.7), whisper-medium (7,0.7), and whisper-large-v3-turbo (2,0.6) are at 0.992. At 300 examples, the standard error of WER at the reported 2-7% levels is of order 0.3-0.6% absolute, so a 0.1-0.2% accuracy-ratio gap is within the noise. Please re-evaluate every recommended configuration on the full 2,703-example LibriSpeech clean validation split and report full-set WER and accuracy ratios, ideally with bootstrap confidence intervals.
  2. [Section 3.2, Equations (1)-(2)] Configurations with sparsification stage i=1 appear throughout Table 1, but Equation (1) defines the importance score from the previous encoder layer's post-softmax self-attention score A^{l-1}. For i=1 there is no previous encoder layer, so A^0 is undefined. The paper needs to state explicitly how importance scores are obtained for stage 1 (for example, from the first encoder layer's attention before sparsification, or from some other representation), or restrict the search to i>=2; without this clarification the i=1 entries are not reproducible.
  3. [Section 6.1, Figure 3] The adequacy analysis for the 300-example subset is performed on a single configuration, whisper-large-v3-turbo at layer 21 with sparsity 0.6, which is a late-layer configuration rather than one of the recommended early-stage recipes. That analysis therefore does not establish that the subset is adequate for the configurations actually selected in the search, e.g., layers 1-3 with sparsity 0.4-0.7. The adequacy analysis should cover the chosen early-stage recipes, or the final accuracy claim should be verified on the full validation split.
  4. [Table 1, Figure 2] Uncertainty reporting is missing from the main empirical claims. Table 1 gives no confidence intervals or repeated-run variability for either WER or RTF, and Figure 2 plots single measured values. Given that several reported accuracy ratios sit within a few tenths of a percent of the 0.99 boundary and that RTF at batch size 1 is subject to system noise, the main claims would be substantially strengthened by bootstrap intervals over the evaluation subset and by repeated runtime measurements for the reported top configurations.
minor comments (5)
  1. [Section 6.2] The sentence 'We asked if any statistics than than arithmetic mean serve as a better aggregation function' contains a duplicated 'than' and should read 'whether any statistic other than the arithmetic mean...'.
  2. [Equation (3)] The optimization constraint mixes percentage notation used in the tables with a ratio; please state explicitly whether WER in Equation (3) is expressed as a fraction or as a percentage.
  3. [Section 6.4] The phrase 'Appendix 6.4 demonstrates...' refers to the appendix itself; it should read 'we demonstrate' or 'Section 6.4 demonstrates'.
  4. [Figure 2] With ten models and many configurations, the symbol and color encoding in Figure 2 is difficult to read; consider per-model small multiples or a separate panel for each model.
  5. [Table 1 caption] The caption says 'Parenthesized numbers show relative changes from baseline'; for WER these numbers are accuracy ratios (1-WER)/(1-WER0), not simple relative WER changes. Please clarify the caption or text.

Circularity Check

1 steps flagged · score 6.0 of 10

Accuracy-degradation claim is enforced by the same 300-example WER used to select configurations; reported ≥0.99 ratios are true by construction on the search subset.

  1. fitted input called prediction [Section 3.1, Section 3.4 (Eq. 3), Table 1, Section 4]
    "‘A subset, 300 task examples, of the clean validation split is used for architecture search’ (Sec. 3.1); Eq. (3): ‘1−WER(i, s) ≥ 0.99 (1−WER0)’; Table 1 caption: ‘WER and RTF of the top-3 architecture search results for each model’; Sec. 4: ‘We identify top-3 configurations (circled) on the Pareto fronts of each model that satisfy the accuracy constraint as admissible solutions to report in Table 1.’"

    The Table 1 entries are exactly the admissible solutions to Eq. (3), and Eq. (3)'s accuracy constraint is evaluated with WER measured on the same 300-example subset used to run the search. Hence every reported accuracy ratio of 0.99 or higher is guaranteed by construction on that subset: the 'under 1% degradation' statement is the selection filter itself, not an independently tested prediction. No full validation-split evaluation of the chosen (i,s) recipes is reported; Appendix 6.1's adequacy check measures only a different configuration (layer 21, sparsity 0.6), so it does not break the circularity for the headline entries.

full rationale

This paper is an empirical study rather than a formal derivation, and there is no self-citation chain: EAS is an explicitly credited adaptation of cumulative-attention token dropping from LLM KV-cache work (Wang et al. 2021; Zhang et al. 2023a), and runtime measurements are self-contained wall-clock benchmarks. The load-bearing accuracy claim ('under 1% accuracy degradation' / 'maintains ≥99% of baseline accuracy') is, however, circular in the statistical sense: the configurations reported in Table 1 are the solutions of the constrained optimization in Eq. (3), whose constraint is evaluated with WER on the same 300-example subset that defines the architecture search. Every reported ratio ≥0.99 is therefore true on that subset by construction; the paper does not re-evaluate the chosen (i,s) recipes on the full 2,703-example split, and the Appendix 6.1 adequacy check covers only a different, late-layer configuration (layer 21, sparsity 0.6). Several headline entries (whisper-base (1,0.6): 0.990; whisper-large-v3-turbo (2,0.6): 0.992) sit within the standard error of the 0.99 boundary, so the '≤1% degradation' generalization is not independently established. The aggregation-function choice in Appendix 6.2 is also selected using WER on the same benchmark, a milder form of the same selection-on-evaluation issue. No other circular steps were found; the speedup numbers and the early-stage/40-60% sparsity observation have independent empirical content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions: attention-mean is a valid importance oracle for audio tokens; audio sequences are redundant enough for early token dropping; the 300-example LibriSpeech subset gives WER estimates accurate enough for the 1%-accuracy constraint; and sparsifier overhead is small in wall-clock RTF. Free parameters are limited to the hand-chosen accuracy tolerance and the chosen aggregation function. No invented entities are introduced.

free parameters (2)
  • accuracy degradation tolerance = 0.01 relative (1%)
    Hand-chosen constraint in Equation (3) defining admissible solutions; the reported 'under 1% accuracy degradation' is this threshold, not an emergent bound from the data.
  • importance aggregation function = arithmetic mean
    Chosen because an appendix comparison (Table 2, Figure 6) showed mean giving the lowest WER when dropping at the last encoder layer; the central claim depends on this selection.
assumptions (4)
  • domain assumption Mean of post-softmax attention over heads and source tokens is a valid measure of token importance for audio tokens.
    Equation (1) borrows the heavy-hitter heuristic from LLM token-eviction literature; no theoretical justification is given, and the paper validates it only empirically, mainly at the last layer.
  • domain assumption Audio hidden-state sequences are sufficiently redundant that a fraction of tokens can be dropped early with little loss.
    Stated in the introduction as known speech compressibility; underpins the entire approach and the interpretation of 40-60% sparsity as removing redundant tokens.
  • domain assumption The 300-example LibriSpeech clean validation subset yields WER estimates consistent with the full validation set for all configurations considered.
    Section 3.1 and Appendix 6.1; the adequacy analysis covers only one configuration on one model, yet the architecture search and headline numbers rely on this subset.
  • domain assumption The runtime overhead of the sparsifier (top-k computation and gather) is negligible relative to the speedup from shorter sequences.
    Section 3.3 measures wall-clock RTF, so overhead is implicitly included, but the paper does not isolate the sparsifier cost from the savings in the reported speedups.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Early Attentive Sparsification Accelerates Neural Speech Transcription." pith.science (2026). https://pith.science/paper/4FGGSO3R

@misc{pith2026250615912,
  author       = {Pith},
  title        = {Pith review of: Early Attentive Sparsification Accelerates Neural Speech Transcription},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4FGGSO3R}},
  note         = {Machine review of arXiv:2506.15912}
}
read the original abstract

Transformer-based neural speech processing has achieved state-of-the-art performance. Since speech audio signals are known to be highly compressible, here we seek to accelerate neural speech transcription by time-domain signal sparsification early in the neural encoding stage, taking advantage of the interpretability of the self-attention mechanism in transformer audio encoders. With the Whisper family of models, we perform a systematic architecture search over the joint space of sparsification stage (a certain encoder layer) and compression ratio (sparsity). We found that the best resulting solutions under 1% accuracy degradation choose to sparsify the hidden state to 40-60% sparsity at an early encoding stage, and thereby achieve up to 1.6x runtime acceleration in English speech transcription tasks on Nvidia GPUs without any fine-tuning.

Figures

Figures reproduced from arXiv: 2506.15912 by the authors.

Figure 1
Figure 1. Architecture of the Whisper model modified with early attentive sparsification. The original Whisper model is composed of an encoder stack E’s (middle column) and a decoder stack D’s (right column), of depth L. Flowing through the encoder stack are hidden states of sequence dimension T, determined by embedded audio signal duration. At an early encoding stage i ∈ {1, · · · , L}, we compress the hidden state by a time… view at source ↗
Figure 2
Figure 2. Empirical results of architecture search for OpenAI’s Whisper (top) and HuggingFace’s Distil-Whisper (bottom) models, showing tradeoff between WER and RTF in the context of the benchmark transcription task. Each colored symbol represents a unique (i, s) configuration (see Section 3) for a certain model (identity of the model encoded by symbol shape). Large, black symbols represent the baseline configuration for each… view at source ↗
Figure 3
Figure 3. Accuracy ratio between model with encoder sparsi￾fication and baseline model at different dataset sizes, where accuracy = 1 − W ER. Hollow circles represent mean of accu￾racy degradation with error bars for one standard deviation. 6.2. Aggregation function for importance score We asked if any statistics than than arithmetic mean serve as a better aggregation function in computing importance scores from attention sco… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Average number of tokens generated per example vs spar￾sity for whisper-large-v3-turbo model at sparsification layer of 20. The dashed line indicates the average number of to￾kens generated per example for the baseline model. et al., 2023) because we benchmark with flo…
Figure 4
Figure 4. Figure 4: Average time spent at different components of the Whisper model per example vs sparsity for whisper-large-v3-turbo model at sparsification layer of 20. To reduce the effect of system noise, the same experiment was repeated 3 times and plotted on top of each other. 6.4.…
Figure 6
Figure 6. Figure 6: WER versus sparsity for dropping at the last encoder layer using importance score aggregated across all encoder layers. Sub figures correspond to different models and colored lines represent various aggregation function, with random meaning randomly dropping audio samp…
Figure 7
Figure 7. Figure 7: Empirical results of architecture search for the OpenAI whisper-large-v3-turbo model with different attention imple￾mentations. Convention same as in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 8 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492--28518. PMLR, 2023

    2023

  3. [3]

    Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks

    Alex Graves, Santiago Fern \'a ndez, Faustino Gomez, and J \"u rgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369--376, 2006

    work page 2006

  4. [4]

    Deep speech 2: End-to-end speech recognition in english and mandarin

    Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning, pages 173--182. PMLR, 2016

    work page 2016

  5. [5]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 0 12449--12460, 2020

    2020

  6. [6]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36: 0 34661--34710, 2023 a

    work page 2023

  7. [7]

    Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache

    Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Atlas Wang. Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems, 6: 0 381--394, 2024

    work page 2024

  8. [8]

    Spatten: Efficient sparse attention architecture with cascade token and head pruning

    Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 97--110. IEEE, 2021

    work page 2021

Show all 24 references
  1. [9]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  2. [10]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:...

    2021

  3. [11]

    Librispeech: An asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206--5210, 2015. doi:10.1109/ICASSP.2015.7178964

    2015

  4. [12]

    Google USM : Scaling automatic speech recognition beyond 100 languages

    Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. Google USM : Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037, 2023 b

    2023 arXiv

  5. [13]

    Anatomy of industrial scale multilingual ASR

    Francis McCann Ramirez, Luka Chkhetiani, Andrew Ehrenberg, Robert McHardy, Rami Botros, Yash Khare, Andrea Vanzo, Taufiquzzaman Peyash, Gabriel Oexle, Michael Liang, et al. Anatomy of industrial scale multilingual ASR . arXiv preprint arXiv:2404.09841, 2024

    2024 arXiv

  6. [14]

    Conformer: Convolution-augmented transformer for speech recognition

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020

    2005 arXiv

  7. [15]

    Fast conformer with linearly scalable attention for efficient speech recognition

    Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvvada, Ankur Kumar, Jagadeesh Balam, et al. Fast conformer with linearly scalable attention for efficient speech recognition. In 2023 IEEE Automatic Speech...

    2023

  8. [16]

    Sequence transduction with recurrent neural networks

    Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711, 2012

    2012 arXiv

  9. [17]

    Rnn-transducer with stateless prediction network

    Mohammadreza Ghodsi, Xiaofeng Liu, James Apfel, Rodrigo Cabrera, and Eugene Weinstein. Rnn-transducer with stateless prediction network. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7049--7053. IEEE, 2020

    2020

  10. [18]

    Pruned rnn-t for fast, memory-efficient asr training

    Fangjun Kuang, Liyong Guo, Wei Kang, Long Lin, Mingshuang Luo, Zengwei Yao, and Daniel Povey. Pruned rnn-t for fast, memory-efficient asr training. arXiv preprint arXiv:2206.13236, 2022

    2022 arXiv

  11. [19]

    End-to-end speech recognition: A survey

    Rohit Prabhavalkar, Takaaki Hori, Tara N Sainath, Ralf Schl \"u ter, and Shinji Watanabe. End-to-end speech recognition: A survey. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32: 0 325--351, 2023

    2023

  12. [20]

    Revisiting convolution-free transformer for speech recognition

    Zejiang Hou, Goeric Huybrechts, Anshu Bhatia, Daniel Garcia-Romero, Kyu Han, and Katrin Kirchhoff. Revisiting convolution-free transformer for speech recognition. 2024

    2024

  13. [21]

    Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling

    Sanchit Gandhi, Patrick von Platen, and Alexander M Rush. Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling. arXiv preprint arXiv:2311.00430, 2023

    2023 arXiv

  14. [22]

    Open automatic speech recognition leaderboard

    Vaibhav Srivastav, Somshubra Majumdar, Nithin Koluguri, Adel Moumen, Sanchit Gandhi, et al. Open automatic speech recognition leaderboard. https://huggingface.co/spaces/hf-audio/open_asr_leaderboard, 2023

    2023

  15. [23]

    Liteasr: Efficient automatic speech recognition with low-rank approximation, 2025

    Keisuke Kamahori, Jungo Kasai, Noriyuki Kojima, and Baris Kasikci. Liteasr: Efficient automatic speech recognition with low-rank approximation, 2025. URL https://arxiv.org/abs/2502.20583

    2025 arXiv

  16. [24]

    Flashattention-2: Faster attention with better parallelism and work partitioning, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.