REVIEW 4 major objections 6 minor 42 references
BEST-STD: Bidirectional Mamba-Enhanced Speech Tokenization for Spoken Term Detection
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper claims that query-by-example spoken term detection can be replaced by discrete, speaker-agnostic token sequences from a bidirectional Mamba encoder, enabling inverted-index retrieval that beats DTW-based baselines on LibriSpeech…
desk verdict A plausible new speech tokenization approach for spoken term detection, but the evaluation protocol is under-specified and the runtime claim is unmeasured; the method deserves peer review with major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the bidirectional Mamba encoder: two identical Mamba state-space blocks process the input sequence forward and time-reversed, and their outputs are combined and projected into $\ell^2$-normalized embeddings. A contrastive loss, using DTW to pick positive frame pairs between two utterances of the same word and random frames from other words as negatives, pushes the encoder to make embeddings speaker-invariant; a commitment loss pulls embeddings toward quantizer centroids. The discrete tokens are produced by nearest-centroid assignment, and retrieval rests on an inverted index of token bigrams scored by Jaccard similarity.
What would settle it
A concrete check: corrupt the DTW alignments during training by adding random monotonic frame shifts and measure MAP and MRR on LibriSpeech; if retrieval accuracy does not drop, the DTW-alignment premise is not load-bearing.
Extended reading notes
Core claim
The central discovery is that a bidirectional state-space encoder can be trained, without word labels, to emit token sequences that stay consistent across speakers and across renditions of the same term, and that those tokens are discriminative enough to make the detour through DTW unnecessary. The method learns frame-level contextual embeddings from log-Mel spectra using two Mamba blocks reading the input in opposite temporal directions, aligns pairs of same-word utterances with DTW to define anchor-positive contrastive pairs, and quantizes the embeddings with a vector quantizer updated by exponential moving averages. Retrieval then treats token sequences as text: it builds an inverted index over token bigrams and scores candidate segments by maximum Jaccard similarity. The paper reports that this pipeline consistently outperforms MFCC, phone-posterior, and bottleneck features matched with DTW, as well as tokenizers based on HuBERT, WavLM, SpeechTokenizer, and EnCodec, on both in-vocabulary and out-of-vocabulary query sets.
Load-bearing premise
The self-supervised training depends on DTW alignment between different utterances of the same word being correct at the frame level, so that the contrastive pairs it creates are truly the same sound; if the alignment is wrong, the tokens may become less speaker-invariant.
Editorial extensions
If this is right
- QbE-STD can be run with text-search data structures, so large audio archives can be searched much faster than with segmental DTW.
- The tokenizer's consistency across speakers transfers to better retrieval for both in-vocabulary and out-of-vocabulary terms, because tokens represent subword units rather than whole words.
- Bidirectional Mamba gives more useful temporal modeling for STD than a Transformer trained under the same framework.
- Larger codebooks improve the MTWV retrieval score while slightly lowering MAP and MRR, so codebook size can be tuned to the application.
- Homophones can be tokenized identically, creating false positives that a future extension would need to resolve.
Reading between the lines
- If the tokens are indeed speaker-agnostic subword units, the same encoder could be applied to out-of-domain audio or languages not seen in training, since no word lexicon or ASR is required.
- The DTW-alignment premise could be stress-tested by replacing the aligner during training; if the method is robust, retrieval accuracy should degrade only slowly as alignments are corrupted.
- The inverted-index scheme could be bolted onto any discrete speech tokenizer, meaning the speed gains are not necessarily tied to the Mamba encoder itself.
- A retrieval-time disambiguation step using a small pronunciation or text lexicon could remove the homophone false positives without changing the tokenizer.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes BEST-STD, a query-by-example spoken term detection system built on discrete speech tokens. Speech is mapped to frame embeddings by a bidirectional Mamba encoder, trained with a self-supervised contrastive objective in which DTW-aligned frames from different utterances of the same word form positive pairs, plus a vector-quantization commitment loss. At test time, archive audio is segmented and tokenized into bigram sequences indexed in an inverted index; a query is tokenized and scored against candidate segments by maximum Jaccard similarity. Experiments on LibriSpeech and TIMIT compare the method against DTW-based feature baselines and existing speech tokenizers, reporting higher Jaccard consistency of token sequences and higher MAP/MRR/MTWV, as well as a claim of faster retrieval.
Significance. If the empirical claims hold, the paper would make a useful contribution by showing that discrete, speaker-agnostic speech tokens can support effective query-by-example search with an inverted-index retrieval scheme, avoiding expensive DTW at search time. The self-supervised objective is clearly specified, the comparison covers several representative baselines, and the authors provide a public code URL, which are strengths. The token-consistency analysis, however, largely measures the property that the training objective directly optimizes, so it is not an independent validation of speaker invariance; the independent evidence is the retrieval comparison in Table II, whose current evaluation protocol is under-specified in a way that affects the interpretability of the headline numbers.
major comments (4)
- [III.A and II.D] The evaluation protocol for Table II is under-specified in a way that directly affects the headline accuracy claim. Section III.A states that spoken terms for testing were extracted from the train-clean-100 set, which is the same set used as the speech archive. Section II.D scores every candidate segment against the query token sequence, and the paper never states that the query's source segment or source utterance is removed from the candidate list or from the ground truth. Because each query is a segment of an archive track, the source segment will contain the exact query tokens and will be retrieved with near-perfect Jaccard similarity, giving every query a guaranteed self-match. This inflates MAP/MRR for all systems and makes the numbers in Table II (e.g., MAP 0.86/0.83) not interpretable as detection performance; please state whether self-matches were excluded and, if not, re-run the evaluation with the source segment/utterance removed and report the corrected numbers.
- [II.C and III.D] The retrieval setup is not reproducible because the segment hop h and the segment-level relevance criterion are never specified. Section II.C defines overlapping segments by length l and hop h, but no value for h is given; Section III.D states that 1-second (l) segments were used for training, but does not state the archive indexing parameters. It is also unclear whether a candidate segment is relevant if it overlaps the query, contains the full word, or satisfies some other condition, and how multiple overlapping relevant segments contribute to MAP/MRR/MTWV. Please specify h, the relevance definition, and the pooling rule, and report results under that protocol.
- [IV.B] The efficiency claim is unsupported. The text states that the inverted index delivers "significantly faster retrieval times compared to DTW-based methods," but no runtime, memory, or complexity measurements are reported anywhere. Please provide retrieval-time measurements on the same archive, or a clear complexity analysis, to support the efficiency claim.
- [II.B] The training objective assumes that the DTW alignments in Eq. (4) are semantically correct. If the alignments are noisy, the anchor-positive pairs selected in Eq. (5) may pair frames that are acoustically similar but phonetically unrelated, and the model could learn spurious cross-speaker correspondences. The paper provides no validation of DTW alignment quality or an ablation against alternative alignment strategies; this is a load-bearing assumption for the claimed speaker-agnostic tokenization and should be addressed, for example by measuring alignment consistency or comparing with a simpler fixed-alignment baseline.
minor comments (6)
- [Throughout] There are typos and capitalization inconsistencies, including "retreiving," "meeetings," and "bidrectional" in the Figure 1 caption; please proofread the manuscript.
- [II.C] The notation in Eq. (11) is inconsistent: C is defined as a union of (i,j) index pairs from the inverted index, but the following sentence refers to a "candidate frame c_k" as though C contained frames; please clarify that candidates are indexed segments.
- [II.B, Eq. (5)] In Eq. (5), the symbol z_i appears in the cosine similarity, but z_t is the defined frame embedding; please correct the subscript and define the cosine similarity explicitly.
- [III.C] The EnCodec baseline is cited as "[26]" in Section III.C, but reference [26] is the S4 paper; EnCodec corresponds to reference [25] (Défossez et al.).
- [IV.A] The Jaccard-consistency analysis in Table I measures the same token-consistency property that the training objective directly optimizes, so it is not an independent test of speaker invariance; consider reporting per-speaker-pair statistics or a separate speaker-discrimination evaluation to support the speaker-agnostic claim.
- [III.B and Table II] Table II reports single numbers without significance tests or confidence intervals; given that each query set has only 300 terms, differences between close configurations (e.g., BEST-STD 256 vs. BEST-STD 512) may not be reliable; please add significance tests or confidence intervals, or clarify whether the numbers are averages over multiple runs.
Circularity Check
No significant circularity: central claims are empirical comparisons against external baselines, not derivations from fitted inputs.
full rationale
The paper's main claims are that BEST-STD outperforms existing STD baselines on LibriSpeech/TIMIT retrieval and that its discrete tokens are more speaker-invariant as measured by same-term Jaccard consistency. Neither claim reduces to a fitted input: the retrieval evaluation is against external systems (MFCC, BNF, HuBERT, WavLM, EnCodec, SpeechTokenizer) using standard metrics (MAP, MRR, MTWV), and the token-consistency table compares BEST-STD with those same external tokenizers. The training objective (contrastive loss with DTW-selected anchor-positive pairs plus VQ commitment) is not defined in terms of the evaluation metrics; it optimizes frame-level cosine alignment, while the reported consistency is a separate token-level Jaccard measure. The self-citations [17]-[19] are background references to the authors' prior audio-fingerprinting work and are not load-bearing for the proposed method. No uniqueness theorem or ansatz is imported from prior author work to force the architecture choice. The manuscript does contain evidentiary weaknesses—the retrieval protocol does not state that a query's source segment is excluded from the archive, the segment hop h is unspecified, and the 'significantly faster' runtime claim is not backed by measurements—but these are correctness/evidence concerns, not circularity, and do not make any stated result equivalent to its inputs by construction.
Assumptions & free parameters
free parameters (4)
- Codebook size K =
256, 512, 1024 tested; best MAP/MRR at 256-512, best MTWV at 1024
- Segment length l =
1 second
- Contrastive temperature tau =
0.2
- Commit loss weight lambda =
0.1
assumptions (4)
- domain assumption Different utterances of the same spoken term can be aligned frame-wise with DTW to provide correct positive pairs.
- domain assumption Jaccard similarity over token bigrams is sufficient for detecting spoken terms in overlapping segments.
- standard math HiPPO/Diagonal initialization and zero-order hold discretization provide a valid state-space model for speech.
- standard math Exponential moving average codebook updates prevent codebook collapse.
Cite this review
Pith. "Pith review of BEST-STD: Bidirectional Mamba-Enhanced Speech Tokenization for Spoken Term Detection." pith.science (2026). https://pith.science/paper/XFLM6AKG
@misc{pith2026241114100,
author = {Pith},
title = {Pith review of: BEST-STD: Bidirectional Mamba-Enhanced Speech Tokenization for Spoken Term Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/XFLM6AKG}},
note = {Machine review of arXiv:2411.14100}
}
read the original abstract
Spoken term detection (STD) is often hindered by reliance on frame-level features and the computationally intensive DTW-based template matching, limiting its practicality. To address these challenges, we propose a novel approach that encodes speech into discrete, speaker-agnostic semantic tokens. This facilitates fast retrieval using text-based search algorithms and effectively handles out-of-vocabulary terms. Our approach focuses on generating consistent token sequences across varying utterances of the same term. We also propose a bidirectional state space modeling within the Mamba encoder, trained in a self-supervised learning framework, to learn contextual frame-level features that are further encoded into discrete tokens. Our analysis shows that our speech tokens exhibit greater speaker invariance than those from existing tokenizers, making them more suitable for STD tasks. Empirical evaluation on LibriSpeech and TIMIT databases indicates that our method outperforms existing STD baselines while being more efficient.
Figures
Reference graph
Works this paper leans on
-
[1]
Alberti, Christopher, Michiel Bacchiani, Ari Bezman, Ciprian Chelba, Anastassia Drofa, Hank Liao, Pedro Moreno et al. ”An audio indexing system for election video material.” In 2009 IEEE International Confer- ence on Acoustics, Speech and Signal Processing, pp. 4873-4876. IEEE, 2009
work page 2009
-
[2]
Ogata, J. and Goto, M., 2009. PodCastle: Collaborative training of acous- tic models on the basis of wisdom of crowds for podcast transcription. In Tenth Annual Conference of the International Speech Communication Association
work page 2009
-
[3]
Wang, Y .Y ., Yu, D., Ju, Y .C. and Acero, A., 2008. An introduction to voice search. IEEE Signal Processing Magazine, 25(3), pp.28-38
work page 2008
-
[4]
Mamou, J., Ramabhadran, B. and Siohan, O., 2007, July. V ocabulary independent spoken term detection. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 615-622)
work page 2007
-
[5]
Miller, D.R., Kleber, M., Kao, C.L., Kimball, O., Colthurst, T., Lowe, S.A., Schwartz, R.M. and Gish, H., 2007, August. Rapid and accurate spoken term detection. In Interspeech (V ol. 7, pp. 314-317)
work page 2007
-
[6]
Wang, D., Frankel, J., Tejedor, J. and King, S., 2008, March. A comparison of phone and grapheme-based spoken term detection. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 4969-4972). IEEE
work page 2008
-
[7]
Saraclar, M. and Sproat, R., 2004. Lattice-based search for spoken utterance retrieval. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004 (pp. 129-136)
work page 2004
-
[8]
Can, D. and Saraclar, M., 2011. Lattice indexing for spoken term detec- tion. IEEE Transactions on Audio, Speech, and Language Processing, 19(8), pp.2338-2347
work page 2011
Show all 42 references
-
[9]
and Bourlard, H., 2020
Ram, D., Miculicich, L. and Bourlard, H., 2020. Neural network based end-to-end query by example spoken term detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28, pp.1416- 1427
2020
-
[10]
and Bourlard, H., 2018, September
Ram, D., Miculicich, L. and Bourlard, H., 2018, September. CNN Based Query by Example Spoken Term Detection. In Interspeech (pp. 92-96)
2018
-
[11]
Segmental DTW: A parallelizable alternative to dynamic time warping
Tsai, T.J., 2021, June. Segmental DTW: A parallelizable alternative to dynamic time warping. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 106-110). IEEE
2021
-
[12]
and Lee, L.S., 2016
Chung, Y .A., Wu, C.C., Shen, C.H., Lee, H.Y . and Lee, L.S., 2016. Unsupervised learning of audio segment representations using sequence- to-sequence recurrent neural networks. In Proc. Interspeech (pp. 765- 769)
2016
-
[13]
and Livescu, K., 2016
He, W., Wang, W. and Livescu, K., 2016. Multi-view recurrent neural acoustic word embeddings. arXiv preprint arXiv:1611.04496
2016 arXiv
-
[14]
and Lee, L.S., 2018, December
Chen, Y .C., Huang, S.F., Shen, C.H., Lee, H.Y . and Lee, L.S., 2018, December. Phonetic-and-semantic embedding of spoken words with applications in spoken content retrieval. In 2018 IEEE Spoken Language Technology Workshop (SLT) (pp. 941-948). IEEE
2018
-
[15]
and Livescu, K., 2016, March
Kamper, H., Wang, W. and Livescu, K., 2016, March. Deep convolu- tional acoustic word embeddings using word-pair side information. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4950-4954). IEEE
2016
-
[16]
Acoustic span embeddings for multilingual query-by-example search
Hu, Y ., Settle, S., and Livescu, K., 2021, January. Acoustic span embeddings for multilingual query-by-example search. In 2021 IEEE Spoken Language Technology Workshop (SLT) (pp. 935-942). IEEE
2021
-
[17]
and Arora, V ., 2022
Singh, A., Demuynck, K. and Arora, V ., 2022. Attention-based audio embeddings for query-by-example. In Proceedings of the 23rd Interna- tional Society for Music Information Retrieval Conference, ISMIR 2022 (pp. 52-58)
2022
-
[18]
FlowHash: Accelerating Audio Search with Balanced Hashing via Normalizing Flow
Singh, A., Demuynck, K., and Arora, V ., 2024. FlowHash: Accelerating Audio Search with Balanced Hashing via Normalizing Flow. IEEE/ACM Transactions on Audio, Speech, and Language Processing
2024
-
[19]
Simultaneously learning robust audio embeddings and balanced hash codes for query-by-example
Singh, A., Demuynck, K., and Arora, V ., 2023. Simultaneously learning robust audio embeddings and balanced hash codes for query-by-example. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE
2023
-
[20]
and Khudanpur, S., 2015, April
Panayotov, V ., Chen, G., Povey, D. and Khudanpur, S., 2015, April. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 5206-5210). IEEE
2015
-
[21]
and Pallett, D.S.,
Garofolo, J.S., Lamel, L.F., Fisher, W.M., Fiscus, J.G. and Pallett, D.S.,
-
[22]
and Mohamed, A., 2021
Hsu, W.N., Bolte, B., Tsai, Y .H.H., Lakhotia, K., Salakhutdinov, R. and Mohamed, A., 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29, pp.3451-3460
2021
-
[23]
and Wu, J., 2022
Chen, S., Wang, C., Chen, Z., Wu, Y ., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X. and Wu, J., 2022. Wavlm: Large-scale self- supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6), pp.1505-1518
2022
-
[24]
and Qiu, X., 2023
Zhang, X., Zhang, D., Li, S., Zhou, Y . and Qiu, X., 2023. Speechtok- enizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692
2023 arXiv
-
[25]
and Adi, Y ., 2022
D ´efossez, A., Copet, J., Synnaeve, G. and Adi, Y ., 2022. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438
2022 arXiv
-
[26]
and R ´e, C., 2021
Gu, A., Goel, K. and R ´e, C., 2021. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396
2021 arXiv
-
[27]
and R ´e, C., 2020
Gu, A., Dao, T., Ermon, S., Rudra, A. and R ´e, C., 2020. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33, pp.1474-1487
2020
-
[28]
and Berant, J., 2022
Gupta, A., Gu, A. and Berant, J., 2022. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35, pp.22982-22994
2022
-
[29]
and Gu, A., 2024
Dao, T. and Gu, A., 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060
2024 arXiv
-
[30]
and Vinyals, O., 2017
Van Den Oord, A. and Vinyals, O., 2017. Neural discrete representation learning. Advances in neural information processing systems, 30
2017
-
[31]
and Wu, Y ., 2021
Yu, J., Li, X., Koh, J.Y ., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y ., Baldridge, J. and Wu, Y ., 2021. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627
2021 arXiv
-
[32]
The Finley affair: A signal event in the history of forecast verification
Murphy, A.H., 1996. The Finley affair: A signal event in the history of forecast verification. Weather and forecasting, 11(1), pp.3-20
1996
-
[33]
Learning to rank for information retrieval
Liu, T.Y ., 2009. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval, 3(3), pp.225-331
2009
-
[34]
and Penagarikano, M., 2013
Rodriguez-Fuentes, L.J. and Penagarikano, M., 2013. MediaEval 2013 spoken web search task: system performance measures. n. TR-2013- 1, Department of Electricity and Electronics, University of the Basque Country
2013
-
[35]
and Cernock ´y, J., 2018, June
Silnova, A., Matejka, P., Glembek, O., Plchot, O., Novotn ´y, O., Grezl, F., Schwarz, P., Burget, L. and Cernock ´y, J., 2018, June. BUT/Phonexia Bottleneck Feature Extractor. In Odyssey (pp. 283-287)
2018
-
[36]
and Trancoso, I., 2016, September
Abad, A., Ribeiro, E., Kepler, F.N., Astudillo, R.F. and Trancoso, I., 2016, September. Exploiting Phone Log-Likelihood Ratio Features for the Detection of the Native Language of Non-Native English Speakers. In INTERSPEECH (pp. 2413-2417)
2016
-
[37]
and Chou, J.C., 2021
Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., Zhong, J. and Chou, J.C., 2021. SpeechBrain: A general-purpose speech toolkit. arXiv preprint arXiv:2106.04624
2021 arXiv
-
[38]
and Zhang, J., 2023
Wang, C., Liao, M., Huang, Z., Lu, J., Wu, J., Liu, Y ., Zong, C. and Zhang, J., 2023. Blsp: Bootstrapping language-speech pre- training via behavior alignment of continuation writing. arXiv preprint arXiv:2309.00916
2023 arXiv
-
[39]
and Zhang, C., 2023
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z. and Zhang, C., 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289
2023 arXiv
-
[40]
and Zhou, J., 2023
Chu, Y ., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C. and Zhou, J., 2023. Qwen-audio: Advancing universal audio under- standing via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919
2023 arXiv
-
[41]
and Wei, F., 2024
Hu, S., Zhou, L., Liu, S., Chen, S., Hao, H., Pan, J., Liu, X., Li, J., Sivasankaran, S., Liu, L. and Wei, F., 2024. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00656
2024 arXiv
-
[1993]
NIST speech disc 1-1.1
DARPA TIMIT acoustic-phonetic continous speech corpus CD- ROM. NIST speech disc 1-1.1. NASA STI/Recon technical report n, 93, p.27403
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.